Pro Application Lifecycle Management with Visual Studio 2012

Second Edition

Joachim Rossberg

Mathias Olausson

Pro Application Lifecycle Management with Visual Studio 2012

Copyright © 2012 by Joachim Rossberg and Mathias Olausson

This work is subject to copyright. All rights are reserved by the Publisher, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission or information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology now known or hereafter developed. Exempted from this legal reservation are brief excerpts in connection with reviews or scholarly analysis or material supplied specifically for the purpose of being entered and executed on a computer system, for exclusive use by the purchaser of the work. Duplication of this publication or parts thereof is permitted only under the provisions of the Copyright Law of the Publisher’s location, in its current version, and permission for use must always be obtained from Springer. Permissions for use may be obtained through RightsLink at the Copyright Clearance Center. Violations are liable to prosecution under the respective Copyright Law.

ISBN-13 (pbk): 978-1-4302-4344-1

ISBN-13 (electronic): 978-1-4302-4345-8

Trademarked names, logos, and images may appear in this book. Rather than use a trademark symbol with every occurrence of a trademarked name, logo, or image we use the names, logos, and images only in an editorial fashion and to the benefit of the trademark owner, with no intention of infringement of the trademark.

The use in this publication of trade names, trademarks, service marks, and similar terms, even if they are not identified as such, is not to be taken as an expression of opinion as to whether or not they are subject to proprietary rights.

While the advice and information in this book are believed to be true and accurate at the date of publication, neither the authors nor the editors nor the publisher can accept any legal responsibility for any errors or omissions that may be made. The publisher makes no warranty, express or implied, with respect to the material contained herein.

President and Publisher: Paul Manning

Lead Editor: Tom Welsh

Technical Reviewers: Jakob Ehn, Todd Meister, Gregg Boer, and Anutthara Bharadwaj

Editorial Board: Steve Anglin, Ewan Buckingham, Gary Cornell, Louise Corrigan, Morgan Ertel, Jonathan Gennick, Jonathan Hassell, Robert Hutchinson, Michelle Lowman, James Markham, Matthew Moodie, Jeff Olson, Jeffrey Pepper, Douglas Pundick, Ben Renow-Clarke, Dominic Shakeshaft, Gwenan Spearing, Matt Wade, Tom Welsh

Coordinating Editors: Corbin Collins and Mark Powers

Copy Editors: Linda Seifert, Pat Morris, and Brendan Frost

Compositor: SPi Global

Indexer: SPi Global

Artist: SPi Global

Cover Designer: Anna Ishchenko

Distributed to the book trade worldwide by Springer Science+Business Media New York, 233 Spring Street, 6th Floor, New York, NY 10013. Phone 1-800-SPRINGER, fax (201) 348-4505, e-mail orders-ny@springer-sbm.com, or visit www.springeronline.com.

For information on translations, please e-mail rights@apress.com, or visit www.apress.com.

Apress and friends of ED books may be purchased in bulk for academic, corporate, or promotional use. eBook versions and licenses are also available for most titles. For more information, reference our Special Bulk Sales–eBook Licensing web page at www.apress.com/bulk-sales.

Any source code or other supplementary materials referenced by the author in this text is available to readers at www.apress.com/978430243441. For detailed information about how to locate your book’s source code, go to www.apress.com/source-code.

To Eddie and Amelie

—Joachim Rossberg

To Susanne, Linn and Linus.

—Mathias Olausson

Contents at a Glance

About the Authors

About the Technical Reviewers

Acknowledgments

Introduction

Part 1: Application Lifecycle Management

Chapter 1: Why Application Lifecycle Management Matters

Chapter 2: Introduction to Application Lifecycle Management

Chapter 3: Development Processes and Frameworks

Chapter 4: ALM Assessments

Chapter 5: ALM Using TFS

Part 2: Planning (Agile Project Management)

Chapter 6: Introduction to Agile

Chapter 7: Work Item Tracking and Process Customization

Chapter 8: Agile Project Management With TFS

Chapter 9: Metrics and ALM Assessment for Agile Project Management

Part 3: Analysis, Modeling, and Design (Architecture)

Chapter 10: Prototyping Using Storyboarding and Feedback Tracking

Chapter 11: Top Down Design Studies (UML)

Chapter 12: Using Architecture Explorer

Chapter 13: Using Layer Diagrams

Chapter 14: Metrics and ALM Assessment for Architecture, Analysis and Design

Part 4: Building (Developer Practices)

Chapter 15: Version Control

Chapter 16: Unit Testing

Chapter 17: Code Quality

Chapter 18: Performance and Profiling

Chapter 19: Metrics and ALM Assessment for Developer Practices

Part 5: Validating (Software Testing)

Chapter 20: Overview of Software Testing

Chapter 21: Manual Testing

Chapter 22: Automated Testing

Chapter 23: Visual Studio Lab Management

Chapter 24: Metrics and ALM Assessment for Software Testing

Part 6: Releasing

Chapter 25: Continuous Delivery

Chapter 26: Build Automation

Chapter 27: Deployment

Chapter 28: Release Management

Chapter 29: Metrics and ALM Assessment

Part 7: Team Foundation Server

Chapter 30: Architecture and Implementation Planning

Chapter 31: TFS Installation, Upgrade, and Administration

Chapter 32: TFS Reporting

Chapter 33: Working in Heterogeneous Environments

Chapter 34: TFS In The Cloud

Index

Contents

About the Authors

About the Technical Reviewers

Acknowledgments

Introduction

Part 1: Application Lifecycle Management

Chapter 1: Why Application Lifecycle Management Matters

Understanding the Cornerstones of Business

Processes

Business Rules

Information

Understanding the Need for Business Software

Today’s Business Environment and the Problems We Face

Project Health Today: Three Criteria for Success

Factors Influencing Projects and Their Success

Project Success in Research

The Standish Group

Robert C. Glass

Size and Volatility Survey

Conclusions

IT Budget Spending

Development vs. Operations

Factors Influencing IT Spending

Summary

Chapter 2: Introduction to Application Lifecycle Management

Roles in the ALM Process

Four Ways of Looking at Application Lifecycle Management

The SDLC View

The Service Management or Operations View

The Application Portfolio Management View

The Unified View

Three Pillars of Application Lifecycle Management

Traceability of Relationships Between Artifacts

Automation of High-Level Processes

Visibility into the Progress of Development Efforts

A Brief History of ALM Tools

Application Lifecycle Management 1.0

Application Lifecycle Management 2.0

Application Lifecycle Management 2.0+

ALM and PPM

Summary

Chapter 3: Development Processes and Frameworks

The Waterfall Model

Spiral Model

Rational Unified Process (RUP)

The Principles of RUP

The RUP Lifecycle

Disciplines

Manifesto for Agile Software Development

Extreme Programming (XP)

Scrum

Empirical Process Control

Complexity in Projects

What Scrum Is

The Roles in Scrum

The Scrum Process

Capability Maturity Model Integration (CMMI)

Summary

Chapter 4: ALM Assessments

Microsoft Application Platform Optimization (APO) Model

Infrastructure Optimization Model

Business Productivity Infrastructure Model

Application Platform Capability Assessment

Summary

Chapter 5: ALM Using TFS

Application Lifecycle Management Overview

Team Foundation Server Overview

Team Foundation Server

Process Template

Visual Studio 2012 Editions

Web Access

Microsoft Office

Integrated Development Environment (IDE) Integration

Traceability

The TFS Work Item Tracking System

Visibility

Collaboration

Work Items for Collaboration

The Gap Between IT and Business

Use of One Tool/Role Based

Extensibility

Summary

Part 2: Planning (Agile Project Management)

Chapter 6: Introduction to Agile

The Scrum Process

Roles in Scrum

Product Owner

Scrum Master

The Development Team

Definition of Done

Agile Requirements and Estimation

Requirements

Estimation

Backlog

Summary

Chapter 7: Work Item Tracking and Process Customization

Traceability

The TFS Work Item Tracking System

Work Items

The Work Item Form

Work Item Traceability

Work Item Queries

Conclusions on the Work Item Tracking System

Process Customization

Modifying the Process Template

Common Adaptations of the Process Template

Web Access Customization

Using WITAdmin to Change the Web Access UI

Changing the Product Backlog Add Dialog

Changing the Columns in the Product Backlog List

Adding States to the Board

Summary

Chapter 8: Agile Project Management With TFS

Case Study

Company Background

The Pilot Project

Scrum TFS Template

Work Items

TFS Web Access and Portal

Reports and Queries

Project Startup Phase

PO Sets Off to Work

Building the Initial Team

Requirements

Building the Backlog and Prioritizing It

Definition of Done

Estimation

Risk Assessment

Grooming the Backlog

Building the Team

Initial Velocity

Release Planning

Estimated Time Plan

Estimated Project Cost

Running the Project

During the Sprint(s)

Summary

Chapter 9: Metrics and ALM Assessment for Agile Project Management

Metrics

Standard Reports

Scrum

MSF for Agile

MSF for CMMI

Custom Reporting

Data Warehouse Model

Assessment Questions

Summary

Part 3: Analysis, Modeling, and Design (Architecture)

Chapter 10: Prototyping Using Storyboarding and Feedback Tracking

PowerPoint Storyboarding

How to Do It

When to Use Storyboarding

Feedback Tracking Using Microsoft Feedback Client

Summary

Chapter 11: Top Down Design Studies (UML)

UML Basics

UML Support in Visual Studio 2012

Activity Diagram

Use Cases and Use Case Diagrams

Sequence Diagrams

Class Diagrams

Component Diagrams

Integration with TFS

Summary

Chapter 12: Using Architecture Explorer

Architecture Explorer

Dependency Graphs

Analyze and Explore Code Using Architecture Explorer

Summary

Chapter 13: Using Layer Diagrams

Patterns

Using Layer Diagrams

Creating a Layer Diagram

Mapping Code to Layers

Viewing Dependencies using Layer Explorer

Sharing Models

Layer Validation

Manually Validate Layer

Validate in local build

Validate in TFS Build

Creating an Architecture Template

Summary

Chapter 14: Metrics and ALM Assessment for Architecture, Analysis and Design

Metrics

Standard Reports

Assessment

Summary

Part 4: Building (Developer Practices)

Chapter 15: Version Control

Getting Started with Source Control

Using the Team Explorer

Workspaces

Configuring a Workspace

Using the Source Control Explorer

Committing Work

Working with Committed Code

Enforcing Development Practices with Check-in Policies

Putting Work Aside

Conflict Resolution

Comparing Changes

Working with Concurrent Changes

Working with My Work

Summary

Chapter 16: Unit Testing

What Is Test Driven Development?

Principles for Unit Testing

Working with Mock Objects

Separation of Concerns (Object Factories)

Inversion of Control (Dependency Injection)

Visual Studio Unit Tests

Elements of a Unit Test

Test Calling Order

Assertions

Creating a Unit Test

Context for the Test Run

Test Explorer

Running Tests

Working with Code Coverage

Adding Code Coverage to a Project

Configuring Code Coverage for a Build Definition

Data-Driven Automated Tests

Working with External Files

Managing Test Dependencies

Working with Other Test Frameworks

Implementing Mock Objects

Summary

Chapter 17: Code Quality

Code Analysis

Creating Custom Rule Sets

Integrating Code Analysis with TFS Builds

Code Analysis Check-in Policy

Code Metrics

Code Clones

Code Review

Summary

Chapter 18: Performance and Profiling

Profiling Overview

Profiling In Visual Studio 2012

Setting Up Profiling

Running a Profiling Session

Examining the Results

IntelliTrace

Working with IntelliTrace

Using IntelliTrace in Production

PreEmptive Analytics

Installing PreEmptive Analytics

Configuring Your Team Projects for PreEmptive Analytics

Adding Exception Reporting to a .NET Application

Examining Incidents in Team Foundation Server

Summary

Chapter 19: Metrics and ALM Assessment for Developer Practices

Metrics

Code coverage

Code metrics

Compiler warnings

Code analysis warnings

Standard Reports

Custom Reporting

Data Warehouse Model

Assessment

Summary

Part 5: Validating (Software Testing)

Chapter 20: Overview of Software Testing

Agile Testing

Defining Tests

Acceptance Criteria

Planning

Test Specification and Test Matrix

Evolving Tests

Strategy for Automated Testing

Platform Support for Testing Practices

Summary

Chapter 21: Manual Testing

About Microsoft Test Manager

Connecting Microsoft Test Manager to TFS

Planning the Tests

What Is a Test Plan?

Moving On

Designing Test Cases

What Is a Test Case?

Creating Test Cases

Moving On

Running Tests

Filtering Test Runs

Working with the Test Runner

Analyze Test Runs

Running Exploratory Tests

Test Settings

Integration with Builds

Reporting Bugs and Validating Fixes

Creating a Bug

Verifying Bugs

Fast-Forward Playback

Summary

Chapter 22: Automated Testing

Working with Automated Tests

Supported Test Types

Creating a Test Environment

Traceability

Automating Test Cases Using Coded UI Tests

Coded UI Tests

UI Test Architecture

Supported Platforms

Elements of a Coded UI Test

Using the Coded UI Test Builder

Validation

Using the UI Map Editor

Data-Driven Automated Tests

Using Unit Tests as Automated Tests

Running Automated Tests

Running Automated Tests from MTM

Running Automated Tests from TFS Build

Running Automated Tests from the Command Line

Performance Testing the Application

Creating Load Test Scenarios Using Web Performance Tests

Setting Up a Load Test

Running a Load Test

Summary

Chapter 23: Visual Studio Lab Management

Architecture

Capabilities

Components

Architecture

Setting Up Visual Studio Lab Management for Kangreen

Environments

Designing Environments

Working with Standard Environments

SCVMM Environments

What About Support for VMware?

Test Settings for Lab Environments

Creating Environments for Kangreen

Cloning Environments

Using Lab Management for Manual Testing

Selecting an Environment to Use

Connecting to an Environment for Testing

Automated Testing and Lab Management

Running Automated Tests from MTM

Running Automated Tests as Part of a Build

Running Automated Tests from the Command-line

Summary

Chapter 24: Metrics and ALM Assessment for Software Testing

Metrics

Standard Reports

Bug Status Report

Reactivations Report

Bug Trend Report

Test Case Readiness Report

Test Plan Progress Report

Custom Reporting

Data Warehouse Model

Assessment

Summary

Part 6: Releasing

Chapter 25: Continuous Delivery

Continuous Integration

Why Should We Implement Continuous Integration?

Components in Continuous Integration

Continuous Delivery

The “null” Release Cycle

Release Management

Summary

Chapter 26: Build Automation

TFS Build Architecture

Installing the Build Service

Installing the Build Controller

Installing the Build Agent

Creating an Automated Build

Running a Build

Using the Build Explorer

Using the Web Access to Manage Builds

Implementing Continuous Integration

Cloning a Build

Gated Check-in Builds

Running a Private Build

Configuring Builds

Required Parameters

Basic Parameters

Advanced Parameters

Customizing Builds

Built-in Build Activities

TFS Build Extensions

Creating a Build Process Template for Assembly Versioning

Logging In the Build Process

Extending Builds

Creating a Solution for Developing Build Extensions

Implementing an XAML Activity to Execute a Remote Command

Implementing a Code Activity

Build on Team Foundation Service

Feedback

Build Notifications

Team Explorer and Web Access Favorites

Summary

Chapter 27: Deployment

Designing a Deployment Solution

Deployment to Windows Azure

Running Automated Tests as Part of the Build

Running Tests on the Build Server

Running Tests on a Test Environment

Fail a Build on Test Failure

Implementing the Build-Deploy-Test Workflow

Implementing a BDT Workflow

Designing the BDT Workflow

Implementing the BDT Process

Running the BDT Workflow

Customizing the Lab Build Template

Summary

Chapter 28: Release Management

Release Management

Agile Release Planning

Release Management in Visual Studio 2012

Release Notes

Release Notes in Visual Studio 2012

Versioning

Versioning Requirements

Versioning Code

Versioning Test Cases

Versioning Builds

Summary

Chapter 29: Metrics and ALM Assessment

Metrics

Standard Reports

Custom Reporting

Data Warehouse Model

Assessment

Summary

Part 7: Team Foundation Server

Chapter 30: Architecture and Implementation Planning

TFS Architecture

Logical Application Tier

Logical Data Tier

Team Foundation Client Tier

Optional Components on Separate Servers

Team Foundation Build

Team Foundation Server Proxy

Planning the Implementation

Physical Server Location to Host Our TFS Tiers

Ports and Network Protocols

System Requirements

Service Accounts

Simple Implementation

Medium Implementation

Complex Implementation

Previous Version of TFS Already Deployed

Summary

Chapter 31: TFS Installation, Upgrade, and Administration

Installation Options

Basic

Advanced

Application-Tier

Upgrading from Earlier TFS Versions

Team Foundation Server Administration

Adding TFS Collections After Installation

Attaching a TFS database

Enabling Web Features

Enabling Email Alerts

Summary

Chapter 32: TFS Reporting

Architecture

Data Sources for Reporting

Warehouse Adapters

Relational Data Warehouse Data Models

Analysis Services OLAP Cube

Setting Up Access to Report Data

Monitoring the TFS Data Warehouse

Creating Reports

Work Item Queries

Excel Reports

SQL Server Report Builder

SQL Server Report Designer

Summary

Chapter 33: Working in Heterogeneous Environments

What is Team Explorer Everywhere?

A Little Bit about Licensing

Features Available in Team Explorer Everywhere

Installation in Eclipse

Connect to TFS

Using TEE from Eclipse

Ant and Maven 2 Build Improvements

Summary

Chapter 34: TFS In The Cloud

Windows Azure Overview

Compute

Storage

Fabric

The Azure Platform

TFS on Azure

Getting Started

Documentation and Help

Services Offered With TFS in the Cloud

Source Control

Work-item tracking

Process Templates

Testing

Hosted Build Service

What about Reporting?

Benefits of TFS in the Cloud

Concerns with TFS in the Cloud

Summary

Index

About the Authors

Joachim Rossberg has worked as an information technology consultant since 1998. He is primarily a Scrum Master and Product Owner but has an extensive history as a system developer/ designer. He has demonstrated his technical background with various achievements over the years: MCSD, MCDBA, MCSA, and MCSE. His specialties include project management, Scrum, development processes, and Visual Studio. Joachim is now working for Knowit System Development in Gothenburg, Sweden.

Mathias Olausson is the ALM practice lead for Transcendent Group, specializing in software craftsmanship and application lifecycle management. With over 15 years of experience as a software consultant and trainer, he has worked in numerous projects and organizations, which has been very valuable when using Visual Studio as a tool for improving the way we build software. Olausson has been a Microsoft Visual Studio ALM MVP for four years. He is also active as a Visual Studio ALM Ranger, most recently in the role of project lead for the Visual Studio Lab Management Guide project. Olausson is a frequent speaker on Visual Studio and Team Foundation Server at conferences and industry events and blogs at http://msmvps.com/blogs/molausson.

About the Technical Reviewers

Jakob Ehn is a current Microsoft Visual Studio ALM Most Valuable Professional (MVP) and also a Visual Studio ALM Ranger. Jakob has 15 years of experience in the IT industry, and currently works as a solution architect at Inmeta Crayon ASA, specializing in Visual Studio ALM. He actively participates in the MSDN forums, and contributes to different open source projects, such as the Community TFS Build Extensions and the Community TFS Build Manager.

Todd Meister has been working in the IT industry for over 15 years. He’s been a Technical Editor on over 75 titles, ranging from SQL Server to the .NET Framework. Besides technical editing titles, he is the Senior IT Architect at Ball State University in Muncie, Indiana. He lives in central Indiana with his wife, Kimberly, and their five skilled children.

Gregg Boer is a Principal Program Manager at Microsoft with 25 years of experience in software. Over his career, Gregg has worked as a Project Manager, Program Manager, Requirements Lead, Software Engineer, Analyst, QA Lead, and Software Designer. Most of his experience is focused on leading small to medium software development teams (10–15 people). Over the years Gregg believes he has made many mistakes, learned many lessons, and worked with many smart people to develop great software. He loves the software industry, and considers himself a lucky man to have been a part of it for so long.

Gregg joined Microsoft in 2005 because he believed in the vision of Team Foundation Server. He truly believes that the right set of collaborative tools could help make the lives of software engineers better. Most recently at Microsoft, Gregg is working on the team developing a set of world-class Agile Tools built on top of the TFS Platform.

Anutthara Bharadwaj is Group Program Manager in the Visual Studio ALM group at Microsoft Corporation. Anu has enjoyed building and breaking software in the various roles she has worked in through her nine-year Microsoft stint, shipping multiple versions of Visual Studio ALM. Anu has worked on some cool tools with Exploratory Testing as a central theme and has three patents from her research work in this area. Anu is passionate about software development and testing and has spoken at conferences worldwide. She blogs at http://blogs.msdn.com/anutthara and tweets at http://twitter.com/anutthara.

Acknowledgments

I would like to thank all the people that helped us write this book. Unfortunately, I do not have room to mention each one of you, but to some I want to give special thanks.

First of all I am thankful that Mathias wanted to co-author this book. Your expertise is invaluable and the book is so much better than I could ever hope for.

Mark Powers at Apress. Thanks for pushing us forward the entire journey. It was a pleasure working with you on this.

Anutthara Bharadwaj, Gregg Boer, Todd Meister, and Jakob Ehn, our great technical reviewers. Thanks for coming with great input and feedback that helped improve the book.

Sam Guckenheimer at Microsoft. Thanks for getting us in contact with great technical reviewers.

Per Wallentin, Fredrik Abrahamsson, and Martin Skarp at Knowit Sweden. Thanks for all support.

Urban Bäckemo at Knowit System Development. Thanks for feedback and inspiration to the agile parts.

All you other great people at Apress. Thanks!

Thanks as well to my family that put up with me during writing.

And last but not least. Thanks to all you readers out there. If it wasn’t for you we would not be doing this.

—Joachim Rossberg

I would like to thank my co-author Joachim for taking me on this journey. It’s been an exciting project and a pleasure to work with you. We’ve talked about writing a book many times and I’m really happy we finally got the first one out!

Thanks to everyone at Apress, it’s been great working with all of you. A special thanks to our editors Mark Powers and Tom Welsh for keeping us on track. Your knowledge and advice has been invaluable.

I would also like to thank our technical reviewers Anutthara Bharadwaj, Gregg Boer, Todd Meister and Jakob Ehn for their feedback and ideas, which helped us shape this book. Jakob, you have been the primary reviewer for the chapters I contributed, and I’m very thankful for your thorough readings and great feedback. I owe you one!

Thanks to Magnus Timner, Gustav Sjöberg, and the rest of the ALM team at Transcendent Group for your feedback and support. I’m fortunate to have such competent colleagues to discuss thingswith!

Finally, thanks to my family for your love and understanding throughout this project.

—Mathias Olausson

Introduction

You can have the best coders in the world working in your teams, but if your project management isn’t up to scratch, your project is almost certain to be delayed, to come in over budget, and in some cases, to fail entirely. By taking precise control of your application development process, you can make changes, both large and small, throughout your project’s life cycle that will lead to better-quality, finished products that are consistently delivered on time and within budget.

Application lifecycle management (ALM) is an area of rapidly growing interest within the development community. Because its techniques allow you to deal with the process of developing applications across many areas of responsibility and across many different disciplines, its effects on your project can be wide-ranging and pronounced. It is a project management tool that has practical implications for the whole team—from architects to designers, from developers to testers.

Who This Book Is For

This book is for anyone interested in improving the development efforts in their organizations. It doesn’t matter if you are a manager, developer, tester, Scrum Master, or anything else. You can all benefit from what you will learn here. The Application Lifecycle Management process includes anyone involved in the lifecycle of an application, and Team Foundation Server 2012 and Visual Studio 2012 have something for each and every one of you. Maybe the most important lesson is that you are all working on the same team, and you are all responsible for the outcome of your development process. This realization cannot come from a tool like Team Foundation Server or Visual Studio. It is something that you need to figure out all by yourself.

How This Book Is Structured

This book is split into seven parts that will show you how you can use Visual Studio and Team Foundation Server (TFS) 2012 to implement an Application Lifecycle Management (ALM) process in your organization.

Part I explains what Application Lifecycle Management is and what problems it aims to solve. We also cover different project management processes and frameworks so that you can select the most appropriate for your organization.

Part II focuses on agile project management and how Visual Studio and Team Foundation Server 2012 can help by supporting an agile project management approach.

Part III discusses the architecture features of Visual Studio and Team Foundation Server 2012. There are several tools available that can help developers and architects in their daily work.

Part IV covers the developer tools of Visual Studio and TFS 2012. Here you see how these tools integrate with an overall ALM process that enables you to gain better control of development efforts.

Part V shows the testing features of Visual Studio and TFS 2012. It is intended for developers and testers alike.

Part VI describes how to create an effective build and release process.

Part VII focuses on Team foundation Server and covers its architecture and its extensibility, and not only on the Windows platform.

Contacting the Authors

Should you have any questions or comments—or spot a mistake you think we should know about—you can contact the authors at Rossberg@gmail.com or mathias@olausson.net.

PART 1

Application Lifecycle Management

Part I of this book covers the concept of Application Lifecycle Management (ALM). We show you what ALM is and why it matters, as well as how it can help you and your organization be more efficient in your development efforts.

We also take a look at some of the most common development processes and frameworks available to run projects. Choosing the best process is important for the success of any project. There has been a movement towards agile frameworks in recent years, leaving waterfall and RUP behind in many organizations.

Before you try to implement an ALM process in your organization it is important to know what the pains in the current processes are. By performing an ALM assessment you can evaluate where you need to focus and come up with an action plan tailored for your needs.

Last but not least, you see how Visual Studio 2012 and Team Foundation Server 2012 can help to implement an effective and successful ALM solution.

CHAPTER 1

Why Application Lifecycle Management Matters

Modern organizations depend on software and software systems in many ways. Business processes are often implemented in a digital flow and without software to support this, even small companies would experience problems. For most companies, the world has changed quickly in the last few years and they need to adapt constantly.

If you want it these days, information is available at your fingertips all the time. Remember the days back when we were teenagers? Music and movies were, and always will be, two of the top interests. This obsession started during the teen years, and we chased rare records of favorite artists and hard-to-find horror movies everywhere. When a rare vinyl pressing of a precious record from the United States was found, for instance, we were ecstatic. Not to mention the emotional turmoil when we managed to purchase a Japanese edition of the same record. Those days we wrote snail mail asking for mail-order record catalogs from all over the world, based on ads in magazines such as Rolling Stone or Melody Maker. After carefully considering what we wanted to purchase, we wrote down the purchase order, enclosed crisp bills, and sent a letter with the order inside. Then came the long wait for the package. And believe you me, this wait could be long indeed. Nowadays we just access the Internet, check some sites, and directly purchase what we want by using a credit card. The stock of many sites is so huge compared to what it was in our teens, and we can usually find what we want very quickly. In a few days the package comes, and we can start using the things we bought.

We communicate differently as well. Sites such as Facebook, Twitter, and so on have generated millions of followers, not only by the early adopters of technology, but by our societies as a whole. The numbers of smartphones (iPhone, Android devices, Windows Phone, and more), tablets, and other means of communication practically have exploded, at least in the parts of the world where the infrastructure for this is available.

With the new opportunities organizations have to do business, much has changed in the world for us, including the reality for our companies. Companies now have to deal with a global environment, presenting both opportunities and challenges. Business has changed and still is changing at a rapid pace. We need to be clear on why we develop business systems and software. For companies, development of software has changed as well. Nowadays many organizations have large development teams working on software to support the business. Many times the teams are spread globally. This poses many potential problems, such as collaboration issues, source code maintenance, requirements management, and so on. Without processes to support modern software development, business will likely suffer.

Development teams in organizations use new collaboration tools such as Visual Studio Team Foundation Server, the focus of this book. TFS, as it is generally called, is an Application Lifecycle Management (ALM) platform tying together a company’s business side with its information technology (IT) side. Application Lifecycle Management itself is, briefly, the process an organization uses to care for an application or software system from its conception to its retirement. ALM is the glue that ties together the development processes and defines the efforts necessary to coordinate the process.

Understanding the Cornerstones of Business

First let’s define the term business. What do we mean when we talk about this concept? After agreeing on this, we can reach an understanding of what business software is so we don’t talk about two different things here. When we discuss business in this book, we are talking about not only the commercial part of the company, but all the functions in the company. This means that business software is intended not only for e-commerce, but for all the functions in an enterprise.

There are three cornerstones in business system development that are important:

- Processes

- Business rules

- Information

These three are dependent on each other. Think about it this way. Let’s makes an analogy with the human body. If the processes are the muscles of our company and the rules are the brain and nervous system, we can say that the information can be seen as the spine. None of them could function without the others.

Processes

A company uses different processes to support its business. For developers, project managers, software designers, or people with other roles in a development project, it is easy just to focus on the development process. We are often interested in development processes such as the Scrum process or the Extreme Programming (XP) process. The business people mostly focus on the business side of course, and have no interest in learning about the development process.

Of course, a large company needs processes for procurement, sales, manufacturing, and so on, and the development process is just one of them. The other processes are needed for the company to function and survive.

Obviously, business processes are valid not only for commercial companies but for all organizations, including those in the public sector.

SCRUM, XP, AND RUP

In case you don’t have the full picture of what Scrum, eXtreme Programming (XP), or Rational Unified Process (RUP) are, we will cover them later in this section. For now, suffice it to say that all three are development process models you can use for controlling your development efforts in projects.

Scrum is an iterative and incremental agile software development method for managing software projects and product or application development (http://en.wikipedia.org/wiki/Scrum_(development)).

Although Scrum was intended to be for management of software development projects, it can be used in running software maintenance teams, or as a program management approach.

Scrum is a process skeleton that includes a set of practices and predefined roles. The main roles in scrum are the scrum master, who maintains the processes and works similar to a project manager; the product owner, who represents the stakeholders; and the team, which includes the developers.

During each sprint, a 1530 day period (length decided by the team), the team creates an increment of potential shippable (usable) software. The set of features that go into each sprint come from the product backlog, which is a prioritized set of high-level requirements of work to be done. What backlog items go into the sprint is determined during the sprint planning meeting. During this meeting, the product owner informs the team of the items in the product backlog that he wants completed. The team then determines how much of this they can commit to complete during the next sprint. During the sprint, no one is able to change the sprint backlog, which means that the requirements are frozen for a sprint.

Extreme Programming (XP) is a software development methodology which is intended to improve software quality and responsiveness to changing customer requirements. As a type of agile software development, it advocates frequent “releases” in short development cycles (timeboxing), which is intended to improve productivity and introduce checkpoints where new customer requirements can be adopted.

Other elements of Extreme Programming include: programming in pairs or doing extensive code review, unit testing of all code, avoiding programming of features until they are actually needed, a flat management structure, simplicity and clarity in code, expecting changes in the customer’s requirements as time passes and the problem is better understood, and frequent communication with the customer and among programmers. The methodology takes its name from the idea that the beneficial elements of traditional software engineering practices are taken to ”extreme” levels, on the theory that if a little is good, more is better (http://en.wikipedia.org/wiki/Extreme_programming).

The Rational Unified Process (RUP) is an iterative software development process framework created by the Rational Software Corporation, a division of IBM since 2003. RUP is not a single concrete prescriptive process, but rather an adaptable process framework, intended to be tailored by the development organizations and software project teams that will select the elements of the process that are appropriate for their needs. RUP is a specific implementation of the Unified Process.

Business Rules

The second cornerstone is the business rules the organization needs for it to function well. The business rules tell us what we can and cannot do in the company. They also tell us what we must do. If we compare the processes to the muscles of our body, we can say the rules are equivalent to our brain and nervous system—that is, the things controlling our actions and deeds.

Information

A third cornerstone of any company is its information, that is, information about the company and what is going on in it. For example, we can have all customer information, order information, product catalogs, and so on here. Without access to relevant information at the correct time, the business will quite simply not function. Consider this example: it is impossible for a company to sell any of its products if it has no information about which products it has or what price they sell for.

Understanding the Need for Business Software

So to get back to the question about business systems and software: the reason business software exists is to support the business. Business software should take business needs and requirements and turn them into business value through the use of business software. Application Lifecycle Management is one the processes that can help us deliver this business value. And if IT people do a poor job of building this kind of software or systems by having a broken ALM process, the business will obviously suffer.

This is the reason we need to think about why we develop business software and business systems all the time (no, you do not have to think about it in your free time, even though your manager probably thinks so). We do not write software for an enterprise to fulfill our technological wishes alone; we write it to make the business run smoother and create more value (see Figure 1-1). This does not, however, make it less cool or interesting to learn new technology or write smarter code. Fortunately, these are important parts of any software or system.

Figure 1-1.  The reason we write business software is to turn business needs and opportunities into business value

Today’s Business Environment and the Problems We Face

With the new opportunities organizations have for business these days, much has changed in terms of the realities they face:

- Companies now have to deal with a global environment, presenting both opportunities and challenges. A global way of doing business means competition can come from all sorts of places. Low-cost countries such as China and India can offer many of the same products and services as high-cost countries. This is a great challenge for development organizations all over the world. Consulting firms are opening development shops in low-cost countries, and other companies use the services they provide. An organization’s internal development department may also see their work move to these countries. So no matter where we work, globalization affects us and our jobs, and competition is fierce. In order for us to handle this situation, it is essential to have control over our ALM process. Through this process, we find the support for collaboration between the dispersed teams we see these days, which can give us the competitive edge we need to face competition from others. We need to automate and fine-tune the ALM process we use in our organizations so that we can face challenges, keep costs down, and win deals.

- This new reality has forced businesses to become more agile—ready to transform quickly to gain competitive advantages. This obviously affects the way we must architect and write business systems as well. In the ALM process, these topics are addressed and can help us achieve agility.

- Communication has become more complex and different than previously. Production of products and services is spread over the world, and gone are the days when one industrial plant supplied everything for a company. For us in IT, this means that software development has moved to countries such as India or China and we need to handle this somehow. This is quite a challenge. Just consider the potential communication problems in a company with offices or manufacturing spread across the globe—not to mention problems with time and cultural differences.

As you can see, business processes can (and do) change rapidly. Hence, the supporting IT systems must also be ready for quick changes. If we do not have systems that allow this, business will suffer. This is one of the main reasons ALM tools such as Team Foundation Server (TFS) have emerged. Without an effective development process tied closely to the business side and supported by a set of tools, we will run into problems and risk being left behind by competitors already using such tools. And it is not only the ALM tools that are important; we need to consider the whole ALM process as well, including the way we run our development projects.

Project Health Today: Three Criteria for Success

What do we mean when we talk about project health? How can we measure this? Many surveys indicate the same criteria for success (or failure, if you are so inclined). Let’s take a closer look. There is slight variation, but these three can be said to be the main criteria:

- Project delivered on time

- Project delivered on budget

- Project goals met

Let’s discuss these three a bit. Is it reasonable to use these criteria to evaluate project success or failure? I am a bit skeptical and will explain why.

Projects Delivered on Time

In traditional project models, a lot of effort is put into time estimates based on the requirements specifications. This way of estimating was (and still is) great for construction of buildings, roads, aircraft, and other traditional engineering efforts. These are the projects that traditional project management wisdom comes from.

Such projects are far more static than most software development projects. The engineering discipline is also rigorous in its requirements management process, which helps a lot. You don’t see as many changes to requirements during the process, and the ones that do occur go through a comprehensive change request process. Many companies use Capability Maturity Model Integration (CMMI) to improve their process and thus be better at controlling projects. CMMI enables an organization to implement process improvement and show the level of maturity of a process.[1]

CMMI can be used to guide process improvement across a project, a division, or an entire organization. The model helps integrate traditionally separate organizational functions, set process improvement goals and priorities, provide guidance for quality processes, and provide a point of reference for appraising current processes.

Based on some experiences at the Swedish Road Administration (SRA), where Joachim has been for seven years, design risks when building a road, for instance, are pretty low, design costs are small, especially compared to building costs, and so on. Here you set the design (or architecture) early in the project based on pretty fixed requirements. From this, you can more easily divide the work into smaller pieces and spread them elegantly across your Gantt chart. This also means you can assign resources based on a fixed schedule. Another benefit is that project management will be easier because you can check off completed tasks against your Gantt schema and have better control over when tasks are completed and if there is a delay or lack of resources during the process. On the other hand, if you get an engineering process wrong, lots of money has been wasted, and in the worst case, somebody has lost their life because of poor control of the process.

When it comes to more-complex building, such as a new tunnel the SRA built in Gothenburg and which was opened in 2006, things are a bit different. A tunnel of this magnitude was not something that the construction companies built every day. This made it harder for the team to estimate time and money for the tunnel. In this case, the tunnel opened at almost the estimated opening date . It differed by a couple of months as I recall, which must be considered well done because the whole project took more than five years to complete. The reason for this was that everything from risk management to change requests, and all construction-related tasks, were handled with rigorous processes.

We think that one thing that greatly differs between construction processes and software development processes is that construction workers know that if they make a mistake, somebody might get hurt or die. We in the software development industry tend not see that connection clearly, as long as we aren’t working with software for hospitals, or other such areas. This could be one reason that we haven’t implemented better processes before.

In his book Agile Software Engineering with Visual Studio: From Concept to Continuous Feedback 2nd Edition (Addison-Wesley Professional, 2011), Sam Guckenheimer calls the way of breaking down the project into work tasks a work-down approach because it is easy to see this as a way of burning down a list of tasks. This method of managing projects, he argues, is great for projects with low risk, low variance, and a well-understood design. In the IT world, you can see that implementations of standard products could benefit from this model. In such projects, you can do some minor customizations of the product, and the development effort is pretty small, especially compared to the effort put into business analysis, testing, and so on.

When it comes to IT projects with a lot of development effort, things change. The uncertainty in the projects increases because there are so many ways for things to change unexpectedly. This inherent uncertainty in complex IT projects makes it hard to estimate tasks in a correct way early on. Things happen along the way that throw aside earlier estimates.

Considering this, is it then realistic to measure a complex IT project against planned time? To really know how projects are doing, we might want to consider whether this is just one of the measurements we can use.

Projects Delivered on Budget

Much of the same reasoning in estimating the time of a project applies to estimating costs, because so much of the cost is tied to the people doing the work. But cost involves other factors as well. We have software costs, office costs, and other costs, but these are often easier to estimate than development costs, because they are fixed for the office we use for development. We can put a price tag on a developer, for example, based on that person’s total cost (including location costs,, training, administrative overhead and other overhead costs) the cost of leasing of a computer, and the cost of software licenses. This can be done in advance, and we then know that one developer costs a certain amount of money each day. Development cost, on the other hand, is harder to determine because it is harder to estimate the complexity of the system beforehand. The changes we encounter are hard to estimate in advance and hence the cost is hard to estimate as well.

Project Goal Fulfilled

This is also a tricky criterion, because what does goal fulfillment really mean? Does it mean that all requirements set at the beginning of a project are fulfilled? Or does it mean that the system, when delivered, contains the things the end user wants (and those they maybe don’t want)?

Most surveys seem to take the traditional approach: requirements are set early and never change. But what about the problems we saw with complex projects earlier? Can we really know all the requirements from the start? Something that I think everybody who has participated in a software project can agree on is that requirements change during the course of the project, period!

It might very well be that all the requirements that we knew about from the start have been implemented, but things have changed so much during the project that the users still do not think the project has delivered any value. The project could be seen as successful because it has fulfilled its scope, but is it really successful if the users do not get a system they are satisfied with? Have we really delivered business value to our customer? That is what we really should have as a goal.

All through the development process, we need to identify the business value that we will deliver and make sure we do deliver it. The business value might not be obvious from the start of the project but should be focused on during the process. A good development process and ALM process can help us achieve this.

Let’s now take a look at what factors influence project success.

Factors Influencing Projects and Their Success

As I have said, today’s enterprises face a lot of new challenges. Let’s go through some of these in more detail, starting with the most important one based on the surveys presented earlier but also on my own experience.

The Gap Between Business and IT

Let’s start with the biggest issue, which I mentioned before, as you may recall. IT managers’ top priority was better integration between the company’s business processes and the supporting IT systems. There seems to be quite a collaboration gap between the IT side and the business side, making it difficult to deliver software and systems that really do support the business. IT managers may focus on security, scalability, or availability instead of on supporting the business processes. These are of course important as well, but not the only issues IT should focus on. Business managers, on the other hand, may have trouble explaining what they want from the systems. This collaboration gap poses a great threat not only for projects but also for the entire company.

The Development Process

Let’s continue with the development process. Can this affect success? Obviously, it can. we have seen organizations that have spent lots of effort, time, and money on developing a good process. These organizations have trained both project managers and participants in RUP, XP, or any other development model they chose, and you would think all was dandy. Still, projects seem to suffer quite a lot. One reason for this might be that when a project starts, it is hard to follow the process. RUP, for instance, is often said to be too extensive, with many documents to write and milestones to meet. Let’s face it—even Ivar Jacobson himself seems to think this, considering his latest process development. If the process is seen as a problem or a burden, project members will find ways around it, and the money spent on training and planning will be wasted. The process may also be hard to implement because the tools have no way of supporting it. If we cannot integrate our development process into the tools we use to perform work, we most likely won’t follow the process. Using the process must be easy, and the tools should make the process as transparent as it can be, so that we can focus on work but still follow the process.

When we travel around Sweden talking to organizations about TFS and ALM, we usually ask what development process the organizations use. Often the answer is “the chaos model,” or “the cowboy model,” meaning they use a lot of ad hoc, often manual, efforts to keep it all going. Many say this is due to an inability to integrate their real development model into their tools, but others just had not given it a thought. These companies had hardly considered using any structure in work and if they had, the thoughts had often stayed in the heads of the developers (who quite often are the ones longing for a good process) or managers. Maybe a decision had been made to consider training staff in a model, but the company had never gotten around to it. No wonder these organizations experienced lots of failed or challenged projects.

Speaking of processes. We would say that not having a flexible development process (more on these processes in Chapter 2) most definitely will affect project success. Because business is sure to change during a development project, we need to be flexible in our process so that we can catch these changes and deliver business value in the end. I had a discussion with one of my customers about this some time ago. Most customers agree that there must be a way to make it easier to catch changes to requirements and make the system better reflect reality during the project. Otherwise, the perceived value of the project will suffer. But in this case, the IT manager was almost scared to even consider this. He argued that all requirements must be known at the start of the project and that they must remain static throughout the project. He thought the project would never reach an end otherwise. Not one single argument could break down his wall. He wanted to run his company’s projects by using the Waterfall model (see Chapter 3) as he always had. And still he kept wondering why projects so often ended badly.

Geographic Spread

With development spread across the globe and outsourced development, running projects can be very hard indeed. When development teams in a project are geographically separated, means of communication between them must exist and function seamlessly. For example, how can we share project status in an effective way, so that everybody can see how the project is going? How can we get a good, robust version control of source code and documents to function when we have long distances between teams? How can we catch changes to requirements when users, developers, and decision makers are separated?

This complexity is something that we can see in a recent project at a global company in the dental business. We have development in Sweden, Belgium, the United States, and Canada. This gives us a real headache from time to time especially when we need to collaborate or communicate. In this company we are lacking a clear strategy for how we can improve the ALM process, but we are taking steps to get the whole process going. So far, the improvements are pointing in the right direction so we have decided to continue the project.

The complexity in this takes its toll on scrum masters, product owners, and traditional project managers, and on the projects themselves. Tools and processes must be in place supporting the project teams. Obviously, both time and cost can be severely negatively affected by this fact. If we do not catch requirements changes, fulfillment of project scope (or the perceived added value of the project) will most likely suffer as well, making the project one of the challenged, or in worst cases abandoned, in the statistics.

Synchronization of Tools

Numerous times we have seen situations where a developer (or other team member) must use several tools to get the job done. This poses a problem for developers especially if they work in team(s). A single developer might not have these problems. There is one tool for writing code, one for bug reporting, one for testing, one for version control, one for reports and statistics, and so on. We are sure you recognize this as well. The coordinating effort to keep all information in sync between these systems is immense. Not to mention the difficulties of implementing a development process in all of them, if this is even possible in all systems.

Resource Management

What about Project Portfolio Management, or PPM, as it is also known (see Figure 1-2)? Keeping track of all running projects and their resources can be a considerable problem for enterprises. The investments in applications and projects are enormous, whether from a financial perspective or from a human capital perspective. PPM helps organizations balance the costs and values of IT investments so they can achieve their business goals.[2]

Figure 1-2.  ALM and PPM

Forrester says, “PPM provides a fact-based process for evaluating, prioritizing, and monitoring projects. PPM unites the process of strategic planning, resource and budget allocation, project selection and implementation, and post-project metrics.”[3]

This basically says it all about what issues are covered by PPM.

We can also see that a great portion of IT investments are focused on custom application development. If we cannot manage the resources we have at our disposal, the projects will most definitely suffer. We need to know, for example, that Steve will be available at the time he is needed in our project according to our project plan. If he is not, the schedule might have to change and the project most likely will be affected by both time and cost increases. To make matters worse, tasks depending on Steve’s work might suffer as well. This issue is one of our customers’ top priorities now. A lot of the questions we get when speaking about TFS implementations concern resource management integration with TFS.

Project Size

Project size could also affect the outcome of the projects. This is perhaps no surprise, because complexity usually increases when project size increases. It is hard to manage a project that has many people involved or a long timeline. If you combine a large project size with geographically spread project teams or members, keeping it all from falling apart becomes harder, and it will be harder to foresee everything that can happen. Of course all software development is difficult and involves risks, but these factors tend to make it harder and riskier.

When discussing this topic with coworkers, many have views and opinions, but not that many can reference research directly. They seem to argue on a gut feeling. So let’s take a look at what the research indicates.

Project Success in Research

This section covers some of the research on project success over the years. You will see a well-known report from the Standish Group as well as some disagreement with this report. You will also see what the Swedish IDG has found and some research from ACM.

The Standish Group

The Standish Group performs a survey on a regular basis on the performance of IT projects in the United States and Europe. The first report in 1994 was quite famous. It showed that many IT projects were cancelled or severely challenged. Since then, the Standish Group has performed the survey several times.

In 2009 the figures looked like this[4]:

- 44 percent of projects were challenged (53 percent in 1994).

- 24 percent of projects failed (31 percent in 1994).

- 32 percent were successful (16 percent in 1994).

Figure 1-3 presents these figures in graph form.

Figure 1-3.  The Standish report from 2009 shows figures from 1994 and forward

The figures have improved a little over the years, but still many projects seem to be unsuccessful in some way. In 2009 the results showed a marked decrease in project success rates, with 32 percent of all projects succeeding, which means delivered on time, on budget, and with required features and functions. Projects challenged were 44 percent which means they were late, over budget, and/or with less than the required features and functions and 24% failed, which means cancelled prior to completion or delivered and never used (http://www.standishgroup.com/newsroom/chaos_2009.php). These values have improved in the 2011 report as well. But to lump failed and challenged IT projects into the same bucket is not quite correct. Just because a project is challenged does not mean it has not added value to the company. A project might be late or overrun its budget, but still deliver great value to the company which makes it a well-received project anyway. Keep this in mind when you see figures like the preceding ones mentioned. A little perspective on the figures does not hurt.

Before we leave the Standish report, let’s look at what it says about the reasons for project success. These are interesting no matter whether you believe in the actual success figures of projects. Here are the Standish Group’s top ten reasons for project success[5]:

- User involvement

- Executive management support

- Clear business objectives

- Optimizing scope

- Agile process

- Project manager expertise

- Financial management

- Skilled resources

- Formal methodology

- Standard tools and infrastructure

These are interesting reasons. Keep these reasons in mind when you read about TFS and ALM later in the book. We will also come back to some of them later in this chapter.

Robert C. Glass

The figures from the Standish Group have been challenged by other researchers. Robert C. Glass wrote an interesting article that questions where the data of the Standish report really comes from.[6] He also questions the methods used by the Standish Group. Glass asks us to stand back and ask ourselves two things:

- Does the report represent reality?

- Is it supported by research findings?

Glass asks these questions because many other academic studies and guru reports in this area reference the Standish report from 1994. However, these other studies do not present much new material to support the old findings, according to Glass. Another problem is that the Standish Group does not describe its research process or indicate where their data came from so that we can discuss its validity. This is of course a huge problem.

Size and Volatility Survey

We want to address another survey before moving on. This survey claims that 67 percent of all projects are delivered close to budget, schedule, and scope expectations—quite the opposite of the preceding findings.[7] Of the 412 UK project managers in the study, they on average overshot budget by 13 percent, schedule by 20 percent, and underdelivered on scope by 7 percent. These figures are considerably lower than the Standish Group findings.

Table 1-1 shows the performance variance of the five types of projects defined in this study. Table 1-2 shows the size characteristics of these project types.

Table 1-1. Performance Variance

Table 1-2. Size Characteristics

These figures are interesting, but the following findings are even more so. What the following three figures show is that size matters. Figure 1-4 shows that the size of the project directly affects the outcome. The more effort in terms of person-months we have in a project, the greater are the chances of failure or low performance.

Figure 1-4.  Person-months

Figure 1-5 shows that the duration of the project also impacts performance.

Figure 1-5.  Project length

The longer the project, the greater the risk. Note that it seems like the greatest risk comes when the project duration reaches over 18 months.

Figure 1-6 illustrates the risk of project failure based on the size of the team.

Figure 1-6.  Team size

The risk seems to be the greatest when we have a team size of more than 20 people. Below that, we see that risk is pretty much the same. Compare this to the writings of Fred Brooks and his mythical man-month concept. In his book The Mythical Man-Month: Essays on Software Engineering (Addison-Wesley,1975) the central theme is that “adding manpower to a late software project makes it later” and you see that team size can pose a problem.

To be counted as a low performer, the following three project categories applied:

- Abandoned

- Budget challenged

- Schedule challenged

The size and volatility survey further showed that changes to project targets affected the success rate as well. The more changes, the bigger the chance of being a low performer.

Conclusions

So what do all these figures tell us? Well, we can clearly see that projects are not carried out in the most optimal way. There are still too many challenged and abandoned projects in this day and age. No matter which survey we choose to believe, the figures are worrisome.

If our ALM process is flawed, most of our projects will suffer. The reason why we should take control of the ALM process is that we can deliver better projects; having an overall process helps us. And with this process comes a mindset focused on the application from its birth as a business need to the delivered business value.

Measuring project success or failure is complicated. We need to take this into consideration when we read surveys stating that this or that many projects succeed.

The importance of an overall ALM process is that it can help us control the project outcome better in the end, enabling us to deliver true business value.

The work-down paradigm mentioned earlier has a widely accepted iron triangle showing the relationship between time, resources, and functionality (see Figure 1-7).[8]

Figure 1-7.  The iron triangle

In this we can imagine quality as a fourth dimension, giving us a tetrahedron. In this model, the relationship between the vertices of the tetrahedron is fixed. If you stretch one, at least one other needs to be stretched. If more resources are needed, for example, you might need to stretch time as well, and so on. With complex projects, the scope usually changes unpredictably during the process and suddenly we might need to add more resources, which obviously affect the schedule.

We need to reflect on the results of the surveys before we take them as the truth. This does not mean that the surveys we have referred to are without value. They definitely tell us something is wrong with the way we perform IT projects today. Why do so many IT projects fail? Have we not learned anything from the past years? The IT industry has been around for quite some time now, and we should have learned something along the way, shouldn’t we?

Could it be because the problems we need to solve just keep getting harder and harder? Or is it so hard to estimate and plan for projects that we can only take an educated guess at the beginning of a project? If the latter is true, why do we still measure a project’s success based on time, money, and requirements fulfillment? Maybe we should just shift our focus to business value instead? If the business value is greater than the cost of implementing the solution, time and money (cost) for the project are usually of less importance.

IT Budget Spending

In many companies that we have seen, the better part of the IT budget is usually spent on operations and maintenance and not on new development. The implications of this are that organizations have fewer possibilities to enhance their IT systems, and instead just spend the budget on keeping the systems alive. This could be a problem because the IT budget is needed to develop systems for meeting changes in the business processes.

Development vs. Operations

In 2006, the Corporate Executive Board (CEB) published the results of a survey concerning IT spending.[9] The survey found that of all IT spending in 2006, 30 percent related to IT system development. That might sound like a big part of the cake, but consider where the rest of the money is spent: 70 percent of an IT budget is spent on operations and maintenance (Figure 1-8).

Figure 1-8.  IT spending in companies

These figures are confirmed time after time in my discussions with customers. One of my coworkers told me that when he worked as an IT manager at a large Swedish car manufacturer, the figures applied as well.

Is it really rational to spend 70 percent on operations and maintenance? Wouldn’t it be more interesting to try to switch around these figures? Imagine the possibilities of adding value to an organization with so much more money to spend on IT systems. And think about the cool new features we could try out when we might have a larger budget for testing new technologies. I say we can switch around these figures (see Figure 1-9) without increasing the total IT budget or lowering the quality of operations and maintenance. This is definitely not an impossible task, which we hope this book will show.

Figure 1-9.  Making the switch

Before we get into a discussion about how to take control of our ALM process, let’s look at some of the factors influencing this split of IT money.

Factors Influencing IT Spending

During a recent visit to one of the biggest mail order companies in Northern Europe we learned a few things that are not so uncommon in many companies. They had managed to change their IT budget spending to 30 percent operations and 70 percent development. When asked what one of the most important solutions they had used to accomplish this was, the IT manager said that they had just simply stopped making small changes or fixes to the existing systems. The cost involved in such changes was far too great, so only very important changes were allowed to be implemented. With greater traceability and automated unit testing, he said, they could have continued lowering operation costs.

One other cost driver for maintenance and operations is the retirement of an application or system. Raise your hand, everyone, who has actually planned for this event (okay, you in the back, you can take your hand down now). If this scenario is not planned for in the beginning of an application’s lifecycle, great surprises can occur in its end. One example is from my friend, the car manufacturer. At times his business had to retire applications because the specific platform running the application became obsolete or a new version of the application was developed. Support for the platform might end because of newer platforms replacing it. On some occasions, my friend found that there was no way to migrate the historical data in the application(s). This scenario had not been planned for earlier and suddenly posed a great problem. The car company then had to negotiate an extended support contract with the platform vendor, which was way more expensive than it had been when the application was alive. It was quite ironic that they had to pay more for having access to historical data than they had when they actually used the system(s). One of the solutions to begin turning around the figures was to start planning for application retirement early in the lifecycle. The ALM process was changed so that, for example, data migration was planned for, making it unnecessary to keep applications slow cooking at a great cost.

We cannot cover all cost factors in this book, but we will mention some more key issues here. A big element causing increased operation costs is the way we have traditionally built our systems (and still do build them). Some years ago, client-server solutions dominated much of our IT environment. After that came multitier applications. These gave us great opportunities in writing business systems, with scalability and availability in mind, not only as standard Windows applications but also as web applications. Joachim Rossberg and Rickard Redler even wrote two books on this topic, Designing Scalable .NET Applications (Apress, 2003) and Pro Scalable .NET 2.0 Application Designs (Apress, 2005).

If we have only a few applications, this architecture works fine, but when new applications are added, it gets complex (see Figure 1-10).

Figure 1-10.  A traditional multitier application design. When we have several applications trying to access each other’s functions, problems may arise

Imagine what happens when a new application needs to access data from another. There are ways to solve this, as section 2 of this figure shows. We simply let the data layer of the new application access the business interface layer of the first. This way we can reuse the logic that already existed and not spread around database security and rights management. If we have only a few applications, this works fine, but as you can see in section 3, things can get pretty complicated when only a few more new applications are introduced. Most large enterprises have perhaps hundreds of applications spread across the company’s different locations. If there is no documentation task force in the company, there really is no way to have control over where the business logic is located.

Having an inflexible architecture is a great cost driver in maintenance (and of course in new projects). Imagine the nightmare of implementing a change request or bug fix in this environment. There really is no way to have control over where a change or fix will have its impact, so just a small change will need extensive testing on much more than just where the fix was implemented. This makes the cost of it much greater than it should need to be.

One way to avoid this is to have better ways of documenting traceability from the original requirement to accepted and delivered code. With the right tool(s) and the right work process, it would be much easier to find where a change or fix will have its effect, and testing could be minimized.

Another way to make the testing easier and hence less costly is to have good unit tests ready for the code and ways of running them automatically. This way, it is easier to see whether a change or a fix affects any of the code without manually testing everything else.

Another problem we have often experienced with clients is that a company has a great (and costly) infrastructure in place with redundancy for most applications and databases. Availability is high, and everyone should be happy because it’s really great to have such an environment in place. But think about it this way: is it really necessary for all applications to have such an infrastructure? Consider this: does a low-priority application not requiring 24/7 support or 99.999 percent availability need to run on such an expensive infrastructure? Isn’t it more cost-effective to run those applications on a different platform and spend the money on the systems that really need it?

Furthermore, you could cut costs if you make considerations for the time the system is expected to live. Carefully consider the infrastructure needed for all new applications. Are the requirements the same for the system with a lifespan of two years as it is for one with ten? It could be, of course, based on its business impact, but a system should not be routinely implemented on a specific platform just because you always do it like that.

How about the company operations process? Is one in place or is ad hoc work done? We are talking Information Technology Infrastructure Library (ITIL) or Microsoft Operations Framework (MOF) here. It’s just as important to have a good development process in place as it is to have an operations process. What happens when change requests come? When bug reports are entered? When new releases are introduced? Having a well-defined process in place could be a major factor in cutting operations costs. We will discuss this a bit more in Chapters 6 and 7 because it is an important topic.

With better traceability in our systems in place and tool(s) supporting it, we are certain that we could have a situation where we would not have to stop implementing small changes or features just to cut costs. If we can trace exactly where an update or bug fix will affect the system, we can make sure not to break anything else.

If we also consider our system architecture, we could create a better structured environment where we could more easily see how changes affect the system as well. Converting slowly but steadily to a SOA would be a good start.

Before we leave this chapter, we want to discuss the gap between IT and the business side again. If the business processes and the IT systems are not well aligned, there will be problems when processes change. This is a little bit like what we saw earlier with the application mess we can end up in. It is hard to say where we need to make changes to our systems when new processes need to be implemented or old ones change, if we do not know the structure of our systems. A lot of the costs involved in this process unavoidably end up in the operations budget.

Summary

An alarmingly large portion of IT projects delivered today are challenged or, in the worst case, abandoned. Many projects have overruns in terms of cost and time. Projects are also criticized for not delivering business value, and hence are not what the customer expects them to be in the end. One of the risks is definitely the lack of integration between the business side and the IT side. This gap makes it harder to deliver what we should deliver in our project, which is business value. Having a development process that is ill-defined or not even used is another great risk. Furthermore, the lack of mature ALM tools makes it harder to deliver as well, especially because we have more geographically dispersed development or project teams these days. Considering the amount of money spent on these projects, we can be certain lots of it is thrown away because of these challenges.

We can also be certain that much of our companies’ IT budgets go into operations and maintenance, giving less money to develop better and more-efficient systems that give more value to the business. To the technology-interested, this means less money and opportunity to try new cool techniques or tools, less testing of new technology, and so on. For the business, it means less opportunity to earn money and add value to the company. Either way you choose to see it, this model for IT budget spending is definitely a great problem.

The problems addressed in this chapter can be greatly improved by having control of our whole ALM process. This process, as you will see in the next chapter, focuses on taking the business needs and requirements, and turning them into business value for the organization. ALM does so by enforcing a process on how we work when developing software.

This book will show you how, with the use of a set of tools, Team Foundation Server (TFS), we can take control of the ALM process. The result will be that we can reduce the impact of the problems presented in this chapter. We might not get all the way with the current versions of TFS, but we can certainly come a long way.

Now it’s time to look more closely at the ALM concept in itself. Chapter 2 will cover most aspects of the ALM concept and why it is more than the software development lifecycle only.

[1] Software Engineering Institute, “What Is CMMI,” http://www.sei.cmu.edu/cmmi/index.cfm.

[2] Kelly A. Shaw, “Application Lifecycle Management and PPM,” June 2007, www.serena.com.

[3] Craig Symons with Bobby Cameron, Laurie Orlov, Lauren Sessions, Forrester Research, “How IT Must Shape and Manage Demand,” June 2006, www.forrester.com/Research/Document/Excerpt/ 0,7211,39660,00.html.

[4] The Standish Group International, “Chaos Summary 2008.”

[5] Deborah Hartmann, “Interview: Jim Johnson of the Standish Group,” 2006, www.infoq.com/ articles/Interview-Johnson-Standish-CHAOS.

[6] Robert C. Glass, “The Standish Report: Does It Really Describe a Software Crisis?” August 2006, Communications of the ACM.

[7] Chris Sauer, Andrew Gemino, and Blaize Horner Reich, “ The Impact of Size and Volatility on IT Project Performance,” November 2007, Communications of the ACM.

[8] Sam Guckenheimer and Neno Loje, Agile Software Engineering with Visual Studio: From Concept to Continuous Feedback 2nd Edition (Addison-Wesley Professional, 2011).

[9] Corporate Executive Board, “Application Budget, Staff, and Portfolio Benchmarks,” 2006.

CHAPTER 2

Introduction to Application Lifecycle Management

What do you think about when you hear the term Application Lifecycle Management, or ALM as it is usually referred to? During a seminar tour in 2005 in Sweden presenting Visual Studio Team System, we asked people what ALM was and whether they cared about it. To our surprise, many people equated ALM with operations and maintenance. This is still often the case today when we visit companies.

Maybe that was your answer as well? Doesn’t ALM include more than just operations? Yes, it does. ALM is the thread that ties together the development lifecycle. It involves all the steps necessary to coordinate the development lifecycle activities. Operations are just one part of the ALM process.

Roles in the ALM Process

All software development includes various steps performed by people playing specific roles in the process. There are many different roles or perhaps we could call them disciplines in the ALM process, and I define some of them in this section. (Please note that the process could include more roles, but I have tried to focus on the main ones.) Take a look at Figure 2-1, which illustrates ALM and some of its roles.

Figure 2-1.  The Application Lifecycle Management process and some of the roles in it

It is essential to understand that all business software development is a team effort. The roles collaborate on projects in order to deliver business value to the organization. If we don’t have this collaboration, the value of the system most likely will be considerably lower than it could be. If we look at it one step up from the actual project level, it is also important to have collaboration between all roles involved in the ALM process, so that we perform this process in the most optimal way possible.

The roles in the ALM process include the following:

- Business manager: Somebody has to make the decision that a development activity is going to start. After initial analysis of the business needs, a business manager decides to initiate a project for the development of an application or system that will deliver the expected business value. A business manager, for instance, will have to be involved in the approval process for the new suggested project, including portfolio rationalization, before a decision to go ahead is reached. Other people involved in this process are of course IT managers, because the IT staff will probably be involved in the project’s development and deployment into the infrastructure.

- Project manager, Product owner or scrum master: Suitable individuals are selected to fill these roles and set to work on the project after the decision to go ahead is made. Ideally, these people continue leading the project all the way through, so that we have continuity in project management.

- Project Management Office (PMO) decision makers: These individuals are also involved in planning because a new project might very well change or expand the company’s portfolio.

- Business analyst: After requirements collection starts, the business analyst has much to do. A business analyst is responsible for analyzing the business needs and requirements of the stakeholders, to help identify business problems and propose solutions. Within the systems development lifecycle, the business analyst typically performs a collaborative function between the business side of an enterprise and the providers of services to the enterprise.

- Architect: The architect starts drawing the initial picture of the solution. I will not go into great detail here because Chapter 4 does that. But briefly, the architect draws the blueprint of the system, and the system designers or engineers use this blueprint. This blueprint includes the level of freedom necessary in the system. For instance scalability, hardware replacement, new user interfaces, and so on. All of these issues must be considered by the architect.

- User Experience (UX) design team: Hopefully UX design is a core deliverable and not something we leave to the developers to handle. UX design is sadly overlooked and should definitely have a more consideration. It is important to have close collaboration between the UX team (which could be just one person) and the development team. The best solution is obviously to have a UX expert in the development team all through the project, but that is sometimes not possible. The UX design is so important in making sure users can really perceive the value of the system. We can write the best business logic in the world, but if the UX is badly designed, the users will probably never think the system is any good.

- Database administrators (DBAs): Almost every business system or application uses a database in some way. The DBAs are the ones who can make our databases run like lightning with good up-time, so it is essential to use their expertise in any project involving a database. Be nice to them; they can give you lots of tips on how to make a smarter system.

- Developers: Developers, developers, developers as Microsoft CEO Steve Ballmer shouted in a famous video. And who can blame him? These are the people doing their magic to realize the system that we are building by using the architecture blueprint drawn from the requirements. Moreover, these are the people who have to modify or extend the code when change requests come in.

- Test: I would rather not see testing as a separate activity. Testing is something we should consider from the first time we write down a requirement, and continue doing during the whole process.

- Operations and maintenance staff: Here it is. When an application or system is finished, it is handed over to operations. The operations staff takes care of it until it retires, often with the help of the original developers who come in a do bug fixes and new upgrades. Don’t forget to involve these people early in the process, at the point when the initial architecture is considered, and keep them in the project until all is done. They can give great input as to what can and can’t be done in the company infrastructure. So operations is just one part, but an important one, of ALM.

All project efforts are done as collaborative work. No role can act separate from any of the others if we are to succeed with any project. It is essential for everybody in a project to have a collaborative mindset and to have the business value as the primary focus at every phase of the project.

If you are part of an agile project like a scrum project, you might have only three roles; product owner, scrum master and team members. This does not mean that roles described above do not apply, though! They are all essential in most projects; it’s just that in an agile project you may not be labeled a developer or an architect. Rather, you are there as a team member and as such you and your co-members share responsibility for the work you have committed to. We will go deeper into the agile world later in the book.

Four Ways of Looking at Application Lifecycle Management

Application Lifecycle Management is the glue that ties together all these roles and the activities they perform. Let’s consider four ways of looking at ALM (see Figure 2-2). We have chosen these four because we have seen this separation in so many of the organizations I have worked with or spoken to:

Figure 2-2.  The four ways of looking at ALM

- Software Development Lifecycle (SDLC) view: This is perhaps the most common way of looking at ALM because development has “owned” management of the application lifecycle for a long time. This could very well be an effect of the gap between the business side and the IT side in most organizations, and IT has taken the lead on this.

- Service management or operations view: Operations have also had (in our experience) an unfortunate separation from IT development. This has resulted in operations having their own view of ALM and the problems in this area.

- Application Portfolio Management (APM) view: Again maybe because of the gap between business and IT, I have seen many organizations with a portfolio ALM strategy in which IT development is only one small part. From a business view, the focus has been on how to handle the portfolio itself and not on the whole ALM process.

- Chief information officer (CIO) view (or the unified view): Fortunately, some organizations focus on the whole ALM process by including all three of the preceding views. This is the only way to take control over, and optimize, ALM. For a CIO, it is essential to have this view all the time; otherwise, things can easily get out of hand.

The SDLC View

Let’s take a look at ALM from an SDLC perspective first. In Figure 2-3, you can see the different phases of a typical development project. Keep in mind that this is just a simplified view for the sake of this discussion. We have also tried to fit in the different roles from the ALM process presented earlier.

Figure 2-3.  A simplified view of a typical develoment project

First, somebody comes up with an idea based on an analysis of the business needs: “Hey, wouldn’t it be great if we had a system that could help us do this (whatever the idea is)?” It could also be the other way around: the idea comes first, and the business value is evaluated based on the idea.

So an analysis or feasibility study is performed, costs are estimated, and hopefully a decision is made by IT and business management to start the project as an IT project. A project manager (PM) is selected to be responsible for the project and starts gathering requirements with the help of business analysts, PMO decision makers, and users or others affected. The PM also starts planning the project in as much detail as possible at this moment.

When that is done, the architect starts looking at how to realize the new system, and the initial design is chosen. The initial design is then evaluated and changed based on what happens in the project and what happens with requirements all through the project. After that, the development starts, including work performed by developers, user interface (UI) designers, and DBAs (and any other person not mentioned here but who is important for the project).

Testing is, at least for us, something done all along the way—from requirements specification to delivered code—so this not a separate box in Figure 2-2. After the system has gone through acceptance testing, it is delivered to operations for use in the organization. Of course it doesn’t end here. This cycle is most often repeated over and over again as new versions are rolled out and bug fixes implemented.

What ALM does in this development process is support the coordination of all development lifecycle activities from the preceding process through the following:

- Make sure we have processes that span these activities.

- Manage the relationships between development artifacts used or produced by these activities. (In other words, we talk about traceability here.)

- Reporting on progress of the development effort as a whole.

As you can see from this, ALM does not support a specific activity in itself. Its purpose is to keep all activities in sync. It does this just so we can focus on delivering systems that meet the needs and requirements of the business. By having an ALM process helping us synchronize our development activities, we can more easily see if any activity is underperforming and thus more easily take corrective actions.

The Service Management or Operations View

From a service management or operations view, we can look at ALM as in this quote from ITIL Application Management by the Office of Government Commerce in United Kingdom (TSO, 2002): ALM “focuses on the activities that are involved with the deployment, operation, support, and optimization of the application. The main objective is to ensure that the application, once built and deployed, can meet the service level that has been defined for it.”

When we see ALM from this perspective, it focuses on the life of an application or system in a production environment. If in the SDLC view the development lifecycle started with the decision to go ahead with the project, here it starts with deployment into the production environment. Once deployed, the application is operated by the operations crew. Bug fixes and change requests are handled by them and they also pat it on its back to make it feel good and run smoothly.

This is a quite healthy way of looking at ALM in our opinion, because we think that both development and operations are two pieces of ALM, cooperating in order to manage the whole ALM process. Both pieces are also something that should be thought of when planning a development project from the beginning; we cannot have one without the other.

The Application Portfolio Management View

The third view we will look at is the Application Portfolio Management (APM) view of ALM. In this view, we see the application as a product managed as part of a portfolio of products. We can say that APM is a subset of Project Portfolio Management (PPM), which we talked about in Chapter 1. Figure 2-4 describes this process.

Figure 2-4.  The PMI view of ALM

This view comes from the Project Management Institute (PMI). Managing resources and the projects they work on is very important for any organization. In this view, we can see that the product lifecycle starts with a business plan—the product is the application or system that is one part of the business plan. An idea for an application is turned into a project and carried out through the project phases, until it is turned over to operations as a finished product.

When business requirements change or a new release (an upgrade in this figure) is required for some other reason, the project lifecycle starts again, and a new release is handed over to operations. After a while (maybe years) the system or application is discarded (this is called divestment, the opposite of investment). This view does not specifically speak about the operations part or the development part but should be seen in the light of APM instead.

The Unified View

The final view is a unified view of ALM. In this view, we have made an effort to align all these views with the business. Here we do as the CIO would do: we focus on the business needs, not on separate views. This we do to improve the capacity and agility of the project from start to end. Figure 2-5 shows an overview of how these three views are included in the whole unified ALM aspect of a business.

Figure 2-5.  The CIO’s view takes into consideration all three views previously mentioned

Three Pillars of Application Lifecycle Management

Let’s now look at some important pillars we find in ALM, independent of the view we take. We can find three important pillars in ALM as shown in Figure 2-6.

Figure 2-6.  The three pillars of ALM

Traceability of Relationships Between Artifacts

Some customers we have seen have stopped doing upgrades on his systems that were running in production because the company had poor or even no traceability in its systems. For these customers it was far too expensive to do upgrades because of the unexpected effects even a small change could have. The company had no way of knowing which original requirements were implemented where in the applications. This customer claimed, and we have seen and heard this in discussions with many other customers, that traceability can be a major cost driver in any enterprise if not done correctly.

There must be a way of tracing the requirements all the way to delivered code—through architect models, design models, build scripts, unit tests, test cases, and so on—not only to make it easier to go back into the system when implementing bug fixes, but also to demonstrate that the system has delivered the things the business wanted.

Another reason for traceability is internal as well as external compliance with rules and regulations. If we develop applications for the medical industry, for example, we need to have compliance with FDA regulations. We also need to have traceability when change requests are coming in so that we know where we updated the system and in which version we performed the update.

Automation of High-Level Processes

The next pillar is automation of high-level processes. All organizations have processes, as you saw in Chapter 1. For example, there are approval processes to control hand-offs between the analysis and design or build steps, or between deployment and testing. Much of this is done manually in many projects, and ALM stresses the importance of automating these tasks for a more effective and less time-consuming process. Having an automated process also decreases the error rate compared to handling the process manually.

Visibility into the Progress of Development Efforts

The third and last pillar is providing visibility into the progress of development efforts. Many managers and stakeholders have limited visibility into the progress of our development projects. The visibility they have often comes from steering group meetings, during which the project manager reviews the current situation. Some would argue that this limitation is good, but if we want to have an effective process, we must ensure visibility.

Other interest groups such as project members also have limited visibility of the whole project despite being part of the project. This often comes from the fact that reporting is hard to do and often involves a lot of manual work. Daily status reports would quite simply take too much time and effort to produce, especially when we have information in many repositories.

A Brief History of ALM Tools

We can resolve these three pillars manually if we want, without the use of tools or automation. ALM is not a new process description even though Microsoft, IBM, and the other big software houses right now are pushing ALM to drive sales of TFS or, in IBM’s case, the Collaborative Lifecycle Management suite. We can, for instance, continue to use Excel spreadsheets, or as one of my most dedicated agile colleagues does, use Post-it notes and a pad of paper, to track requirements through use cases/scenarios, test cases, code, build, and so on to delivered code. It works, but this process takes a lot of time and requires much manual effort. With constant pressure to keep costs down, we need to make tracking requirements more effective.

Of course, project members can simplify the process by keeping reporting to the bare minimum. With a good tool, or set of tools, we can cut time (and thus costs) and effort, and still get the required traceability we want in our projects. The same goes for reporting and all those other activities we have. Tools can, in my opinion, help us be more effective, and also help us automate much of the ALM process right into the tool(s).

By having the process built directly into our tools, it is much easier for the people involved to not miss any important step by simplifying anything. For instance, the agile friend we mentioned could definitely gain much from this, and he has now started looking into Team Foundation Server (TFS) to see how that set of tools can help him and his teams be more productive. So process automation and the use of tools to support and simplify our daily jobs are great things because they can keep us from making unnecessary mistakes.

There are eight disciplines in ALM according to Serena Software Inc (Application Lifecycle Management for the enterprise, Kelly A. Shaw, Ph.D,, Serena Software Inc, April 2007, http://www.serena.com/docs/repository/company/serena_alm_2.0_for_t.pdf):

- Modeling: software modeling

- Issue management: keeping track of the incoming issues during both development and operations

- Design: designing the system or application

- Construction: developing of the system or application

- Production Monitoring: the work of the operations staff

- Build: building the executable code

- Configuration and change management: keeping track of changes and configuration of our applications

- Test: testing the software

- Release management: planning the releases of our application

In order to synchronize these we need tools that span all these disciplines and help us automate and simplify the following activities according to Serena Software:

- Reporting

- Traceability

- Policies

- Procedures

- Processes

- Collaboration

Imagine the Herculean task of keeping all those things in order manually. That would be impossible if we wanted to get something right and keep an eye on the status of projects. As we saw from the Standish Group in Chapter 1, projects seem to be going better because the number of failed projects is decreasing. Much of this progress is, according to Michael Azoff at the Butler Group,[1] the result of some “major changes in software development: open source software projects; the Agile development movement; and advances in tooling, notably Application Lifecycle Management (ALM) tools.” Some of these results are also confirmed by later research, for instance by Scott W. Ambler at Ambysoft (http://www.ambysoft.com/surveys/success2011.html) and Alan Radding (http://resources.devx.com/ibm/Article/47065). Now we understand finding tools and development processes to help us in the ALM process is important.

This has led to increasing awareness of the ALM process among enterprises. We can see this among the customers we have. ALM is much more important now than it was only five years ago.

Application Lifecycle Management 1.0

As software has become more and more complex, we have seen that role specialization has increased in IT organizations. This has led to functional silos in different areas (roles) such as project management, business analysis, architecture, development, database administration, testing, and so on. As you may recall from the beginning of this chapter, you can see this in the ALM circle. There is no problem with having these silos in a company, but having them without any positive interaction between them is.

There is always a problem when we build great and impenetrable walls around us. Most ALM vendors have driven the wall construction because most of their tools historically have been developed for particular silos. If we look at build management tools, they have supported the build silo (naturally) but have little or no interaction with test and validation tools (which is kind of strange since the first thing that usually happens in a test cycle is the build)—just to mention one area. This occurs despite the fact that interaction between roles can generate obvious synergies that great potential. We need to synchronize the ALM process to make the role-centric processes a part of the overall process. This might sound obvious, but has just not happened until lately.

Instead of having complete integration between the roles or disciplines mentioned in the beginning of the chapter and the tools they use, we have had point-to-point integration—for example, we could have a development tool slightly integrated with the testing tool (or probably the other way around). Each tool uses its own data repository, so traceability and reporting is hard to handle in such an environment as well (see Figure 2-7).

Figure 2-7.  ALM 1.0

This point-to-point integration makes the ALM process fragile and quite expensive as well. Just imagine what happens when one tool is updated or replaced. Suddenly, the integration might break and new solutions have to be found to get it working again. This scenario can be reality if, for example, old functions in the updated or replaced tool are obsolete and the new one does not support backward compatibility. This can be hard to solve even with integration between just two tools. Imagine what happens if we have a more complex situation, including several more tools. We have seen projects using six or seven tools during development, creating a fragile solution when new versions have been released.

The tools have also been centered on one discipline. In real life, a project member working as a developer, for instance, quite often also acts as an architect or tester. Because the people in each of these disciplines have their own tool (or set of tools), the project member must use several tools and switch between them. It could also be that the task system is separated from the rest of the tools so to start working on a task, a developer must first retrieve the task from the task system—probably print it out, or copy and paste it, then open the requirements system to check the requirement, then look at the architecture in that system, and finally open the development tool to start working. Hopefully, the testing tools are integrated into the development tool; otherwise, yet another tool must be used. All this switching costs valuable time better put into solving the task.

Having multiple tools for each project member is obviously costly as well because they all need licenses for the tools they use. Even with open source tools that may be free of charge, we have maintenance costs, adaptions of the tools, developer costs, and so on. Maintenance can be very expensive, so we should not forget this even when the tools are free. So, such a scenario can be very costly and very complex. It will also most likely be fragile.

As an example, I have two co-workers working at a large medical company in Gothenburg. They have this mix of tools in their everyday work. I asked them to estimate how much time they needed to switch between tools and get information from one tool to another. They said they probably spend half an hour to an hour each day syncing their work. Most times they are on the lower end of this scale, but still in the long run this is a lot of time and money. My friends also experienced big problems whenever they needed to upgrade any of the systems they used.

One other problem with traditional ALM tools worth mentioning is that vendors often have added features, for example, adapting a test tool to support issue and defect management. And in the issue management system, some features might have been added to support testing. Because neither of the tools have enough features to support both disciplines, the users are confused and will not know which tool to use. In the end, most purchase both just to be safe, and end up with the integration issues described earlier.

Application Lifecycle Management 2.0

So let’s take a look at what the emerging tools and practices (including processes and methodologies) in ALM 2.0 try to do for us. ALM is a platform for the coordination and management of development activities, not a collection of lifecycle tools with locked-in and limited ALM features. Figure 2-8 and Table 2-1 summarize these efforts.

Figure 2-8.  ALM 2.0

Table 2-1. Characteristics in ALM 2.0

Characteristics

Benefit

Practitioner tools assembled out of plug-ins

Customers pay only for the features they need.

Practitioners find the features they need faster.

Common services available across practitioner tools

Easier for vendor to deploy enhancements to shared features.

Ensure correspondence of activities across practitioner tools.

Repository neutral

No need to migrate old assets.

Better support for cross-platform development.

Use of open integration standards

Easier for customers and partners to build deeper integrations with third-party tools.

Microprocesses and macroprocesses governed by externalized workflow

Processes are versionable assets.

Processes can share common components.

One of the first things we can see is a focus on plug-ins. This means that from one tool, we can add the features we need to perform the tasks we want. Without using several tools! If you have used Visual Studio, you have seen that it is quite straightforward to add new plug-ins into this development environment. Support for Windows Communication Foundation (WCF) and Windows Presentation Services, for example, was available as plug-ins long before their support was added as a part of Visual Studio 2008.

Having the plug-in option and making it easy for third-party vendors to write plug-ins for the tool greatly eases the integration problems discussed earlier. You can almost compare this to a smorgasbord, where you choose the things you want. So far this has mostly been adopted by development tool vendors such as IBM or Microsoft, but more are coming. Not much has happened outside of development tools so far but TFS offers some nice features that definitely will help us a lot.

Note  Smorgasbord is originally a Swedish term for a big buffet of food where the guests can pick and choose the dishes they want. In Sweden we have smorgasbord at Easter, Christmas, and other holidays.

Teamprise, a third-party vendor, developed a solution giving access to the Team Foundation Server (TFS) from a wide array of platforms, including Mac OS X (see Figure 2-9). In November 2009, Teamprise was acquired by Microsoft. After the acquisition TFS 2010 changed its name to Team Explorer Everywhere. In writing this book we used Team Explorer Everywhere on our Mac OSX laptops using the Eclipse development platform.

Figure 2-9.  Team Explorer Everywhere in Mac OS X

Another thing that eases development efforts is that vendors in ALM 2.0 are focusing more on identifying features common to multiple tools and integrating these into the ALM platform. We find things like the following among these:

- Collaboration

- Workflow

- Security

- Reporting and analysis

You will see later in Chapter 4 that TFS has these features embedded out of the box. Microsoft uses Microsoft Office SharePoint Server/Windows SharePoint Services as part of the TFS suite for collaboration among team members, for example. Active Directory is used for authentication and security. SQL Server obviously is the data repository giving us access to tools such as SQL Server Analysis Services for analytics and SQL Server Report Builder for reports. Microsoft has done a good job in giving us tools to get the job done.

Another goal in ALM 2.0 is that the tools should be repository neutral. They say that there should not be a single repository but many, so we are not required to use the storage solution that the vendor proposes. IBM, for example, has declared that their coming ALM solution will integrate with a wide variety of repositories, such as Concurrent Versions System (CVS) and Subversion, just to mention two. This approach removes the obstacle of gathering and synchronizing data, giving us easier access to progress reports, and so on. Microsoft uses an extensive set of web services and plug-ins to solve the same thing. They have one storage center (SQL Server), but by exposing functionality through the use of web services, they have made it fairly easy to connect to other tools as well.

An open and extensible ALM solution lets companies integrate their own choice of repository into the ALM tool. Both Microsoft and IBM have solutions—data warehouse adapters—that enable existing repositories to be tied into the ALM system. It is probable that a large organization that has already made investments in tools and repositories in the past doesn’t want to change everything for a new ALM system; hence it is essential to have this option.

Any way we choose to solve the problem will work, giving us possibilities of having a well-connected and synchronized ALM platform.

Furthermore, ALM 2.0 focuses on being built on an open integration standard. As you know, Microsoft exposes TFS functionality through web services. This is not publicly documented however so we need to do some research and trial and error before we can get this working. This way, we can support new tools as long as they also use an open standard; and third-party vendors have the option of writing tons of cool and productive tools for us.

Process support built in to the ALM platform is another important feature. By this I mean having the automated support for the ALM process built right into the tool(s). We can, for instance, have the development process (RUP, SCRUM, XP, and so on) automated in the tool, reminding us of each step in the process so that we don’t miss creating and maintaining any deliverables or checkpoints.

In the case of TFS, you will see that this support includes having the document structure, including templates for all documents, available on the project web site, directly after having created a new TFS project. We can also imagine a tool with built-in capabilities helping us in requirements gathering and specification, for instance—letting us add requirements and specs into the tool and have them transformed into tasks assigned to the correct role without having to do this manually.

An organization is not likely to scrap a way of working just because the new ALM tool says it cannot import that specific process. A lot of money has often been invested in developing a process, and an organization is not likely interested in spending the same amount again learning a new one. With ALM 2.0 it is possible to store the ALM process in a readable format such as XML.

The benefits include that the process can be easily modified, version controlled (you need to do this yourself, TFS doesn’t do it for you), and reported upon. The ALM platform can then import the process and execute the application development process descriptions in it. Microsoft uses XML to store the development process in TFS. In the process XML file the whole ALM process is described, and many different process files can coexist. This means we can choose which process template we want to base our project on when creating a new project (see more in Chapter 6).

As we saw earlier, it is important for an enterprise to have control over its project portfolio to better allocate and control resources. So far, none of the ALM vendors have integrated this support into the ALM platform. There may be good reasons for this though. For instance, while portfolio management may require data from ALM, the reverse is probably not the case.

The good thing is that having a standards-based platform makes integration with PPM tools a lot easier.

Application Lifecycle Management 2.0+

So far not all the ALM 2.0 features have been implemented by tools vendors. There are various reasons for this. One of these is the fact that it is not quite easy for any company to move to a single integrated suite, no matter how promising the benefits might look when you see them. To make such a switch would mean changing the way we work in our development processes and even in our company. Companies have invested in tools and practices and spending time and money on a new platform can require a lot more investment.

For Microsoft focused development organizations the switch might not be so hard however, at least not for the developers. They already use Visual Studio, SharePoint, and many other applications in their daily life and the switch is not that big. But Microsoft is not the only platform out there and competitors like IBM, Serena, and HP still have some work to do to convince the market.

We also can see that repository neutral standards and services have not evolved over time. Microsoft’s for instance still rely on SQL Server as a repository and have not built in much support for other databases or services. The same goes for most competition to TFS.

Note  Virtually all vendors will use ALM tools to lock in customers to as many of their products as possible—especially expensive major strategic products like RDBMS. After all they live mostly on license sales.

The growth of agile development and project management in recent years has also changed the way ALM must support development teams and organizations. We can see a clear change from requirements specs to backlog-driven work and the tooling we use needs to support this in a good way.

Agile practices such as build and test automation become critical for our ALM tools to support. Test Driven Development (TDD) continues to rise and more and more developers require their tools to support this way of working. If the tools don’t do that they will be of little use for an agile organization. Microsoft has really taken the agile way of working to their hearts in the development of TFS. We will show you all you need to know about the support for agile practices in TFS all through this book.

We can also see a move from traditional project management toward an agile view where the product owner and scrum master require support from the tools as well. Backlog grooming (the art of grooming our requirements in the agile world), agile estimation and planning, reporting,—important to these roles—need to be integrated to the overall ALM solution. We will come back to these concepts in Chapter 6.

The connection between operations and maintenance becomes more and more important. Our ALM tools should integrate with the tools used by these parts of the organization. Fortunately, Microsoft has plans for this. They plan to make the integration between MS Systems Operations Manager and TFS much simpler and built in out of the box, adding support for maintenance and operations.

In the report “The Time is right for ALM 2.0+” Forrester research presented the ALM 2.0+ concept as you can see in Figure 2-10. (Forrester Research, Inc., The Time Is Right For ALM 2.0+, October 19, 2010) In their report Forrester Research, Inc extended traditional ALM with what they called ALM 2.0+. Traditional ALM covers traceability, reporting, and process automation. Forrester Research, Inc envisions the future of ALM to also include collaboration and work planning. We like this idea and we will try to show you that Microsoft also has support for this in TFS.

Figure 2-10.  Future ALM according to Forrester Research, Inc (Forrester Research, Inc., The Time Is Right For ALM 2.0+, October 19, 2010)

ALM and PPM

ALM and PPM can support each other quite well. Data from the ALM repository can be an excellent source of data for the PPM tool, and hence decisions can be based on the results of the PPM tool. This requires a (one-way) working connection between the two, of course. Manual efforts by cutting and pasting information are not good enough because they are slow and error prone. A good integration between the two repositories gives project portfolio decision-makers access to accurate and timely data. This eases their decision-making process.

Gartner identifies five key activities in the PPM decision process that benefit from a working integration:[2]

- Review current projects and their status.

- Review the application portfolio impact on resources (which resources are available and when are they available, for instance).

- Present new capital requests. (Do we need to add more resources to a project?)

- Reprioritize the portfolio.

- Examine investments for effectiveness (basically reviewing the outcome of projects within six months of delivery).

A lot of data important for these activities can be found in the ALM repository.

Microsoft’s solution to the ALM and PPM integration still has some room for improvement but works very well.

Microsoft also offers the Microsoft Office Enterprise Project Management (EPM) solution (http://www.microsoft.com/project/en-us/solutions.aspx). This is Microsoft’s end-to-end collaborative project and portfolio environment. This solution aims to help an organization gain visibility, insight, and to control across all work, enhancing decision making, improving alignment with business strategy, maximizing resource utilization, and measuring and helping to increase operations efficiency. We will not delve in to the specifics of this solution here but will tell you a little about its three parts.

First, there is the Microsoft Office Project Professional. If you are a project manager, chances are that you know this product already. If not, many of us have definitely seen the Gantt schema the project manager has produced. We can use Project Professional as a stand-alone product if we need it for only single projects. But the real value comes when we connect it to the second part of the Microsoft EPM solution: the Microsoft Office Project Server. This server offers the possibilities of resource management, scheduling, reporting, and collaboration capabilities in the Microsoft EPM solution. We use the Project Server to store project and resource information in a central repository.

The third part of Microsoft’s EPM suite used to be the Microsoft Office Project Portfolio Server. Since version 2010 this is built in to Microsoft Project Server. This gives us a management solution, enabling organizations to get control of their product portfolios so that they best align with the company business strategies.

All in all this helps us to handle our portfolios, projects, and resources so that we can plan our needs.

Summary

In this chapter, we have described an overview of what ALM aims for and what it takes for the ALM platform to support a healthy ALM process. We have seen that there are four ways of looking at ALM:

- Software Development Lifecycle (SDLC) view

- Service management or operations view

- Application Portfolio Management (APM) view

- Chief information officer (CIO) view (or the unified view)

We know that traceability, automation of high-level processes, and visibility into development processes are three pillars of ALM. Other important key components are collaboration, workflow, security, reporting, analytics, being open standards based, being plug-in friendly, and much more, and they are focus areas of ALM. You have also seen that ALM is the coordination and synchronization of all development lifecycle activities.

A good ALM tool should help us implement and automate these pillars and components to deliver better business value to our company or organization.

Chapter 3 delves little bit deeper into different project management processes and frameworks. We will cover some history and also some more “modern” processes.

[1] Michael Azoff, “Application Lifecycle Management Making a Difference,” February 2007, Enterprise Networks and Services, OpinionWire.

[2] “Application Lifecycle Management and PPM,” Serena, 2006.

CHAPTER 3

Development Processes and Frameworks

Before we start looking at the actual products that make up Team Foundation Server (TFS) we want to discuss the development processes or perhaps development frameworks (as some call their process) a bit. Unfortunately this topic is pushed back when many companies implement TFS. Some seem to argue that one of the two process templates Microsoft ships with TFS is enough. These processes are automations of the development process in the overall ALM process previously discussed in Chapter 2. However, not even Microsoft suggests that we should use these without customization, so spend some time planning this before implementing TFS, and also keep in mind that you will probably need to make ongoing adjustments.

A software development process can be a structure imposed on the development of a software product. That is, a way of working that we should follow to successfully deliver an application or a system. We saw in Chapter 1 that the process was an important aspect of successful project completion, so this chapter to discuss this topic.

Throughout the years, many development processes have come and gone. If we look, they are probably around in the background. They all have tried to improve on the former version or have added a new aspect to development. The goal has many times been the same for them all, even though the road to that goal has varied. These days the agile movement has shown good results. This is discussed in depth later in this book.

One of the best known models has been around since 1970, when it was presented in an article by Winston W. Royce[1] What is interesting about the model is that Royce actually presented it as an example of a flawed, nonworking model. Obviously, people did not bother about this fact and started using it as a way to run development projects anyway. The model I am referring to is of course The Waterfall model, even though Royce did not call it that.

The Waterfall Model

The Waterfall model is a sequential process through which development is seen as a flow steadily moving downward, just like a waterfall, through its different phases. Figure 3-1 shows the different phases of the model.

Figure 3-1.  The Waterfall development processes

Royce wrote about seven phases in his original article:

- Requirements specification

- Design

- Construction (a.k.a. implementation)

- Integration

- Testing and debugging (a.k.a. validation)

- Installation

- Maintenance

As seen Figure 3-1, we usually speak of only five of these phases, because the model has evolved over the years. The thought is that the phases are carried out sequentially and we never go back to what has been done. So when requirements specifications, for example, are done, they are virtually written in stone. After the spec is written we move on to the next step, in this case the design phase, where we model our system and lay out the architecture. This effort can be seen as the blueprint of the system. In this phase we transform the requirements into a design we can give to the developers to realize into code.

When the blueprint is ready we can move on to the next step, which is implementation. Now the coders do their magic and the blueprint is transferred into code. At the end of this phase, individual software components are integrated and tested as a system. Different teams of developers might have developed the components, perhaps at different locations, which complicate things as well, because communication tends to be limited in such cases. As you can understand there is an inherent problem here. If we test the system only after development is done (perhaps 12 months after coding began), we might end up with lots of surprises. Just consider the immense rework needed if something is wrong at this point. Many months of work might be going down the drain and the project is surely seen as a failure.

When the implementation phase is done, we move on to testing the software. Hopefully, faults from earlier phases are found and removed in this part of the process. This might actually be the first time our customer or stakeholders see the final product. If more than a year has passed since the project started, much may have happened to the requirements, and because we cannot go back to earlier phases, we are stuck with requirements that are actually wrong. When testing is complete, we install the software and deliver it to maintenance.

What is important to remember here is that we do not move to the next phase until the former is completely done and finished. There is no jumping back and forth between them and they cannot overlap.

The Waterfall model has been widely accepted and is used a lot —especially in the public sector such as at the U.S. Department of defense, NASA, and many other large government projects. This has been loosened a bit lately (luckily) and more agile methods like SCRUM are being implemented at these organizations as well.

As we can see, the Waterfall method could be great when we know that nothing much will change during our project. Let’s say that we are about to build a road. After gathering all requirements for the road we can assume that they will not change much during the process. The same goes for the blueprints. Sure, some small things might change, such as placement of road signs and streetlights for instance, but on the whole it is pretty solid after it is approved. When we have such a project, the Waterfall model works very well. If we transform the road example to a development project, we can say that implementing a standard product, like an economy system, might be very static once the requirements are set and the model could work very well then. But even with such development efforts, things change a lot anyway. According to Ken Schwaber of the Agile Alliance, and co-father of Scrum, about 35 percent of all requirements in a project change, which is a very high number and hence provides risk for the project.

Generally, one could say that the earlier we can find bugs, the easier and less expensive they are to fix. McConnell estimates that “a requirements defect that is left undetected until construction or maintenance will cost 50 to 200 times as much to fix as it would have cost to fix at requirements time.”[2] This is the reason why all phases in the Waterfall model must be 100 percent complete and absolutely correct before we move on to the next phase. The aim is to catch errors early to avoid problems and costs in the end.

Another cornerstone is documentation. Great focus is spent on documenting work. We need design documents, source code documents, and so on. The reason for this is that we should avoid problems if one of our team members falls off and nobody knows what he or she has been doing. Much knowledge could be lost unless we have good documentation. If one person disappears, it should be relatively easy for the replacement to familiarize himself with the system and quickly become productive.

These are pretty solid arguments for the model, at least at first sight. But as we saw in Chapter 1, most development projects are more complex than implementing a standard system. This means that it is almost impossible to get one phase perfect before moving on to the next. Just to get all requirements correct is a tremendous task because the user/stakeholder probably won’t be aware of exactly what they want unless they have a working prototype to investigate. Then, and only then, can they truly comment on it. It is also then that they will get a feeling of what is possible to accomplish with the software. If this awareness occurs late in the project the changes to requirements they would want are hard to implement.

Another problem that we often run into is that during design the architect or designers cannot know all the difficulties that could happen during implementation. Some areas can be very hard to build or integrate that we were not aware of earlier. How do we handle that if we cannot go back to the design and change it once we have left it? Some requirements might also be contradictory, but this might only show during implementation. This will obviously be hard to solve as well without changing work done in earlier phases.

It seems like not even Winston Royce really believed in the Waterfall model as we saw earlier. Instead he was writing his paper about how to change this model into an iterative one, with feedback from each phase affecting subsequent phases. Strange thing that this fact has been virtually ignored and that the Waterfall model has been given so much attention through the years.

Spiral Model

Barry Boehm defined the Spiral model in his 1988 article.[3] Although not the first model to discuss iterative development, it was in fact the first model to explain why the iteration matters.

Originally, the iterations were typically six months to two years long. Each phase starts with a design goal and ends with the customer or stakeholder reviewing the progress so far. At each phase of a Spiral project, analysis and engineering efforts are applied, with a focus on the end goal of the project, the business value.

The steps in the Spiral model can be described as follows (see Figure 3-2). Remember this is a simplified view:

Figure 3-2.  The Spiral model

1. The requirements are defined in as much detail as possible. Interviews of a number of users representing all the external or internal users and other aspects of the existing system are often used at this stage.

2. A preliminary design is created for the new system.

3. Based on the preliminary design a first prototype of the new system is created. Often this is a scaled-down system, showing an approximation of the characteristics of the final product.

4. A second prototype is evolved by a four-step procedure:

a.  Evaluating the first prototype in terms of its strengths, weaknesses, and risks

b.  Defining the requirements of the second prototype

c.  Planning and designing the second prototype

d.  Constructing and testing the second prototype

If the customer thinks the risks are too great the project can be aborted. Risk factors can be development cost overruns, miscalculation of operating-cost or any other factor that could, in the customer’s judgment, result in a less-than-satisfactory final product.

- 5.  The existing prototype is evaluated in the same manner as was the first prototype, and, if necessary, another prototype is developed from it according to the four-step procedure outlined previously.

- 6.  These steps are iterated until the stakeholder is satisfied and convinced that the refined prototype represents the final product he or she wanted.

- 7.  The final system is constructed, based on the approved prototype.

- 8.  The final system is extensively evaluated and tested. Routine maintenance is carried out on a continuing basis to prevent large-scale failures and to minimize downtime.

It seems like the Spiral model is used on larger projects mostly, but I have never been a part of this kind of project, and neither has any of my co-workers, so it’s hard to say if it scales that well.

Rational Unified Process (RUP)

During the 1980s a team at Rational Software began looking into a new development model. Rational Unified Process (RUP) as such, is said to have been created in 1996 when Rational acquired the Objectory Process, written by Ivar Jacobson. They started this work by going back to the Spiral model created by Barry Boehm in 1988 and then started to look into why software projects failed. What was the root cause of this in the past? Furthermore, they took a good look at which software processes were around at the time and how each of them tried to solve these causes of failure. Some of the reasons they found were:

- Ad hoc requirements management

- Complexity

- Ambiguous and imprecise communications

- Undetected inconsistencies in requirements, designs, and implementations

- Insufficient testing

- Subjective assessment of project status

- Uncontrolled change propagation

- Poor or insufficient automation

The people at Rational found that project failure most often was caused by a combination of several symptoms. They also concluded that every project that fails does so in its own unique way. After analyzing their results, the team designed a collection of software best practices, which they named the Rational Unified Process (RUP).

What is important to remember is that RUP is not a single, concrete prescriptive process. It is an adaptable process framework intended to be adjusted by the organization and software team that will use it. The project team should choose the parts of the process that are appropriate for the needs of that specific development task at hand.

The Principles of RUP

The Rational team based their framework on six key principles for business driven development.[4]

- Adapt the process. The project or organization must, as we saw previously, select the parts they need from the RUP framework. Things to consider here are, for example, how project governance, project size, regulations, and such issues affect the degree of formality that should be used. There are preconfigured process templates for small, medium, and large projects in RUP so that we can choose more easily. Most companies that I have seen usually adapt RUP in their own way. One of my former employers had several different RUP adaptations based on different project types.

- Balance stakeholder priorities. RUP tries to take a shot at balancing out the business goals and stakeholder needs between the parties involved, because these often differ and are conflicting.

- Collaborate across teams. Hopefully as we all know, software engineering is a team process. We have various participants in a project, from stakeholders to developers. In Chapter 1 we saw that much development these days does not happen at one location, but could actually be geographically dispersed all over the world. This means that collaboration and communication between participants must be good—not only for requirements issues but also for all aspects of the development process. Project status, test results, bug status, release management, design and architecture diagrams, and much more must be at hand for those who need it, as well as at the time they need it.

- Demonstrate value iteratively. One problem with the waterfall model is that it does not allow us to go back a phase if we find things in one phase that throws things in earlier phases to the ground. By working iteratively we deliver working software in increments. For each iteration we collect feedback from everybody, including stakeholders, and use this as an input to the next iteration. This way we can influence the development process and hence the business value while the project is executed. By focusing strongly on iterative development and good risk management, RUP allows projects an iterative risk assessment process that is intended to ease the effort in delivering a successful project in the end.

- Elevate the level of abstraction. By elevating the abstraction level, RUP encourages the use of software patterns, 4GL, frameworks, reusable components, and so on. This approach hinders developers from going directly from the spec to writing their own custom-made code. This also means that we discuss architecture at a higher level than before. By using UML (Unified Modeling Language) or some built-in features of the development tool (see Chapter 6 for Visual Studio’s architecture tools) in conjunction with a higher abstraction level, we elevate product architecture to a level where nontechnical stakeholders can better participate.

- Focus continuously on quality. Surprisingly enough we do not focus on quality enough during many projects. I have had contractors at the Swedish Road Administration that didn’t have this in focus in their projects. Instead, their primary goal was to suck as much money as possible from the SRA (and from me as a taxpayer). This only caused problems, as you would guess, because if the SRA did not keep an extra eye open, the projects were unsuccessful. RUP encourages continuous quality checks through development. Automation of test scenarios for example helps us deal with an increasing amount of tests caused by the iterative process and the practice of test-driven development.

The attentive reader (yes, I mean you!) has already noticed that if we take the starting letter from each of these principles we get the ABCs of RUP:

- Adapt the process

- Balance stakeholder priorities

- Collaborate across teams

- Demonstrate value iteratively

- Elevate the level of abstraction

- Focus continuously on quality

The RUP Lifecycle

So, what does the RUP lifecycle look like? There are four major phases. And no, we do not talk waterfall here (see Figure 3-3). The phases are

- Inception

- Elaboration

- Construction

- Transition

Figure 3-3.  The RUP development processes

Inception Phase

As Figure 3-3 shows, the Inception phase has a strong focus on business modeling and requirements specifications. The difference from the Waterfall model is that we do not close these topics after the phase has ended. Instead, they are a constant part of all phases through the project until its end. This phase establishes a baseline so that we can compare actual expenditures to planned expenditures along the project. Before we move on to the next phase we need to pass a milestone called the Lifecycle Objective Milestone (see Figure 3-4).

Figure 3-4.  The Lifecycle Objective Milestone

To pass this milestone we need to meet these criteria:

- Stakeholder concurrence on scope definition and cost/schedule estimates.

- Agreement that the right set of requirements has been captured and that there is a shared understanding of these requirements.

- Agreement that the cost/schedule estimates, priorities, risks, and development process are appropriate.

- All risks have been identified and a mitigation strategy exists for each.

If we are not satisfied with the outcome of this milestone or the phase, we can choose to cancel or report this phase for redesign.

Elaboration Phase

During the Elaboration phase we start to see what the project will look like. In Figure 3-3 you see that analysis and design has its biggest effort here but that it will be required to continue through the other phases. There are also other activities we perform in this phase. We start to think about the implementation, how the code will be written, what to code, and so on. The thought is that most use cases are developed during elaboration, where actors are identified and the flow of the use cases are thought out.

To pass the Lifecycle Architecture Milestone that finishes the Elaboration phase (see Figure 3-5) we should have completed 80 percent of the use case models.

Figure 3-5.  The Lifecycle Architecture Milestone

We should also have created a description of the architecture of our software. The risk list should have been written, as well as a development plan for the entire project. These are the main criteria for passing the Lifecycle Architecture Milestone:

- Is the vision of the product stable?

- Is the architecture stable?

- Does the executable demonstration show that the major risk elements have been addressed and credibly resolved?

- Is the plan for the construction phase sufficiently detailed and accurate? Is it backed up with a credible basis of estimates?

- Do all stakeholders agree that the current vision can be achieved if the current plan is executed to develop the complete system, in the context of the current architecture?

- Is the actual resource expenditure versus planned expenditure acceptable?

There are a few more criteria we must meet before we can pass this milestone, but we will not go into them here. If we cannot pass the milestone, we can either cancel or redesign, just like in the preceding phase. When we continue to the next phase, project changes are more difficult to solve if we do not have a model that covers such events.

Construction Phase

Now we are ready for the Construction phase. This is where the coding starts and when we will implement our architecture. To make sure we catch changes of requirements we do the development in iterations, each delivering a working prototype. We can show this to stakeholders and end-users so that they have a chance to provide feedback on it. When going in to this phase the use cases have been prioritized and divided across the iteration. One good practice is to focus on the highest risk use cases first, or at least as early as possible, so that we can catch their implications early. To end this phase, we must pass the Initial Operational Capability Milestone (see Figure 3-6) by answering the following questions:

- Is this product release stable and mature enough to be deployed in the user community?

- Are all stakeholders ready for the transition into the user community?

- Are the actual resource expenditures versus planned expenditures still acceptable?

Figure 3-6.  The Initial Operational CapabilityMilestone

Transition Phase

When we reach the last phase, the Transition phase, we have moved our system/software from the developers to the end-users. This phase, as well as the Elaboration and Construction phases, can be performed iteratively. During transition we train the end-users and the operations department in their new system. We also do beta testing of the system to make sure we deliver what the end-users and stakeholders expect. This means that we not necessarily have the same expectations as when the project started but expectations that have changed through the process. If we do not meet either the end-users’ expectations or the quality level determined during Inception, we do a new iteration of this phase. When we have met all objectives, the Product Release Milestone (see Figure 3-7) is reached and the development cycle ends. The following questions must be answered at this point:

- Is the user satisfied?

- Are the actual resources expenditures versus planned expenditures still acceptable?

Figure 3-7.  The Product Release Milestone

Disciplines

In RUP we speak about disciplines. There are nine disciplines in which we categorize our tasks in a software development project according to RUP. First, we have the engineering disciplines:

- Business modeling

- Requirements

- Analysis and design

- Implementation

- Test

- Deployment

Then we have three supporting disciplines:

- Configuration and change management

- Project management

- Environment

Let’s spend a few moments going over these in more detail. This is interesting especially when we compare this lineup to Scrum later in this chapter.

Business Modeling Discipline

The Business modeling discipline is the first one out. The aim of this discipline is to establish a better understanding between business engineering and software engineering. A welcome advance compared to the waterfall approach, because we already have seen that bridging the gap between these two is important. If you turn to Chapter 4 you will see our idea of how this could be done. Business modeling explains how to describe a vision of the organization in which the system will be deployed. It also tells us how to use this vision as a basis when outlining the process as well as when selecting roles and responsibilities.

Requirements Discipline

This discipline is responsible for gathering the requirements and uses these to describe what the system is supposed to do, so that developers and stakeholders can agree on what to build.

Analysis and Design Discipline

This discipline takes the requirements and transforms them into a design of the system. The aim is to have a design that can be changed easily when functional requirements change, which of course they will during the project. The design model is an abstraction of what the source code will look like. We can call it a blueprint if we like, which shows us components, classes, subsystems, and so on. Having well-defined interfaces between components is another important task for this discipline. We also develop descriptions of how objects of the design collaborate to perform the use cases.

Implementation Discipline

The Implementation discipline takes the blueprint and converts it to executable code. We also find testing of developed components as units here. Components developed by individual teams are integrated into an executable system by this discipline. Focus is very much on component-based development, which is a way of developing that encourages reuse of existing components. To be honest, this is a good idea in theory, but in real life I have seen very few good examples of component reuse. Most times reuse is a developer using previously built snippets of code instead of components. It is a good effort to propagate components reuse, but in reality it seems like it is not working.

Test Discipline

There are several purposes with the Test discipline:

- Verifying interaction between objects.

- Verifying all components.

- Making sure all defects are fixed, retested, and closed.

- Verifying that all requirements have been implemented (correctly).

- Identifying defects.

- Making sure defects are addressed.

- Making sure defects are addressed before deployment.

RUP states that testing should be an integrated part of the whole development project, and we cannot agree more. The purpose is to find defects as early as possible, when they can be fixed using minimal effort. There are four quality dimensions by which tests are carried out:

- Reliability

- Functionality

- Application performance

- System performance

Deployment Discipline

The activity in the Deployment discipline needs to be planned for early in the project. Deployment is mostly centered at the end of the Construction phase and the Transition phase, but to successfully deploy an application or system we need to plan for this event earlier on. This discipline focuses on delivering successful product releases. It also focuses on delivering the software to the end users. Included in this work are the tasks of packaging, distributing, and installing the software. Furthermore, the people in this discipline provide help to users, so that deployment runs smoothly.

Configuration and Change Management Discipline

RUP distinguishes three areas within the Configuration and change management discipline. There is configuration management, which is the structuring of products. We also need a way to keep control of our releases and the artifacts belonging to them, and these are tasks belonging to this area. The second area is change request management, where we keep track of all change proposals we receive for the different versions of the software. The third and last area is status and measurement management. When a change request is created it goes through different states in its workflow. It transforms from new, to logged, to approved, to assigned, to completed. This area describes how we can get reports on status of the software and its change requests and different releases. These reports are important for the project team as well as for stakeholders, so that they have a good understanding of the current project status.

Project Management Discipline

As we have seen in the preceding sections we have two dimensions of a project in RUP. We have the four phases and the iterations within them. The Project management discipline focuses on planning the phases, which is done in the phase plan. Planning how many iterations (in the Iteration plan) might be needed and also how to handle risks through the project. This discipline also monitors the progress of the project. There are some things RUP does not include in the project management discipline however. It does not cover managing people, which is usually a project management responsibility. Budget management and contract management also are not included.

Environment Discipline

The Environment discipline is the final discipline in RUP. Contrary to what one might think, we do not include the software environment here. Instead we find the environment for the project—that is, the processes and tools we should use when running the project, what work products we should deliver (more about these in a moment), and so on.

Work Products, Roles, and Tasks

All through the project we have various deliverables that should be produced. RUP called them artifacts originally and that is the term that has stuck in most people’s minds. However, after IBM took over RUP responsibilities, the term work products was coined and we will use this new term from now on.

A work product could be an architecture model, the risk list, the iteration plan, and so on. It is a representation of the results of a task. We find all documents and models we produce in the project under the term.

A task is a unit of work, which provides a meaningful result. A task is assigned to a role. A role in its turn defines a set of related skills, competences and responsibilities.

For example, the work product iteration plan is the result of a task, produce iteration plan, which is performed by the role project manager. Another example could be the task defining the architecture, which produces the result, or work product, architecture model. This is performed by the role called architect.

There are several benefits of using RUP. Many projects have completed successfully using an adaptation of this framework. We would say that project quality is significantly improved by using RUP. The problem however is that RUP has grown to be an almost impenetrable framework. There is too much to consider and choose from which makes it very hard to adapt correctly. One of our colleagues said that he did not like a model that needed adaptation to that extent. And we can understand that. Also the process is too strict and not truly iterative compared to Scrum or any other truly agile methodology. Even Ivar Jacobson, one of RUPs founders, seems to have realized that the framework has grown too immense. He has during the recent years improved on RUP and created a new more agile framework.

Manifesto for Agile Software Development

In 2001 a group of people met at a Utah ski resort to talk, ski, relax, and try to find common ground for software development. This is the result:

We are uncovering better ways of developing software by doing it and helping others do it. Through this work we have come to value:

- Individuals and interactions over processes and tools

- Working software over comprehensive documentation

- Customer collaboration over contract negotiation

- Responding to change over following a plan

That is, while there is value in the items on the right, we value the items on the left more.

Kent Beck

Mike Beedle

Arie van Bennekum

Alistair Cockburn

Ward Cunningham

Martin Fowler

James Grenning

Jim Highsmith

Andrew Hunt

Ron Jeffries

Jon Kern

Brian Marick

Robert C. Martin

Steve Mellor

Ken Schwaber

Jeff Sutherland

Dave Thomas

This manifesto is taken from the agilemanifesto.org web site and represents the values for a new development approach, and is signed by all persons mentioned. They continue:

“On February 11-13, 2001, at The Lodge at Snowbird ski resort in the Wasatch mountains of Utah, seventeen people met to talk, ski, relax, and try to find common ground and of course, to eat. What emerged was the Agile Software Development Manifesto. Representatives from Extreme Programming, SCRUM, DSDM, Adaptive Software Development, Crystal, Feature-Driven Development, Pragmatic Programming, and others sympathetic to the need for an alternative to documentation driven, heavyweight software development processes convened.”

These values were a start to a new movement in the software development community, and have gained a great number of followers since. In addition to its values, the Agile Manifesto lays down the following principles:

- Our highest priority is to satisfy the customer through early and continuous delivery of valuable software.

- Welcome changing requirements, even late in development. Agile processes harness change for the customer’s competitive advantage.

- Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale.

- Business people and developers must work together daily throughout the project.

- Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done.

- The most efficient and effective method of conveying information to and within a development team is face-to-face conversation.

- Working software is the primary measure of progress.

- Agile processes promote sustainable development. The sponsors, developers, and users should be able to maintain a constant pace indefinitely.

- Continuous attention to technical excellence and good design enhances agility.

- Simplicity—the art of maximizing the amount of work not done—is essential.

- The best architectures, requirements, and designs emerge from self-organizing teams.

- At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly.

We believe that most of these values and principles should be present in all software development, but sadly that is not always the truth. Many times projects we have participated in have delivered a large chunk of software after several months of development. Only then has the customer been brought in to evaluate the work, so both collaboration and incremental delivery has been neglected. Many times the customer has had a lot to say about the result, so we needed to write many change requests to fix the issues or explain why the software works as it does, and as the customer expected. So we feel that these values and principles are very important and a key concern in our previous projects.

As seen here, representatives of various development methods signed the manifesto. Perhaps best known are Extreme Programming (XP) and Scrum, but many of the others are well known as well.

I have chosen to cover Extreme Programming briefly and Scrum a little more extensively in this chapter. Many of the XP methods are widely used in Scrum projects (and other projects as well). A good example is Test Driven Development (TDD), which we will come back to in Chapter 6. For now, let’s start with a short introduction to XP.

Extreme Programming (XP)

Extreme Programming (XP) is a deliberate and disciplined approach to software development. It stresses customer satisfaction, an important part of the Agile Manifesto. The methodology is designed to deliver the software the customer needs, when it is needed. XP focuses on responding to changing customer requirements, even late in the life cycle, so that customer satisfaction (business value) is assured.

XP also emphasizes teamwork. Managers, customers, and developers are all part of a team dedicated to delivering high-quality software. XP implements a simple and effective way to handle team work.

There are four ways XP improves software team work; communication, simplicity, feedback, and courage. It is essential that XP programmers communicate with their customers and fellow programmers. The design should be simple and clean. Feedback is supplied by testing the software from the first day of development. Testing is done by writing the unit tests before even writing the code. This is called TDD and has started to become a very well used practice in many projects, not only agile ones. We will see later how TFS implements TDD.

The software should be delivered to the customers as early as possible and a goal is to implement changes as suggested. XP stresses that the developers should be able to courageously respond to changing requirements and technology based on this foundation.

In RUP we have use cases and in XP we have user stories. These serve the same purpose as use cases, but are not the same. They are used to create time estimates for the project and also replace bulky requirements documentation. The customer is responsible for writing the user stories and they should be about things that the system needs to do for them. Each user story is about three sentences of text written by the customer in the customer’s own terminology without any technical software jargon that a developer might use.

Another important issue is that XP stresses the importance of delivering working software in increments so that the customer can give feedback as early as possible. By expecting that this will happen, developers are ready for implementing changes.

The last topic I want to highlight with XP is pair programming. All code to be included in a production release is created by two people working together at a single computer. This should increase software quality without impacting time to delivery. Although we have never had the benefit of trying this ourselves, co-workers we have spoken to who have used pair programming are confident that it will add as much functionality as two developers working separately. The difference is that quality will be much higher. I can make a reference to my old job as an assistant air traffic controller here. Many are the times when we sat in the tower, airplane traffic was so heavy traffic the air traffic controller soon had so much to do that he or she needed help with keeping track of every airplane. We are aware that this is not the same thing, but is the fact remains that two pairs of eyes see more than one pair and this is what makes pair programming so tempting to us.

To learn more about Extreme Programming we encourage you to visit http://www.extremeprogramming.org/.

Scrum

Over to one of our favorite development models: Scrum. With all the attention Scrum has been getting the last years you might be misled to believe that it is a fairly new model. The truth is that the Scrum approach, although not called Scrum at the time, was first presented as “the rugby approach” in 1986. In the Jan-Feb 1986 issue of the Harvard Business Review Takeuchi and Nonacha described this approach for the first time.[5] In the article they argued that small cross-functional teams produced the best results from a historical viewpoint.

It wasn’t until 1990 however, that the rugby approach was referred to as Scrum. In 1990, DeGrace and Stahl[6] highlighted this term from Takeuchi and Nonacha’s original article. The term comes from rugby originally (see Figure 3-8), where it means the quick, safe, and fair restart of a rugby game after a minor infringement or stoppage.[7] This is also where the following quotation comes from:

Figure 3-8.  A real scrum!

“A scrum is formed in the field when eight players from each team, bound together in three rows for each team, close up with their opponents so that the heads of the front rows are interlocked. This creates a tunnel into which a scrum-halt throws in the ball so that front-row players can compete for possession by hooking the ball with either of their feet”.

Keep this definition in mind as we describe the development version of Scrum.

Ken Schwaber started using Scrum at his company in the early 1990s. But to be fair it was Jeff Sutherland who was the first to call it Scrum.[8] Schwaber and Sutherland teamed up and presented this approach publicly in 1996 at OOPSLA (Object-Oriented Programming, Systems, Languages, and Applications) in Austin, Texas. They collaborated to use their experience and industry best practices to refine the model so that it got its present look. Sutherland described the model in Agile Software Development with Scrum in 2001.[9]

Empirical Process Control

So what is this model or framework all about? First, let’s define two ways to solve different problems. We touched on the problems with projects in Chapter 1. When we have a problem that is similar time after time (like road construction, for example or implementing a standard system) we pretty much know what we have to expect of the various tasks at hand. We can then easily use a process, like the Waterfall model, perhaps, that produces acceptable quality output over and over again.[10] This approach is called a defined process control.

When it comes to a more complex problem however, like building a software system, we earlier saw that the traditional models do not work. We then must use something called empirical process control according to Schwaber.[11] Empirical process control has three legs to stand on:

- Transparency

- Inspection

- Adaptation

“Transparency means that the aspects of the process that affect the outcome must be visible to those controlling the process.”[12] This means that to be able to approve the outcome we must agree on what the criteria for the outcome are. Two persons cannot say they are “done” with a task unless they both agree what the criteria for “done” are.

The next leg is inspection. The process must be inspected as frequently as necessary to find unacceptable variances in the process. Because all inspections might lead to a need for making changes to the process itself, we also need to revise the inspections to fit the new process. To accomplish this, we need a skilled inspector that knows what he or she is inspecting.

The last leg is adaptation. We saw that an inspection might lead to a change of the process. This is one example of an adaptation. Another can be that we need to adapt the material being processed as a result of an inspection. All adaptations must be made as quickly as possible to minimize deviation further on.

Schwaber ruses the example of code review when he discusses empirical process control. “The code is reviewed against coding standards and industry best practices. Everyone involved in the review fully and mutually understands these standards and best practices. The code review occurs whenever someone feels that a section of code is complete. The most experienced developers review the code, and their comments and suggestions lead to the developer adjusting his or her code.”[13] Simple isn’t it? I could not have said it better myself.

Complexity in Projects

What makes a software development process so complex anyway? We discussed it a little previously but let us dive a little bit deeper into it here. In theory it might seem pretty straightforward to build software systems. We write code that logically instructs the CPU to control the computer. I mean how hard can it be? Alas, it is not that simple I’m afraid. The persons writing the code are complex machines in themselves. They have different backgrounds, IQs, EQs, views, attitudes, and so on. Their personal life also adds to their complexity.

The requirements might also be complex and they also have a tendency to change over time. According to Schwaber a large percentage of the requirements gathered at the beginning of a software project change during the project. And 60 percent of the features we build are rarely or never used in the end. Many times in my projects several persons are responsible for the requirements at the customer. Quite often they have diverging agendas as to why and what to build. Often the stakeholders have a hard time expressing what they really want. It is when they see a first prototype of the system that they fully start to see the possibilities with the software, and only then can they begin to understand what they want.

Rarely is it the case that just one computer is involved in a system either. Interaction between several machines is mostly the case. We might have a web farm for our GUI, a cluster for our application tier, a backend SQL Server, some external web services and often a legacy system, all needing to integrate to solve the needs of the new system.

When complex things interact—as people, requirements, and technology do in a software project—the level of complexity increases greatly. So it is safe to say that we don’t have any simple software problems anymore. They are all complex. Schwaber realizes this as well, and in Figure 3-9 we can see his complexity assessment graph.

Figure 3-9.  Schwaber’s complexity graph

The projects in the anarchy area are chaotic and unworkable. To get them to reach their finish lines we probably need to resolve serious issues before even starting the project.

What Scrum tries to do is address this inherent complexity by implementing inspection, adaptation, and visibility as we previously saw in empirical process control . Scrum does so by having simple practices and rules, as you will now see.

What Scrum Is

Scrum is a powerful, iterative, and incremental process. Many get fooled by its perceived simplicity, but keep in mind that Scrum can handle CMMI at level 5, which not many other processes can do. Figure 3-10 shows the skeleton of the Scrum model to which we attach the rules and practices. Each iteration consists of several daily inspections.

Figure 3-10.  The Scrum skeleton

During these inspections, team members evaluate each other’s work and the activities performed since the last inspection. If necessary adjustments (adaptation) are found they are implemented as quickly as possible. The iterations also conclude with inspections when more adaptations can be made. This cycle repeats until it is no longer funded.

All the requirements that are known at the beginning of the project are gathered in the product backlog, which is one of the artifacts of Scrum. We will soon come back to this. The project team reviews the backlog and selects which requirements should be included in the first iteration, or sprint as it is called in Scrum. These selected requirements are added to the sprint backlog where they are broken down into more detailed items. The team then makes their best effort to turn the sprint backlog into a shippable increment of the final product. The team is self-managing, which means they collectively decide who does what and what is the best way to solve the problems.

The increment is presented to the stakeholder(s) at the end of the sprint so they can inspect it and make any adaptations necessary to the project. The sprint is most often 30 days, although often I have seen sprints somewhere between 2-4 weeks in many projects. It depends a bit on the sprint backlog items. When I took my Scrum master certification, Ken Schwaber related that he once had had to have a one-week sprint in a project. The reason for this was that the team malfunctioned and this way he could more easily catch the reason for this and adjust the process so that the project ran more smoothly.

The stakeholder’s adaptations and feedback are put into the product backlog and prioritized again. Then the team starts the process all over again and selects the backlog items they think they can finish during the next sprint. These are put into the sprint backlog for the next sprint and broken down to more manageable items. And so it continues until the stakeholder thinks they have received the business value they wanted and funding stops.

If we look at the three legs of empirical process control again we can see that Scrum covers them nicely. Transparency is implemented by letting the team and stakeholders agree on what is the expected outcome of the project and of each iteration. Inspection occurs daily and also at the end of each sprint. Adaptations are the direct result of these inspections and a necessary thing in Scrum.

The Roles in Scrum

There are different roles in Scrum just as there are in all previously mentioned models. But the difference is that Scrum has fewer roles, which are not defined in the same strict way as in the others. Scrum has the three roles:

- The product owner

- The team

- The Scrum master

The Product Owner

Let’s start with the product owner. He or she is responsible to those funding the project to deliver a product or a system that gives the best Return On Investment (ROI) they could get from the project. The product owner must acquire the initial funding of the project and also make sure it is funded through its lifespan. The product owner represents everyone with a stake in the project and its result. At the beginning of a project the product owner gathers the initial requirements and puts these into the project backlog. It is the product owner who ultimately decides which requirements have the highest priority based on ROI or business value (for example) and decides into which sprint they should go. During the project the product owner inspects the project and prioritizes the product backlog and sprint backlogs so that the stakeholders’ needs are met.

The Team

The team is responsible for the development. There are no specific roles in the team. Because the team is cross-functional and self-organizing it is their responsibility to make sure they have the competencies and staff they need for solving the problems. So it is not the scrum master who decides who does what and when, as a project manager would do in a traditional approach. These are some of the reasons behind this thought, as taught by Ken Schwaber in his scrum master course:

- People are most productive when they manage themselves;

- People take their commitment more seriously than other people’s commitment for them (like when a project manager commits that the person should accomplish something);

- People have many creative moments during down time;

- People always do the best they can; and,

- Under pressure to work harder, developers automatically and increasingly reduce quality.

The team should be seven persons plus or minus two for optimal result. A logical team consists of one programmer, one tester, a half time analyst/designer, and a half time technical writer. The optimal physical team has 2.5 logical teams. The team decides which items in the backlog they can manage for each sprint based on the prioritized backlog. This whole thinking is a giant leap from traditional project management and takes some getting used to. Some persons do not accept this and find it impossible to work this way.

The Scrum Master

The scrum master is responsible for the Scrum process and has to make sure everybody in the team, the product owner, or anyone else involved in the project knows and understands the process. The scrum master makes sure that everyone follows the rules and practices of the Scrum process. So the scrum master does not manage the team, the team is as we saw self-managing.

If a conflict occurs in the team, the scrum master should be the “oil” that helps the team work out their problems smoothly. It is also the scrum master’s responsibility to protect the team from the outside world so that they can work in peace and quiet during the sprint, focused on delivering business value. The following lists the scrum master responsibilities again according to Ken Schwaber’s course material:

- Removing the barriers between development and the customer so the customer directly drives development;

- Teaching the customer how to maximize ROI and meet their objectives through Scrum;

- Improving the lives of the development team by facilitating creativity and empowerment;

- Improving the productivity of the development team in any way possible; and,

- Improving the engineering practices and tools so each increment of functionality is potentially shippable.

The Scrum Process

Now that we know the basics of Scrum it is time to take a look at what happens during a Scrum project.

The product owner, after arranging initial funding for the project, puts together the product backlog by gathering functional as well as non-functional requirements. The focus is to turn the product backlog into functionality and it is prioritized so that the requirements giving the greatest business value or having the highest risk come first. Remember that this approach is a value-up paradigm where we set business value first.

Note  Value up measures value delivered at each point in time and treats the inputs as variable flows rather than a fixed stock. If you want to learn more about this, please see Software Engineering with Microsoft Visual Studio Team System by Sam Guckenheimer, Addison Wesley, 2006.

Then the product backlog is divided into suggested releases (if necessary), which should be possible to implement immediately. This means that when a release is finished we should be able to put it into production at once so that we can start getting the business value as quickly as possible. We do not have to wait until the whole project is done until we can start getting return on our stakeholder’s investments.

Because the Scrum process is an adaptive process, this is just the starting point. The product backlog and the priorities change during the project as business requirements change and also depending on how well the team succeeds in producing functionality. The constant inspections also affect the process.

When a sprint is starting, it initiates with a sprint planning meeting. At this meeting the product owner and the team decides, based on the product owner’s prioritization, what will be done during this sprint. The items selected from the product backlog are put into the sprint backlog.

The sprint planning meeting is time-boxed and cannot last more than eight hours. The reason for this strict time-box is that the team wants to avoid too much paperwork about what should be done.

The meeting has two parts. The first four hours are spent with the team and the product owner, where the latter presents the highest priority product backlog issues and the team questions him/her about them so that they know what the requirements mean. The next four hours are used by the team so that they can plan the sprint and break down the selected product backlog items into the sprint backlog.

When the project is rolling, each day starts with a 15-minute daily scrum or stand up meeting (see Figure 3-11). This is the 24-hour inspection. During this meeting each team member answers three questions:

Figure 3-11.  The sprint in Scrum

- What have you done since the last daily scrum meeting?

- What do you plan to do on this project until the next daily scrum (your commitments)?

- What impediments are in the way of you meeting your commitments toward this sprint and this project?

The reason for this meeting is to catch problems and hence be able to make timely adjustments to the process. It is the scrum master’s responsibility to help the team members get rid of any impediments they may have.

When a sprint comes to an end a sprint review is held. This meeting is also time-boxed, but at four hours instead of eight. The product owner and the stakeholders can get a chance to see what the team has produced during the sprint and reflect on this. But it is important to remember that this meeting is not a demonstration, it is a collaborative meeting between the persons involved.

Now there is only one meeting left; the sprint retrospective. The sprint retrospective takes place between the sprint review and the next sprint planning meeting. It is time-boxed at three hours. The scrum master encourages the team to adjust the development process, still within the Scrum process and practices framework boundaries, so that the process can be more effective for the next sprint.

What happens if we have a larger project than one with only a team of approximately seven persons? Can Scrum scale to handle this? According to Mike Cohn in an article on the Scrum alliance web site,[14] we can use a process called scrum of scrums:

“The scrum of scrums meeting is an important technique in scaling Scrum to large project teams. These meetings allow clusters of teams to discuss their work, focusing especially on areas of overlap and integration. Imagine a perfectly balanced project comprising seven teams each with seven team members. Each of the seven teams would conduct (simultaneously or sequentially) its own daily scrum meeting. Each team would then designate one person to also attend a scrum of scrums meeting. The decision of who to send should belong to the team. Usually the person chosen should be a technical contributor on the team—a programmer, tester, database administrator, designer, and so on—rather than a product owner or ScrumMaster”.

By using this technique we can scale Scrum infinitely, at least in theory.

That’s basically it. Scrum is a lean process and appeals a great deal to us. One of the authors (Rossberg) has had the privilege to do his scrum master certification during a course held by Ken Schwaber and this was a very uplifting experience. Unfortunately, some customers or stakeholders can find Scrum a bit vague, so they won’t try it. They think they have more control the way they used to run projects and are perhaps a bit afraid to embrace this modern way of doing projects. This still has not changed over the years, although more and more people we meet have seen what scrum and agile can do to help them run better projects.

Some even think that documentation and planning are not necessary in Scrum. Developers like this idea because they don’t want to write documents, while stakeholders tremble at the thought. But nothing could be further from the truth. Scrum does not say we do not document or plan. The contrary is true. Planning, for instance, is done every day, during the daily scrum (see Figure 3-12). Documents should also be written, but we scale away documents that are not necessary—documents that are produced only for the sake of documentation and almost never are read after they are produced. We document what is needed for the system and the project. We document our code; we document traceability, and so on.

Figure 3-12.  Planning in Scrum

We have found that some companies think they are using Scrum just because they develop iteratively. In many cases they have changed the Scrum process so that it will not help them solve their development problems, problems that are clearly visible in a true Scrum project. Instead they have used Scrum as make-up covering the bad spots and when the project still fails, they argue that Scrum doesn’t work either, we still do not deliver value, or still have over runs, and so on. So when implementing Scrum follow the process and framework, adjust the organization to Scrum not the Scrum process to the organization.

How about architecture, the IT architect shouts. Don’t we need to set the architecture before we start coding? Well, no! Scrum uses the concept of emerging architecture, which means we do a base architecture during the first sprint. This is evolved and built upon as the sprints continue. So the architecture is emerging as we go along, giving us agility in this concept as well.

When a customer is hesitating, we must use our most persuasive skills to get them to at least do a pilot on a project and see how it goes. It is our firm belief that most will be pleasantly surprised afterward. When they see that they get a product better meeting their requirements and giving better ROI in a shorter time than traditional projects, they usually melt.

Capability Maturity Model Integration (CMMI)

Before moving on, we want to say a few words about Capability Maturity Model Integration (CMMI). One of the implementations of the process templates in TFS uses CMMI as a basis, so you need to know more about CMMI before we see Microsofts implementation. CMMI is a way an organization can implement process improvement and measure the level of maturity of a process (http://www.sei.cmu.edu/cmmi/).

CMMI is a process improvement approach whose goal is to help organizations improve their performance across a project, a division, or an entire organization. The model helps integrate traditionally separate organizational functions, set process improvement goals and priorities, provide guidance for quality processes, and provide a point of reference for appraising current processes.

CMMI-DEV contains four process areas (PA) categories, each of which has one to four goals; each goal is composed of practices and there are 22 practices in all. These goals and practices are called specific goals and practices, as they describe activities that are specific to a single process area. An additional set of goals and practices applies across all the process areas; this set is called generic goals and practices.

The CMMI levels of maturity are based on Philip Crosby’s manufacturing model.[15] The following levels exist:

- Level 0: Incomplete process.

- Level 1: Performed process. Little or no control of the process. The project outcome is unpredictable and reactive. All the process areas for performed process have been implemented and work gets done. However, the planning and implementation of process have not yet been implemented.

- Level 2: Managed process. The organization has satisfied all requirements for implementing managed processes. Work is implemented by skilled employees according to policies.

- Level 3 Defined process. At this level a set of standard processes within the organization can be adapted according to specific needs.

- Level 4: Qualitatively managed process. All aspects of a project are qualitatively measured and controlled. Both the operational as well as the project processes are within normal control limits.

- Level 5: Optimizing process. At this level there is a continuous project improvement. CMMI level 5 focus on constant process improvement and also on reducing common cause variation.

Summary

This chapter focused on describing some development frameworks you can use for your development processes. Even though we suggest you consider using Scrum or any agile method, we discuss the others here as well, so that you can compare and make a decision for yourself.

Choosing the process model is important because it affects the outcome of a project. So before implementing Team Foundation Server please consider this topic and do not rush into the templates that Microsoft provides just because they are included out of the box. The templates can be used as a start, but Microsoft encourages you to change these templates to fit your organization as well. However, remember that in a truly agile manner always be prepared to change and adjust your process template, no matter which process you choose. Start with a pilot project and go from there.

[1] Winston W. Royce, “Managing the Development of Large Software Systems,” 1970, Proceeding, ICSE ‘87 Proceedings of the 9th international conference on Software Engineering

[2] Steve McConnell, Rapid Development: Taming Wild Software Schedules, (Microsoft Press, 1996).

[3] Barry Boehm, “A Spiral Model of Software Development and Enhancement,” ACM SIGSOFT Software Engineering Notes, Volume 11 Issue 4, August 1986

[4] Rational Unified Process: Best Practices for Software Development Teams, https://www.ibm.com/developerworks/rational/library/content/03July/1000/1251/1251_bestpractices_TP026B.pdf.

[5] Takeuchi and Nonacha, Harvard Business Review, Jan/Feb 1986, (http://www.sao.corvallis.or.us/drupal/files/The%20New%20New%20Product%20Development%20Game.pdf).

[6] DeGrace and Stahl, “Wicked Problems, Righteous Solutions,” 1990, http://w.leanconstruction.org/pdf/RighteousSolution.pdf.

[7] www.planetrugby.com

[8] Jeff Sutherland, “Agile Development: Lessons Learned from the First Scrum,” http://www.scrumalliance.org/resources/ 35, 2004.

[9] Ken Schwaber and Mike Beedle, Agile Software Development with Scrum, (Prentice Hall, 2001).

[10] Ken Schwaber, The Enterprise and Scrum, 2007

[11] Ibid.

[12] Ibid.

[13] Ibid.

[14] (http://www.scrumalliance.org/articles/46-advice-on-conducting-the-scrum-of-scrums-meeting)

[15] Philip Crosby, Quality is Free: The Art of Making Quality Certain, McGraw-Hill, 1979.

CHAPTER 4

ALM Assessments

Often in our work we need to perform an assessment of a given situation. It could be describing a system’s present architecture and then coming up with a plan for improving it, or it could be assessing how a system scales. When we start working with people instead of technology, the focus of the assessments is on a different level: suddenly there is a need to consider human factors and not only technological topics. Before TFS entered the scene, we conducted surveys of how a department functioned, for instance. We did such assessments by interviewing people from the organizations and thereby getting a picture of the situation.

Interviews are complex and result in a lot of information that you need to process afterward. Mostly we have prepared by writing questions on a form used for the interviews. Any follow-up questions that came up were carefully documented and included in later assessments. So this form started to become our tool, even though we did not have it in an application. Instead, we used a Word document that was updated and printed for each interview.

However, we have been a bit reluctant to use only tools. We thought about digitizing the questions into a web application or the like and letting the subjects answer the questions themselves, but we didn’t want to abandon the interview part. Tools can help, but they can also hinder because you can become too dependent on them. Another aspect of using tools for this purpose is that if we let a single person answer our questions in, let’s say a web form, we wouldn’t be around to ask the follow-up questions. These questions enable us to learn so much more than we would if we only looked at the answers to the original bunch of questions.

We started working with TFS a few years back when the product was new. We felt that it was a good foundation for taking control of an ALM process. It lacked (and still does in some cases) some of the support we wanted, but it was a good start, and nothing else was offered on the Microsoft platform to compete with it. The more we dived into it, the more we started thinking of how we best could evaluate an organization to implement TFS at their site. We realized very quickly that ALM was an important part of an organization’s ability to improve their software development cycle, and TFS was an excellent tool to help our customers. We could see that we had tools for visibility, traceability, and automation of high-level processes that fit nicely into the ALM concept.

Microsoft released their ALM assessments on the Web some years ago (www.microsoft.com/almassessment), including the Application Platform Optimization (APO) model. Microsoft provided its APO model to help IT organizations understand and adopt a more flexible and agile application platform. These felt like a good start, so we set off to figure out how to best use these tools to help our customers implement TFS. Previously, we found that many of our customers used only a fragment of the true potential of TFS (most often the version-control system). If we could show them why they should use more features, they would get so much more out of it and at the same time be more successful in running their projects.

Microsoft’s assessments were tools pretty much like web questionnaires, which is why we set off to work on creating an assessment based on Microsoft’s Application Platform Capability Assessments, but mixed with an interview part as well. The reason for this is that a tool cannot read between the lines. It cannot hear what a person says apart from what is actually answered. It is important to have the capability to ask follow-up questions, to clarify and discuss where needed.

One great advantage of the Microsoft assessments is that after everything is entered into the tool, the tool can handle much of the data processing automatically. This decreases the labor attached to the manual processing of the large amount of data an interview gives you. We “only” need to make sure that we enter the most realistic values into the system, and that’s where the interview part can help us out. By using the questions from the online assessment as a form for our interviews, we have a good foundation for interviews. Then we can use the results of the interviews, including follow-up questions and observations, to enter data into the assessments form, giving us the capability to reflect on each question and choose the best answer for the organization we are working with.

Microsoft Application Platform Optimization (APO) Model

APO is part of Microsoft’s Dynamic IT initiative that also includes the Infrastructure Optimization model and the Business Productivity Infrastructure model. There are four primary goals of the Dynamic IT initiative:

- Manage complexity and achieve agility

- Protect information and control access

- Advance the business via IT solutions

- Amplify the impact of your people

These models are aimed at helping customers better understand the current IT capabilities in their organizations and based on these results take the capabilities to a higher level of maturity.

In this section, we’ll give you a brief overview of the Infrastructure Optimization model and the Business Productivity Infrastructure model before focusing on APO.

Infrastructure Optimization Model

With the Infrastructure Optimization model, Microsoft focuses on four areas: the desktop infrastructure, the server infrastructure, the remote infrastructure (which covers how to handle remote offices or locations), and virtualization. Based on best practices internal to Microsoft as well as on feedback from customers, Microsoft has provided an approach they say will do three things:

- Control costs by looking over hardware, utilities, and space expenses in the data center to see where we can reduce costs. The costs also can be controlled or reduced by optimizing deployment testing and training expenses as well as reducing security breaches (in addition to other strategies not covered in this book).

- Improve service levels. This can be done, for example, by reducing service interruptions caused by security breaches, having a robust disaster recovery strategy, and avoiding desktop configuration conflicts.

- Drive agility. Here we find topics familiar to us by now, such as increasing the ability to adapt to business changes.

Business Productivity Infrastructure Model

Microsoft provides a definition for what optimizing our Business Productivity Infrastructure means (http://microsoftio.partnersalesresources.com/bpio.aspx). Microsoft defines it as follows:

“The Business Productivity Infrastructure Optimization (IO) model includes a complete set of technologies that helps streamline the management and control of content, data, and processes across all areas of your business. It helps simplify how people work together, makes processes and content management more efficient, and improves the quality of business insight while enabling IT to increase responsiveness and have a strategic impact on the business.

The Business Productivity IO Model defines five capabilities that are required to build a more agile infrastructure”:

- Collaboration

- Unified Communications

- Enterprise Content Management

- Reporting & Analysis

- Content Creation

APO Maturity Levels

Let’s now move our focus to the APO model. Before we explain what this really is, we will spend some time with the maturity levels Microsoft has identified for the assessment of the APO model. There are four optimization levels (see Figure 4-1).

Figure 4-1.  The four optimization levels for categorizing an IT organization (as stated by Microsoft)

Basic

When a company is classified as a basic organization, it is characterized by brittle, disconnected applications and platforms. This fact hinders rapid adjustments to business changes and also hinders the rapid development and interoperability of business-critical applications. The organization makes no real use of business processes, or these processes (if they exist) are often ill-defined. The processes are definitely not automated in any way. Such an organization probably has no tool for collaboration between teams and team members, and definitely lacks the clear connection between IT and business that is crucial for a company to have.

The development process is probably quite rigid, which makes development hard to control. All in all, this leads to higher costs, application backlogs, and lower IT productivity. The IT department is probably seen as just a cost to management and its true potential as a strategic asset is clouded by all problems.

Standardized

The standardized organization has begun to use industry standards broadly across departments, as well as with business partners. These standards could be as simple as starting to use XML, for instance. Furthermore, such an organization has started to take control of their development and data infrastructure, enabling the use of business intelligence reports and analytics. They have also started to automate some of their business processes. The IT department has slowly begun to be seen as a business enabler that could provide help in building more-adaptive systems quickly.

Advanced

At an advanced level, IT is truly seen as a business enabler and partner. Now infrastructure and systems are more easily managed throughout their lifecycles. The business processes are well-defined and well-known. The business side has begun to truly take advantage of IT and can rely on the IT department to quickly make adjustments when changes in the business occur. Such a company has standardized a robust and flexible application platform for the most critical systems and processes.

Dynamic

A dynamic organization is fully aware of the strategic value of its infrastructure. The organization knows IT can help run the business efficiently and stay ahead of market competitors. Costs are controlled to the company’s maximum ability. All processes are automated and integrated into the technology, enabling IT to adjust to business changes and needs. The collaboration between the business and the IT department is working smoothly, as well. It is also possible to measure the effects of business benefits of IT investments, which is a strong argument when showing the value of IT. This kind of organization has also used SOA to the fullest so that cost-effective systems can be developed.

APO Capabilities

With this background, we are ready to have a look at the capabilities included in the APO model. Microsoft defines five capabilities, as shown in Figure 4-2.

Figure 4-2.  Microsoft has defined five capabilities for their APO model (https://partner.microsoft.com/40029360)

User Experience

User experience (UX) is an unappreciated area. UX is important, but most often this capability is not included in projects as a special field. We far too often rely on developers to solve UX problems without thinking of the effects bad UX design could have. Not many developers are skilled in this area and the importance and value of this field is included in this capability. Usability should be a higher priority in most cases. We have seen projects that were considered failures because the user experience was too technical. The developers had a technical view on how to work in the application (or system) and had designed it with this as the primary viewpoint. The design was not in line with how the end user really worked, so the user interface needed a lot of rework, resulting in higher costs and a delayed project.

Business Intelligence

Microsoft also identifies business intelligence (BI) as a capability. Microsoft and many others have a vision that business insight should be provided to all employees. This leads to faster, more reliable, and more relevant decisions in the organization, because all members of the organization have access to the right information to support good decision making. We find areas such as data mining, reporting, data warehousing, data integration, analysis, and more here.

SOA and Business Process

SOA and business process is another capability. SOA can be a great thing to implement in our organizations, as you might have heard over the years. But SOA in our opinion has turned a little cold lately, and we don’t hear much about it. This capability focuses on the integration between Business Process Management (BPM) and SOA.

This is an immature market according to some surveys as only around 30 percent of respondents said they had a combined strategy for SOA and BPM. A significant two-thirds of the organizations had no such strategy, in other words. This might be good for us as consultants because the market exists for helping out, but could be disastrous for some companies if they don’t change this situation.

Having effective business processes that we are able to quickly adjust to new or changed business needs is essential for an organization these days. We need ways to manage our processes and then automate them in our IT infrastructure. BPM will help with managing the processes, and SOA will help with implementing them in our IT environment.

Data Management

Data management covers what an organization should consider when integrating data management and analysis software. How is the data storage handled? Will it support the business-critical systems reliably? This capability also covers how database development is being carried out, how well the database team is integrated into the development projects, and so on. The main focus is to determine how best to build an integrated, well-managed, and always-connected data infrastructure to support our most demanding and mission-critical applications.

Development

Let’s look at the development capability. Here we find the things that can enable an organization to develop applications that connect business processes to meet business needs. It covers areas such as what kind of development platform the organization uses, whether a development process is in place, how the development team and projects are organized, how visibility into the process of a development project is going, and so on.

Application Platform Capability Assessment

Next we will take a look at what an assessment can look like.

This assessment is called the Application Platform Capability Assessment and exists in three versions:

- Application Lifecycle Management

- Business Intelligence

- SOA and Business Processes

We will use Application Lifecycle Management for our discussion because it is the most relevant for covering the development process and the ALM process. It covers all aspects of the ALM process and is very extensive. So in order to get good coverage on what parts of the ALM process you can improve, this is the assessment you should use.

When starting this assessment you can see that we have two options:

- Start Individual Assessment

- Start Team Assessment

We will use the Team Assessment. One big difference with this assessment compared to the Individual Assessment, aside from the number of questions and the detail level in them, is that it is intended to be filled out by more than one person. Microsoft also encourages the use of a partner when gathering information about your organization. The best thing about using a partner for such an assignment is that you get an independent view on the answers and the state of the organization.

The ALM Assessment for Teams includes many areas; it has eight practice areas, all divided further into a various number of practices. The assessment has about 200 questions (this figure is subject to change), so it covers a great deal of material.

The following list shows the eight practice areas and their practices:

- Architecture and Design

- Architecture framework

- Analysis and design

- Database modeling

- Requirements Engineering and User Experience

- Elicitation

- Requirements analysis

- Requirements management

- Traceability

- UX research

- UI design and prototyping

- UI implementation

- End-user documentation

- Development

- Code writing

- Code analysis

- Code reuse

- Code reviews

- Quality metrics

- Database development

- Software Configuration Management

- Collaborative development

- Database change management

- Version-control repository

- Release management

- Build management

- Change management

- Governance

- IT governance maturity

- Application portfolio management

- Compliance management

- Deployment and Operations

- Designed for operations

- Deployment

- Environment management

- Operations

- Customer support

- Database deployment

- Project Planning and Management

- Project initiation

- Project planning

- Project monitoring and control

- Risk management

- Stakeholder management

- Project close

- Testing and Quality Assurance

- Test resource management

- Test planning

- Test management

- Test types

- Database testing

Starting the Assessment

When you start an assessment, you begin by filling in some information about the company. You can see in Figure 4-3 that you also set a time frame indicating the period that you’ll allow people to add information into the assessment. You also can fill out the name of the partner you work with during the process.

Figure 4-3.  Starting an Application Platform Capability Assessment as an owner

The creator, or owner, of the assessment sends an e-mail to all contributors containing the URL of the assessment. Once a participant opens the URL, they are welcomed with a page as seen in Figure 4-4.

Figure 4-4.  Starting an Application Platform Capability Assessment as a contributor

When you enter the assessment, you’ll have one page for each of the practice areas. As you know, these areas are divided into practices, and these are displayed as sections on each practice area page. Each section displays the questions for each practice (see Figure 4-5). Some practices have only one question, and others have more, so variance is great.

Figure 4-5.  Answering an Application Platform Capability Assessment as a contributor

So what kinds of questions can you expect from this assessment? Some are detailed next so you can get an idea of how the practices are examined. Just as you would expect, the questions are more detailed in each area compared to the shorter APO assessment. Let’s take a look at two examples.

Example 1: Assessing Requirements Handling

This first example question asks about how you handle requirements in the organization. It tries to find out whether you update the original requirements when changes happen.

- Requirements Engineering and User Experience—Requirements Management practice

- Q: Are requirements updated as requirements change?

- A: 1. Rarely, 2. Infrequently, 3. Sometimes, 4. Most times, 5. Always, 6. Don’t know

In many organizations, we have seen requirements remain fixed in the requirements specification no matter what happens to the requirements themselves. What usually happens is that the developers go ahead and change the functionality to reflect the requirement change (that might have come as an e-mail or by phone), without changing the documentation.

So discussing this question at the interview will tell you more than the question itself would. Try to find out whether the organization has a requirements system or application, and if they do, whether it is used. You can also ask about how their change requests are handled, whether they have a process for that. If they have such a process, a developer would not implement the change. Instead, the developer would redirect the person initiating the change to the correct step of the change request process—usually by sending a formal change request to the project manager or whoever is in charge of this.

Example 2: Assessing Code Analysis

The next question we will show you covers code analysis. Code analysis enables us to make sure that developers follow a set of rules indicating how code must be written. Code analysis can include everything from naming conventions to more-specific coding practices. This analysis can be automated.

- Development—Code Analysis practice

- Q: Is there good static code analysis?

- A: 1. Rarely, 2. Infrequently, 3. Sometimes, 4. Most times, 5. Always, 6. Don’t know

There is no chance of having good code analysis without having it automated in some way. In Chapter 6, you will see that this is built into TFS, so we have access to it from there.

We use this question to find out more about the company’s use of tools for automating the development process. It’s a good starting point to dive into this subject, and to see whether the company has other areas where automation is or is not used. The answers will help you better understand how the organization can benefit from implementing TFS (or any other ALM tool).

Viewing the Results

When all participants have answered their assessments, the assessment owner closes the assessment so that no one can enter any more information. To see the results, the owner then clicks the Generate Report button (see Figure 4-6).

Figure 4-6.  Generating a report for an Application Platform Capability Assessment

The questions are rated on a five-degree scale (1, 2, 3, 4, and 5) with a sixth choice being the possibility to answer “Don’t know.” The best score is 5, and the lowest is 1. (This is exactly like the scoring system we had in the Swedish schools a few years ago!)

The system calculates the medium score for each capability, for each capability area, and for the whole assessment and presents it graphically to the user (see Figure 4-7).

Figure 4-7.  The report for an Application Platform Capability Assessment

You will see a text overview of the whole assessment. Our demo here shows a pretty good score of 3.56, which puts this organization at the Advanced level. In the table below this score, you can see the individual practice area scores. You can see the maturity level of each area as well. This information is a pretty good summary for management to look at. But if you want to see more detailed information, you can scroll down the web page to see the score and maturity level for each practice, as seen in Figure 4-8.

Figure 4-8.  Report detail from the score for each practice

Now you can pinpoint any problem practices, which are practices with lower scores. You can look for the color red or yellow in the right column (not shown in Figure 4-9) to quickly identify such a practice. A manager might want to dive even deeper into the results, so farther down you will find the score for each question (see Figure 4-9). This setup gives you the capability to identify exactly where you have problems, in which practice—and then to use this information for planning corrective measures.

Figure 4-9.  You can see the score for each question in the assessment

You probably want to download this report to use it internally, and Microsoft allows you to do this. You can save the report in different file formats so that you can process it any way you want.

Note  You do not get any financial information in the Application Platform Capability Assessment report, only maturity scores at different levels of detail.

How to Use the Results

When we assess an organization’s ALM process, we need to gather as much information as possible about the client and the client’s organization. This is hard work if done manually. One would say it is close to impossible to collect that amount of information from so many people in the organization in a cost-effective way, without the use of a tool.

We used the questions from Microsoft’s assessment tools as a basis for conducting interviews with people from ALM organizations. We gathered people from all aspects of the ALM process, making sure both the business side and the IT side were represented. Then we spent 30 to 90 minutes discussing these questions in one-on-one meetings. After conducting all interviews, we completed the assessment ourselves and used the interview result as a basis for answering the questions. This way, we obtained a pretty realistic view of the organizations and their ALM processes. The results have also been better and have been more accepted by the organizations when we have done it this way as compared to when we have let only one person complete an assessment.

There are several ways to use the technique described in this book. We have tried it a few ways. Let’s first start with a few comments on the assessments themselves. Tools are good in most cases. Tools can help us with many tasks and simplify our lives greatly, but tools for assessing a complete ALM process? Could that work?

Using the Application Platform Capability Assessment

The team assessment dives down deep. Keep in mind that a tool cannot elicit all the nuances that an observer can. We use the Application Platform Capability Assessment questions as a basis for interviews, and make sure that we interview people about their special fields only. Architects answer architect questions, project managers answer project management questions, and so on. This approach has worked very well.

We strongly recommend using an external partner for the assessments. It is often easier to look at an organization when there are no strings attached, and no manager to answer to.

Why Do an Assessment?

Why should you spend the time and money doing an ALM assessment? The best reason is that before implementing TFS (or any other ALM tool for that matter), you need to know what the potential pitfalls in the ALM process really are. Every process has room for improvement, and the assessment is a very good way of finding out where improvements are most needed. You need to have as clear a picture as possible of the organization’s maturity level so that you can better anticipate what actions are needed for the organization to improve and thus be more effective.

The value of an assessment can be summarized in terms of what it provides. The assessment:

- Gives an analysis of the strengths and weaknesses with the current way of working

- Gives a foundation for prioritizing the ALM effort in the company

- Gives input to creating a roadmap for the improvement process

- Gives a baseline for follow-up assessments to measure the impact of an improvement project

An ALM Assessment will help your organization understand the current situation and make informed choices on the way to improve the ALM process. Often we think we know where the problem is, but before doing a proper analysis, it is hard to say. Making changes to the wrong thing(s) ultimately costs a lot of money—money better spent on correcting the real problem, which in the end could save money.

An ALM process is not something to implement all at once. You do it best little by little, piece by piece, starting with the lowest hanging fruit. If you just can show the decision makers the improvements of smaller actions, it becomes easier to get them to fund the rest as well.

We come back to this assessment throughout the book and tie together the topics with corresponding parts of the assessment.

Summary

This chapter has discussed the value of doing an assessment of the ALM process before implementing TFS (or any other ALM tool of your liking). The ALM assessments Microsoft offers are good but are best used in collaboration with an external partner carrying out the process in the form of interviews.

Use the assessment as a baseline for evaluating the impact of an improvement project. After a change in your process, you can perform the assessment again and again to measure and make sure you are moving in the right direction.

CHAPTER 5

ALM Using TFS

In this chapter, we will discuss a tool which will make it clear why ALM is an important process for organizations engaged in IT development. A good implementation of ALM will help the organization deliver better business value to fulfill its business needs. Automating tasks by using tools such as Visual Studio 2012 and TFS 2012 can support this process.

In this chapter, you will learn how TFS can be used to fulfill the three main pillars of ALM and the issues addressed by ALM 2.0+, which we covered in Chapter 2. You will start with an overview of ALM and of TFS and then move on to the specifics of using TFS for ALM.

Application Lifecycle Management Overview

As you may recall from Chapter 2, there are three main pillars of an ALM process:

- Traceability of relationships between artifacts. The lack of traceability can be a major cost driver in any enterprise. There must be a way of tracing the requirements all the way to delivered code and back again—through architect models, design models, build scripts, unit tests, test cases, and so on. Practices such as test-driven development and configuration management can help, and these can be automated and supported by TFS.

- Automation of high-level processes. There are approval processes to control handoffs between analysis and design. There are other handoffs among build, deployment, testing, and so on. Much of this is done manually in many projects, and ALM stresses the importance of automating these tasks for a more effective and less time-consuming process.

- Visibility into the progress of development efforts. Many managers and stakeholders have limited visibility into the progress of development projects. Their visibility often comes from steering group meetings during which the project manager goes over the current situation. Other interest groups such as project members may also have limited visibility of the whole project even though they are part of it. This often occurs because reporting is hard to do and can involve a lot of manual work. Daily status reports can quite simply take too much time and effort to produce, for example, especially when we have information in many repositories.

Other important topics that ALM 2.0+ addresses are as follows:

- Improving collaboration. Collaboration is needed between teams, team members, stakeholders, and users, just to mention a few relationships. When development is spread around the world in different locations, collaboration can be hard to manage without the help of a proper tool.

- Closing the gap between IT and business. The big gap between IT and the business side of an organization is a serious problem for organizations, preventing us from delivering the greatest business value we can achieve in our projects.

- Using one tool. The complexity of using several tools for solving project issues as a team member can be tough and costly as well. Switching between tools can be a cost driver. Using one tool enabling us to add plug-ins and use more features directly in our ordinary GUI instead of switching between applications is preferable. So, if you have several roles in a project, you can still use one tool to get the job done.

- Enhancing role switching. ALM also addresses the potential to use one tool when switching among different roles in a project. In many cases, project members play several roles in projects. A developer, for instance, might also work with tests or databases. If that person can use the same GUI for all tasks, there will be minimal overhead for switching between these roles.

Team Foundation Server Overview

TFS has come a long way toward fulfilling the ALM vision, but it does not cover everything. TFS is an open and extensible product that will let us adjust its features to our needs and add the things it might lack at this point to support our specific needs. It is also important to know that Microsoft is spending a lot of time, energy, and money on developing this product further. It is not a toolset that will go away quickly (although one never knows); it is one of the most important toolsets in the Microsoft ecosystem.

Team Foundation Server

You can see that the heart of ALM in the Visual Studio 2012 world is TFS 2012, as shown in Figure 5-1.

Figure 5-1.  Visual Studio 2012 Suite—an overview

TFS exposes different functions and services for developers, project managers, version control, reporting, and build and work item tracking (see Figure 5-2). You will soon take a look at all of these in more detail. Not shown in this picture is that TFS uses Microsoft SQL Server as its data repository.

Figure 5-2.  The heart of TFS—a work item; in this case, a product backlog work item

Note  Work items are used to manage different types of information in TFS. We have work items for requirements, bugs, general tasks, and so on. To put it simply, a work item is a piece of work that must be completed in a project. The work item tracking system is one of the core parts of TFS for our ALM process implementation.

Process Template

What keeps all of these services together is the process template (see Figure 5-3). This is a very interesting part of TFS. The template helps us visualize and automate tasks and steps that the process includes. It helps us by providing document templates for requirements specs, test cases, scenarios, handoffs, and other artifacts we should produce.

Figure 5-3.  The process template customizes TFS behavior

Most companies use some kind of process for their development or ALM. Even though some companies don’t think they have a process, they do. The process might not be written down, but the company still has ways of doing things that in reality is the process—for instance, naming conventions, where to store builds, how to handle change requests, and so on.

In many cases, we have seen companies with lots of money invested in their processes. They have sent staff to training, provided large process manuals, and so on. However, they have had problems getting project members to actually use the processes in their daily work. The excuses are many: the process is hard to understand, remembering all the process steps is difficult, the process is not automated or included in the tools, and many others.

The end result has been that project members use their own variant of the process, causing confusion during the project’s lifetime. This also causes severe problems, as handoffs between the development team and the operations team are often difficult. A typical bad scenario can occur when a system has to wait for deployment because the infrastructure isn’t in place for the new system. Operations was not involved (or perhaps even informed) during the project and suddenly they are expected to run the system on hardware they don’t have.

In TFS, we can implement our development process as a template that will be mandatory for all new projects. When we create a new project, we also create a new instance of the process template. We don’t have to stop at the development project level either. We can implement most parts of our ALM cycle in the template as well, enabling us to take advantage of TFS all along the way. The template helps us visualize and automate tasks and steps that the process includes. It helps us by providing document templates for requirements specs, test cases, scenarios, handoffs, and other artifacts we should produce.

The template also provides information about which reports we have available for each new project—reports that we use to retrieve information about the status of our projects and many other things. The template also contains information about one of the most important core parts of TFS: the work items. These can be adjusted as needed so we can make sure they contain the information the organization must have included with them. This information could be status information for a bug, for instance, such as Active, Resolved, or Closed.

This template is so flexible that we can develop and implement our own process, we can choose to use any of the three that Microsoft supplies, we can use a third-party template, or we can choose to customize the Microsoft templates to our own liking. We can also have several process templates in TFS so we can use different templates for different projects. Because TFS really is not used to its full potential without the process templates, we cannot stress enough that you should consider which templates you want to use and the information you want them to include.

Visual Studio 2012 Editions

Most developers will use Visual Studio to access the features of TFS. There are several editions available:

- Visual Studio Professional, which gives the developer basic functionality, is perfect for the single developer or a very small team not using TFS. It has no client access license (CAL) for TFS, but you can purchase one and get access to version control, work items, and so on.

- Visual Studio Premium is a more advanced version with a TFS CAL included. This version gives access to the most useful features of TFS like testing tools, database development tools, debugging and diagnostics, and more.

- Visual Studio Ultimate gives access to all features of TFS. Here we can get going with lab management, architecture tools, and more testing tools, to mention just some of the features.

- Visual Studio Test Professional is the tool for testers. These tools are also included in the Ultimate edition but it lacks the development tools included in the other editions.

- Team Explorer Everywhere lets developers on other platforms like Eclipse on Mac have access to TFS. We will show you more of this later in the book (see Chapter 33). This is the perfect add-on for teams with development on multiple platforms like .NET and Java.

Web Access

All projects in TFS have their own web sites available. By using Windows SharePoint Services or Server, a team project portal is created when the project itself is created. By using this portal, or the more advanced Team System Web Access, we can access most of the functionality in TFS. The project portal lets us access the parts of TFS that are available from inside Visual Studio, from an easy-to-use interface, especially for nontechnical project members. Figure 5-4 shows what Web Access looks like.

Figure 5-4.  The Team System Web Access start page on TFS

Many of our customers use a team project portal primarily to provide access to reports and documents for nontechnical people not used to the Visual Studio interface. When we want to give an external user (such as a customer or remote stakeholder) access to work item creation and editing, or another more advanced task, we usually use Web Access.

Microsoft Office

Microsoft Office can be used by project managers, product owners, or Scrum masters, for example, wishing to use tools that are familiar to them, such as Microsoft Project and Microsoft Office Excel, during a project. The integration is very nice and valuable to these roles.

Integrated Development Environment (IDE) Integration

When it comes to add-ins, one thing we should mention in particular is the Team Explorer. This tool can be used as an add-in to Visual Studio, and it gives access to TFS directly from within Visual Studio. From here you can open reports, add new work items, and run queries against the TFS database.

TFS is a flexible tool, as we have mentioned. It is also very extensible, as all functionality can be accessed via web services. This is a very nice feature that enables us to build our own support for TFS in other applications, as well. Many third-party vendors have done this, and a wide array of add-ins and tools are available. Our favorites came from Teamprise, a company that has built add-ins to Eclipse so that we can use TFS features in our Java development environment as well. Teamprise was purchased by Microsoft, and its suite of client applications has been available as Team Explore Everywhere since TFS 2010. We will learn more about Team Explorer Everywhere in Chapter 33, but briefly it offers the same IDE integration into both Eclipse and Visual Studio, allowing us to truly work as one team, no matter whether you use Eclipse or Visual Studio.

Traceability

Having traceability in our ALM processes is key to the successful delivery and maintenance of our applications and systems. In Chapter 4, we discussed a company that stopped making changes to its systems just because no one ever knew where a change (or bug fix) might have its impact. We don’t have to have such a situation.

These TFS features can help us with traceability so we can avoid such problems:

- Work item tracking

- Test-driven development/unit testing

- Automated builds/continuous integration

- Check-in policies

- Version-control system

Let’s look at some of the specifics involved with these features, starting with how the work item tracking system implements traceability.

The TFS Work Item Tracking System

Sometimes it seems like we have tons of Post-its on our monitors and desks—each one containing at least one task we are supposed to take care of. We would like to track them in a tool that could help us, but often it just isn’t possible. It could be that some tasks are connected with one project, others with another. We have tried writing them all down in an Excel spreadsheet and saving that to the computer. But soon we find that this spreadsheet is located on our laptops, our customer’s computer, our desktops, another customer computer, and so on. And we have no idea which one is the current version.

The same thing often occurs in projects. Project managers have their to-do lists for a project, and they all have their own way of keeping them updated. Let’s say a Project Manager (PM) uses Excel to keep track of the tasks—the status of tasks, to whom they are assigned, and so on. How can the PM keep the team updated with the latest to-do list? If the PM chooses to e-mail it, chances are that some won’t save the new version to disk or will just miss it in the endless stream of e-mail coming into their mailboxes. Soon there are various versions floating around, and things are generally a mess.

Work Items

In TFS, we have a task-tracking system at our service. The core of this system is represented by the tasks themselves, which as we said earlier are called work items. A work item can be pretty much what we want it to be. It can be a bug, a requirement of some sort, a general to-do item, and so on. Each work item has a unique ID that helps us keep track of the places it is referenced (see Figure 5-5). The ID lets us follow a work item, let’s say a requirement, from its creation to its implementation as a piece of executable software (component).

Figure 5-5.  Each work item has a unique ID

Work items provide a great way for us to simplify our task management in a project while at the same time enabling traceability. No more is there confusion as to which version of the task list is the current one; no more manual labor for gathering status reports on work progress that are used only at steering group meetings. Now we have a solution that lets us collaborate more easily with our teams and enables all members and stakeholders to view status reports whenever they want. We can also more easily collaborate with people outside the project group by adding work items via the web client.

TFS is so flexible in this regard that it lets us tailor the work items as we want them to be. The work item tracking system is one of the core components of TFS. This system enables us to create work items, or units of work, and can be used to enable traceability. We can use the work items included with TFS from the beginning, or we can choose to adjust these to our needs, or even create our own work item types. Each work item instance has a unique ID that we can attach to the things we do in TFS. This enables us to follow one work item—let’s say a requirement, for example—from its creation to its implementation as a piece of executable software (component). We can also associate one work item with others and build a hierarchy of work items.

The work items can contain information in different fields that define the data to be stored in the work item. This means that each field will have a name and a data type.

All work items can have different information attached to them. We can have information about to whom the work item is assigned and the status of the work at the moment (for example, a bug could be open, closed, under investigation, resolved, and so on). The State field can be modified so that each work item type can have its own state mechanism. This is logical because a bug probably goes through states different from those that a general task goes through, for instance. We can also attach documents to the work item and link one work item to other work items. We can even create a hierarchy of work items if we want. Let’s say that we implement a requirement as a work item and that this requirement contains many smaller tasks. Then we can have the requirement itself at the top and nest the other requirements below that so we know which work items belong to which requirement.

When a bug is discovered, for instance, we can quickly follow the original requirement by its work item ID and see in which places in the code we might have to make some fixes. We can also see the associated work items so that we can evaluate whether other parts of the code need to be changed as a result of this bug fix.

TFS saves information about the work item on the data tier, which helps us follow the change history of the work item. We can see who created it, who resolved it, who closed it, and so on. The information in the databases can be used for display on reports, allowing us to tailor these depending on our needs. One report could show the status of all bugs, for instance. Stakeholders can see how many open bugs exist, how many are resolved, and much, much more. It is completely up to us how we choose to use the work items.

If we implement a requirement as a work item, we can use the work item ID to track this requirement through source code and to the final build of the executable system. By requiring all developers to add one or more work item IDs to the check-in using a check-in policy, we can enable this traceability.

Configuration Management Using TFS

In any (development) organization, we need to have control of the versions of our systems we have in production. If we don’t have that, the overall ALM process will suffer, because we will suddenly lose traceability. This will make it harder to implement changes and bug fixes, because we won’t know which versions we need to update.

Without the help of a proper tool, we soon will get lost in the variety of applications we have. TFS can help us with this in many ways. After a brief description of software configuration management, I will cover three of the most important concepts that have great support in TFS and Visual Studio tools:

- Version control

- Release management

- Build management

Note  In software engineering, software configuration management (SCM) is the task of tracking and controlling changes in the software. Configuration management practices include revision control and the establishment of baselines and are very important. There are several goals of SCM, including the following:

- Configuration identification: Ensuring that we know what code we are we working with

- Configuration control: Controlling the release of a product and its changes (version control)

- Build management: Managing the process and tools used for builds

- Defect tracking: Making sure every defect has traceability back to the source

If these issues are not covered by our ALM process, we could very soon find ourselves in a troublesome situation. It is crucial for the development teams to have full control over which versions of the applications exist, which are in production, and where. This topic is closely related to the portfolio management team, and generally a big company has one or more persons devoted to keeping track of this.

Version Control and Release Management in TFS 2012

Using the version-control system in TFS, we can manage and control multiple revisions of the same information in our projects. This information can be source code, documents, work items, and other important information that we want to add version control to. When we want to work on an item under source control, we check it out to our local computer so we can start working on it. When work is done and tested, we check in our changes so the version on the server is updated.

The version-control features of Team Foundation Server 2012 are powerful. They are fully integrated into the GUI, which is something that ALM 2.0+ prescribes as well. If you want to, you can access some of the features from a project portal as well. Many people want to use the command line for their work, and TFS enables them to use the command line for working with version control as well.

However, if you do want to use Visual Studio to access the TFS version-control system, you can do that. The extensibility of TFS makes this possible. One example of this is the Team Explorer Everywhere suite of client applications that can access TFS, including the version-control system. Teamprise has developed an Eclipse plug-in that lets users access TFS from Eclipse instead. Teamprise also lets you access TFS from Mac OS X and Linux command lines. This way, you can more easily integrate different development platforms in a TFS project. You still will use the TFS repository and have the ability to get reports and other information directly from TFS.

Build Management

A build is basically the process of taking the source code and all other items necessary in an application and building it into executable software. Team Foundation Build is the build engine in TFS and executes the build process as defined by the TFS settings. Team Foundation Build is built on the Microsoft build engine (MSBuild), which is the build platform for Microsoft and Visual Studio. You can attach unit tests to a build process so that you automatically run these every time the build process kicks off. Team Foundation Build is fully integrated into the Visual Studio GUI so you don’t have to use separate tools for handling these tasks.

Team Foundation Build supports several types of builds:

- Full builds: We build everything in our project. This can be resource- and time-consuming.

- Partial builds: We build only one or more parts of the system.

- Nightly builds: Many projects build the system during nighttime. Especially if the build process takes a long time, this might be very handy.

- Custom builds: We can use the extensibility of TFS to create our own build types or edit any of the existing ones.

- Incremental builds: We build only the components that have been changed since the last build.

You can also add a number of tasks that you want to be executed when running the build:

- Get the latest source code from the version-control system

- Compile sources

- Perform architectural validation

- Run static analysis tool

- Execute unit tests

- Update work items

- Perform code coverage

- Calculate code churn (how many rows of code have been modified or added since the last count)

- Produce build reports

- Drop exe/output into predefined location

Automation of High-Level Processes

Without one or more templates, TFS will not be used to its full potential, as you saw earlier in this chapter. You could still use its version-control system and some other tools, but the real value comes from using it to automate your ALM process. In the process template, your whole ALM process is defined.

The template defines the following:

- Work item types: Which work item types are necessary and what information they should have attached to them. You can also define the workflow for a work item. For a bug, you might have different states the item flows through, such as Active, Resolved, Closed, and so on.

- Project phases: By using areas and iterations, you can define the initial project phase setup of your projects. If you use RUP, you can define the process phases in that model, or you can create the first sprints of a Scrum project. Areas and iterations are flexible, and you can create your own way of working through these concepts.

- Document structure and templates: The number of documents that should be produced during a project will differ depending on your process model. In the process template, you define the document structure you need and the templates you should use. For instance, you can add templates for requirement specifications or acceptance testing here.

- Reports and queries: In the process template, you can specify which reports and work item queries you need to have as defaults in your projects. You probably want reports and queries showing the progress of your project, such as the status of bugs or work remaining. You can create your own reports by using SQL Server Reporting Services or Excel and add them to all projects by adjusting the process template.

- Security: The template also adds information about which users or user groups have access to what information. You can connect TFS groups to your Active Directory accounts, for instance.

The process template is the overall process for our ALM implementation. Many of our customers create different templates for different kinds of projects. They also create templates for operations, so that when a development project is finished and deployed, the operations staff can use their template to run the system until the system dies. A few customers have started creating a template for Information Technology Infrastructure Library (ITIL), for instance, and we are looking forward to seeing the result of that work.

It is important to remember that you can adjust the process to your needs. You should consider changing the default templates or even replacing them, rather than adjusting your own way of working to the templates that come with TFS out of the box. Microsoft enables this flexibility by letting you easily access the process templates to adjust them or to add new templates.

Visibility

Information about project status is important to all participants of a project—and we don’t mean team members only, but stakeholders and decision makers as well. As project managers, we have spent too much time chasing down information to answer questions about the status of projects, how much work remains, and what the latest bug status is.

TFS provides two primary ways of enabling visibility:

- Reports: Reports are created by using SQL Server Reporting Services and accessing the TFS data tier directly. You can define and adjust these as you want. You can also use Excel to create reports if you prefer. Reports can show various kinds of information, and we will look at them in detail later in the book (see Chapter 32).

- Queries: Queries are used to ask questions of the work item tracking service. One question might be how many bug work items you have. How many and which are dedicated to me? How many bugs are there? And so on. You can create new queries when necessary.

By using these two components, it will be easier to gather the information you need for your status reports for a steering group meeting or project meeting. You won’t have to look around in several places and in several applications for this information anymore; instead, you can use the automated reports and queries from inside TFS.

Project owners, project managers, and Scrum masters will certainly benefit from TFS. Because TFS has all data in the same repository, you can more easily retrieve the correct information when you want it. The flexibility of the SQL Server database that stores all information is great. You can work with the data warehouse information just as you would with any other database.

By using the project portal or Web Access (see Figure 5-6), you can publish information (in the form of custom-built controls that we as users cannot change at this time) so that everybody who has privileges can see them. This is an easy way to make sure that information is available all the time. Just this little, relatively nontechnical improvement will off-load work from the project manager, freeing some of the PM’s or PO’s time for better things.

Figure 5-6.  Viewing reports from Team System Web Access

Collaboration

As you know, TFS comes with Team Explorer, which is an add-in to Visual Studio. With this tool, the developer can access every aspect of a TFS project. The developer can view reports and queries, for instance, as well as access the document in the project. The developer can access the version-control system as well as build systems, tests, and so on.

The Team Explorer is full featured but is still a tool for people used to working in Visual Studio. For us that is no problem, but for most project managers and stakeholders, the GUI is confusing. They want to have an easier-to-use tool to access the relevant information.

Each project that is created with TFS has a project portal created as well. This portal gives us access to reports, documents, project process guidance, and other project-related information through a web interface. This enables people who are not used to the Visual Studio interface to easily retrieve the information they need.

Collaboration, of course, does not only mean giving access to information, even though this is as important as any other means of collaboration. Collaboration also means that we should be able to work together to fulfill one or more goals.

Work Items for Collaboration

You can use the work item features of TFS to enable your process workflows. Let’s say a project manager, or anybody responsible for inputting requirements as work items into TFS, creates a new work item of the Scenario type. This scenario should probably be assigned to a developer to implement. The project manager uses the work item system to assign (see Figure 5-7) the scenario to a specific developer, in this case Joachim. Joachim continues to work on the scenario until it is ready for testing. He then assigns the work item to a tester who performs the testing. When the testing is done, the work item is perhaps closed. If a bug is found, either the tester or anyone finding the bug can use the work item tracking system to see who developed the scenario implementation and reassign it to that developer, in this case, Joachim again. TFS keeps track of who has worked on the work item so that you don’t have to manually keep track of this.

Figure 5-7.  Assigning work items to a specific person

The Gap Between IT and Business

Closing the gap between IT and business is obviously a very tough problem to solve. TFS won’t get us all the way, that’s for sure. We don’t think any tool ever will because so much depends on the people in the organizations, which is an important consideration. But tools can help us bridge the gap, so you should carefully consider how you can use them for this. We need to improve on our ALM process and way of working to start solving this. When we have a new way of working, TFS can support much of our efforts using, for instance, the process template to implement this new way of working.

The gap between the IT and business sides is often a question of work process. It requires considering many things, and when you have a solution or start working toward a solution, you must evaluate what parts of this work process you can automate and use tools for solving. One thing worth mentioning here is that the use of the TFS Project Server Connector with TFS lets you integrate TFS with Microsoft Office Project Server. Having this integration will allow you to better control your resources and better automate this process as well. This way, you can align your portfolio management process better so that you can choose which things to work on more effectively.

Office/MS Project Integration

When we have run projects in the past, we have mostly used Microsoft Office Project to handle project planning, especially the Gantt diagram. We suspect that this is the case for many of our fellow project managers, as well. In many cases, we have used this product not primarily because of the tool itself but because so many of our customers use Microsoft Office that it becomes natural for them to also use Project. Project has its strengths and weaknesses, as all tools do, and we cannot say that we don’t like it, but we have never become friends with it. Sometimes it does things that we don’t expect, and even though we know this is because we are not very familiar with its features, we still blame the product from time to time—unfair, but that’s life sometimes.

Excel and Project are two tools that most companies use on both the business and the IT sides of the company. By being able to use these tools, business people can more easily be a part of the ALM process, because they can use a tool they are already used to working with. A nice feature here is that the communication between Office and TFS is two-way. This means that an update in TFS will be reflected in Office and the other way around. This allows for a dynamic way of working with TFS information.

Use of One Tool/Role Based

A good ALM tool should enable you to use add-ins that will provide new features inside one interface. If a developer needs testing features, you should be able to integrate them into the development tool. The developer should not have to switch tools to do testing tasks. This is also what Visual Studio offers. There is no context switching as team members can use the same GUI no matter what role they are performing at the moment. TFS is also extensible and lets you create your own add-ins as well as purchase third-party add-ins that will be accessible from inside of TFS.

Extensibility

When the built-in features of TFS are not enough, you can use the extensibility features to expand and enhance it. TFS is often seen as a closed black box that Microsoft ships, when it’s more like an enterprise resource planning (ERP) system for ALM. Any ALM environment must be customized for an organization’s processes and the existing applications and services.

Many of our customers have been a bit reluctant to customize TFS. They have instead tried to squeeze their way of working into the templates Microsoft provides with TFS. We think this is the wrong way to do it. Our suggestion is that you start the other way around. Start by asking yourself how your organization wants to work. This process involves all parts of the organization, from the business side to operations. Try to find agreement on how to work in the ALM process. By doing so, you will see that this also is a good start for collaboration in the company.

For instance, consider the work items and the information in them. If the fields and information in the MSF templates are not enough, extend or edit them. TFS lets us do this by changing the process template. You can choose to add the information that you need, and it will be stored in the TFS databases, so you can have access to it from within your reports and queries. Don’t forget to change the reports or queries, as well; otherwise, you will not see your information.

Some of our customers have changed the workflow of a work item by adding more states to it, when the ones supplied have not been enough. Often we have used the TFS Power Tools to do this.

When you have an initial idea of how you want to conduct the ALM process, start looking into what TFS gives you out of the box. Use what can be used, change other things, and build your own solution where needed.

One great strength of TFS is its extensibility and flexibility. You can adjust the whole tool to fit most parts of your ALM process. If you want to, you can develop your own add-ins by giving support to roles not included from the start. We strongly encourage you to use these extensibility features, but in the end it is your choice.

Extensibility is a great way to integrate existing systems and potentially migrate some of them into TFS in order to reduce the toolset in the organization.

Summary

In our opinion, Team Foundation Server can help us implement a good, automated, and robust ALM process. There are features for all aspects of ALM 2.0+. Correctly used, these features will help us improve our ALM process, which in the end will give us better business value and more-successful projects.

The three pillars of ALM—traceability, process automation, and visibility—are all important for any organization to have. TFS is a great foundation on which to build our ALM solutions. TFS has work item tracking for traceability, process template implementation in the tool itself for process automation, and reports and queries for visibility. Through a project portal, accessible via the Internet, you can improve collaboration between all parties having an interest in your projects.

TFS is role based in the sense that it supports different development roles. It has support for architects, developers, DBAs, testers, and more. They are not separate tools either, but they are all accessible from a unified GUI. You can also add custom add-ins to the GUI and do not have to use several tools to get the job done.

Product owners and project managers have the capability to use tools they are already familiar with. Most use Excel or Project for project planning, and there is integration between these tools and TFS. We can easily sync information among these tools.

The extensibility of TFS makes it fairly easy to write your own code integrating TFS with other applications. This is an incredible strength of TFS, and something we should give Microsoft credit for.

So, all in all, Team Foundation Server is a great foundation on which to build your Application Lifecycle Management process.

PART 2

Planning (Agile Project Management)

Part II covers agile project management using Team Foundation Server (TFS) 2012. We first take a look at how we can use Scrum and some agile practices to run our projects in an agile way.

Then we continue to look at how the Work Item Tracking (WIT) system works in TFS 2012. We show you how you can adjust the process templates in TFS so that they better suit your organizational needs. It is important to understand how you can make TFS work for you and not the other way around.

Next we move on to using TFS 2012 to implement the agile process using the Scrum process template included with TFS as a foundation. We follow the fictitious organization Kangreen in their startup of the first agile project in their company.

CHAPTER 6

Introduction to Agile

In Chapter 3 we took a look at different development processes and frameworks. Our experience is that we have seen a great deal of improvement in projects over the last few years. To be more specific, we have seen that the agile influence is making an impact in how our projects are delivering business value.

The focus of this book, when it comes to processes and frameworks, is on Scrum and XP. Partly because we like these ourselves, and partly because Microsoft focuses hard on implementing support in Visual Studio and TFS for these practices. Microsoft for example, includes a great template for running Scrum projects straight out of the box. We will take a close look at this later in this section.

This chapter looks at how we can use Scrum as an agile project management model to deliver software using TFS. We cover the scrum process a little deeper than in Chapter 3, adding how we in practice could use scrum and agile practices such as agile estimation and planning in combination with TFS.

This chapter does not cover eXtreme Programming (XP)—not because it is not useful, but because XP practices are covered in the more technical chapters later in this book. This chapter focuses purely on the project management features for scrum included in TFS 2012.

The Scrum Process

Figure 6-1 shows the Scrum process. The requirements from the business side of an organization are put on a backlog as Product Backlog Items (PBIs). The backlog itself is an ordered list with the (currently) most important requirements at the top. When the first sprint starts, the development team, together with the PO, selects a number of PBIs for the sprint backlog (SP) in the sprint planning meeting. The team commits to delivering these Sprint Backlog Items (SBIs) and starts working on them.

Figure 6-1.  The scrum process

A sprint is usually lasts between 2–4 weeks and is divided into 24-hour increments (working days). Every day the development team and the scrum master (SM) meet in a daily scrum meeting going over the three magic questions:

- What have you done since the last meeting?

- What will you do until the next meeting?

- Do you have any impediments stopping your work?

The end result of a sprint should be a potentially shippable increment of the software. At the end of each sprint there are also two meetings:

- Sprint review. During this meeting the team shows the PO and maybe stakeholders what they have done. The PO signs of on the delivery (unless something has not met the expectations).

- Sprint retrospective. During this meeting the team assesses what was good, what can be improved, or what needs to be changed for the next sprint.

That’s it. No more than that. Seriously. The process is extremely easy to use and learn, but is hard to master. But as we said we have seen great improvements in our companies and at with our customers’ projects taking an agile approach compared to a traditional project approach using waterfall or RUP.

Let’s take a look at the roles in scrum and what their responsibilities are. Many of the following sections of this chapter are short and concise. Keep in mind that there are books and trainings covering these topics, so this book does not give you everything. What we aim to do in this part of the book is to exemplify how TFS can support agile project management, and to do that we need to know a little bit more about some important concepts.

Roles in Scrum

As you will remember from Chapter 2 there are only three roles in scrum:

- Product owner

- Scrum master

- Development team

Together these three roles create the scrum team. In the next section of this chapter, we examine in more detail what responsibilities lie within these roles when actually planning and running a scrum project.

We will take a closer look at these starting with the Product Owner, PO.

Product Owner

The product owner is the role that most equals traditional project managers (PM). The full truth is that the responsibilities of the PM have been divided between all three roles in scrum, but a good part has landed on the PO role. The focus right now in this chapter is on the PO role because the scrum master’s and development team’s responsibilities are covered all through the book. Much of what the PO is responsible for ends up in TFS—especially in the beginning of the project. Here are a few things the PO is responsible for:

- What to build and who should create the vision for the project. It is only the PO that decides what the project should build and deliver. The PO has final say in all decisions regarding the “what” question. This is not always the case because reality might get in the way, but things will be much harder if the PO does not have the authority to make all decisions.

- Project delivery. A PO can never say that the development team did not build what the PO wanted. If that situation occurs, the PO has done a bad job and probably not been as present and dedicated as he or she should have been.

- Requirements and estimation. When the need for a new system (as an example) comes up in the organization, the PO should be responsible for gathering initial requirements and estimating these. We look at this in more detail later in this chapter.

- Creating the initial backlog and keep on grooming it. Based on the requirements, the PO creates the initial backlog. During the project, the PO is responsible for keeping the backlog in good shape (also known as backlog grooming). Included here is the art of breaking down the backlog into manageable pieces, which is something the PO does with the help of the rest of the team.

- Prioritizing the backlog. For the team to build the right thing it is important that the backlog is ordered in some way. By prioritizing the product backlog, the team knows what tasks to take on first. It is the PO that is responsible for this prioritization. The PO can probably not do this by himself so the team is welcome to help and give input.

- Calculate the estimated budget and ROI. With initial requirements and estimation done, the PO can calculate the estimated budget and Return On Investment (ROI) of the project so that he or she can convince stakeholders that the project is necessary to carry out.

- Product management. The PO should know what and why we build something. All requirements and requests coming into the project should be filtered through the PO. After looking at the incoming requests, the PO decides what to place on the backlog. Nobody else decides that.

- Stakeholder management. The PO is of course not alone with all these responsibilities. The PO needs to manage all stakeholders and end user input so that he or she knows what the organization wants; otherwise it is hard to make the decisions. One way of doing this is to schedule repeated stakeholder meetings where the needs and priorities of the organization are discussed.

- Release management. It is important early on in a project to get an overview of the releases in the project. Do we have a specific theme that drives a release or a certain set of functions that should be a release? The PO should make sure this is done early on and then follow up this.

- Team manager/Staff project. The PO is responsible for staffing the project. Initially, the PO needs experienced people that can help with requirements and initial estimation. These people should then follow the project until it ends in the best of worlds. Once the project starts, the PO makes sure it is scaled up in the best possible way.

These are some of the PO responsibilities. We will soon see how much of these are mapped into TFS. First, let’s look at the other two roles.

Scrum Master

The responsibilities of the scrum master (SM) include:

- Protecting the team. The team should be able to work without being interrupted by anything. For example, the team should not be disturbed by managers asking them to spend time on other (non-project) related tasks. If that happens the SM needs to explain to the manager why the team (or team member) cannot do the things the manager asks for. This is essential for the team to deliver what they have committed to deliver.

- Resolving problems. If the team or a team member is blocked by an obstacle and cannot continue working on a user story it is the scrum master that needs to resolve the problem. The SM needs to clarify what the problem is, why the block happened, and how it can be solved.

- Making sure the team (including PO) understands and complies with the scrum process in the right way. It is also the scrum master that needs to make sure that the rest of the organization outside the scrum team understands the process and why it is important to stick to it.

The Development Team

So now we have come to the team, the ones producing the actual code. Here are some responsibilities of the development team:

- Deciding how to build what the PO has decided to build. The team is responsible for coming up with the solution, breaking down user stories, and giving feedback and suggestions to the PO and a lot more.

- Delivering quality code. This means that they need to comply with the requirements in the Definition of Done (DoD), which we will discuss later in this chapter.

- The estimates of user stories, both before and during a sprint. The team needs to estimate as the best they can to give input to the PO. They also need to have a good dialog with the PO so that they can point out alternative ways for the project. If they feel that the PO should make another decision, they are obliged to point out and argue for a better way.

- Following the principles of XP. This might not be in the Scrum guide, but we recommend this from our own experience.

Definition of Done

Definition of Done (DoD) is something really important, but also something we often tend to forget. (Do not confuse Definition of Done with Department of Defense! That is something entirely different and way out of the scope for this book.) In many projects we have seen arguments between the delivering development organization and the person ordering the project about whether a task has been done at the end of (and also during) a sprint or project. It could be that testing has not been done the way that the client assumed it would have been done or that the software does not comply with certain regulations. The following conversation is typical:

- The product owner Sofia stops by the developer Mike to check on how things are going.

- S: “Hi. How’s the new cool feature you are working on coming along?”

- M: “It’s going great. I am done with it right now and will start the next feature soon.”

- S: “Great! Then I can show it to our customer who’s coming here after lunch. He will be very excited!”

- M: “No, no. Hold on. I am not “done” done with it. I still need to fix some test cases, do some refactoring, get it into the build process, and so on. I just thought you wondered if I had gotten somewhere with it. . .”

For the most part this argument could have been avoided if they had sat down together in the beginning and written and signed a DoD.

There are other reasons for having a DoD as well. In order for the team to estimate a user story they need to know when they are done with it. Otherwise, it is very hard to complete the estimate. For a specific user story we know it is done when we have fulfilled the acceptance criteria for it. But where do all those general things like style guides, code analysis, build automation, test automation, regulatory compliance, governance, non-functional requirements, and so on, fit in? They affect the estimate of a user story as well.

Here is where the DoD comes into place yet again. The DoD tells us what other requirements besides the acceptance criteria of the user story itself that we need to fulfill in order to be done with the story. We include the general requirements into the DoD because they affect all user stories in the end.

We can say that Definition of Done is our primary quality document. If we do not fulfill what is in it, we do not deliver quality. It is essential that the PO and the team agree on the DoD. The DoD is part of the agreement between the team and the PO.

There should not be an argument over this concept during the project. If the PO thinks it is too costly to use pair programming or Test Driven Development, have him or her sign the DoD where you specify that these things have been removed. If at the end of a sprint the PO complains about the number of bugs, just present the document and say that the PO has removed essential parts of the testing and hence bugs will be present.

Agile Requirements and Estimation

This topic is huge, but important. We will cover some of the most important topics here and later show you how they are implemented in TFS. But if you want to master this, there are several trainings you can take and books to read.

Most of the agile planning and estimation tips and tricks of this chapter come from the agile community but are not specific to Scrum. Scrum really does not tell us how to do specific things like planning, estimation, and so on. Scrum is the process framework or process method we use for running our agile projects. However, Scrum works excellent together with these concepts we look at now.

Requirements

In agile projects we usually represent our requirements in something called user stories. These can be looked upon as fluffy requirements—a little bit like use cases actually. We write user stories like this:

- As a < type of user > I want < some functionality > so I may have < some business value>

One example could be:

- “As a manager I want my consultants to be able to send in expense reports through the Internet so that we can be more efficient in our expense report process.”

Figure 6-2 shows how Microsoft has implemented the user story into the work item type Product Backlog Item. The terminology is a little different from my previous description, but it works.

Figure 6-2.  The user story implementation in the scrum template Microsoft provides with TFS

User stories capture requirements at a high level and will not be tangled up with detailed functions or implementation details. The details and nonfunctional requirements are instead captured as acceptance criteria for the user story. Based on these acceptance criteria, we can also develop acceptance tests at the same time we write the requirements.

The DoD is also important here because it describes other important requirements all user stories need to fulfill before they are done.

So, how can we go ahead with gathering requirements before we start a project? The PO should use any method he or she thinks is suitable. We often use story-writing workshops where important stakeholders, end users, business analysts, experienced developers, and others may participate to brainstorm the user stories they can think of. During such a workshop we focus on the big picture and do not dive down into details. These big user stories are often called epics because they are big and not broken down into details yet.

But don’t we need to find all requirements at the beginning? No. And that is what makes agile so great. The agile concept builds on the fact that we acknowledge that we don’t know and cannot know all the requirements early in the project. New requirements and changes to early requirements will pop up all through the process, and that is okay The agile approach takes care of this for us. We start with what we have initially and continue handling requirements all through the project. So the short version is to get started right a way and be aware that changes and new requirements will come.

When the initial requirements are done, we have the embryo for the product backlog. However, before we can prioritize and estimate these user stories we need to perform a risk assessment of them so that we can get a grip on any risk associated with each and every one of them. A user story with a great risk associated with it usually takes more effort to finish and should probably be done early in development.

Estimation

To know the effort involved with a user story we need to estimate it. The sum of all initial estimates gives us a (very) rough estimate of how much time the whole project might take. But because we know things usually change over time we do not take this estimate as if it was written in stone.

So, how do we do estimation? We have what we need to do this, we know the requirements, we have a DoD, and we have acceptance criteria.

In the agile world it is recommended to estimate time in something called story points. Story points are not an exact size—instead they are relative.

Here is an easy example we use when running agile training. Take four animals, for instance. Let’s say a cat, a pig, a zebra, and an elephant. Without being a zoologist, most persons can say that the pig is three times the size of the cat, the zebra is twice the size of a pig, and the elephant is maybe four times the size of the zebra. If we sit down a couple of people and discuss these animal sizes, we pretty soon can come up with an agreement on their relative sizes.

The same goes for user stories. Most developers will come up with an agreement pretty quickly about the relative size of user stories. User story A is twice as big as user story B, and so on. We do not need to be very experienced in the details of each user story to come up with this agreement. Novice developers usually end up with the same estimates as experienced. Keep in mind that we are not talking exact time yet, only relative size.

The most common scale for expressing story points is a modified Fibonacci scale. This scale follows the following sequence: 1, 2, 3, 5, 8, 13, 20, 40, and 100.

Very often teams use a technique called planning poker when doing estimates. The deck of cards each player has contains the numbers from the modified Fibonacci scale. Here is how it goes:

- PO/SM reads the first user story.

- The team briefly considers the user story and selects a card each, without showing it to the others.

- The team shows their cards at the same time.

- If the result varies much, the person with the highest and lowest card explain how they reasoned.

- After a short discussion the team plays again.

- When consensus is reached (or the team members are only one step from each other), you are done.

- If you still disagree, the team should pick the highest value.

But what about time then, someone asks. How can we get down to time? There are several things that we need to know to estimate time. The first is the team capacity. Consider the following when calculating the team capacity.

- How long is the sprint?

- How many working days are available in the sprint?

- How many days does each team member work during the sprint? Planned vacation or other days off, planned meetings, and so on.

- Deduct the time for Sprint Planning, Review och Retrospective meetings.

- The result of this is the capacity before “drag” (drag is waste time or unknown activities).

- We should measure drag in each sprint but at the initial planning we really don’t know how much we should calculate with. The longer the project, the more accurate the drag will be.

- If you don’t know from experience what the drag is, 25 percent can be a good landmark. Included in this is 10 percent backlog grooming.

- Now we have the available number of hours in the sprint.

We now should connect points and time. We need to know the team velocity, which is the number of story points the team can handle in a sprint. Initially this is impossible to know. The easiest way to figure it out is to perform a sprint planning meeting. This is the meeting where the team breaks down a user story into manageable tasks. And this is where time becomes interesting. During this meeting the team estimates the tasks in hours so that they can plan the sprint and decide on how many user stories they can take on in their sprint. This is usually the way team performs this:

- Estimate the first user story in detail.

- Break down what the team needs to do to deliver the story.

- Estimate hours for each activity and summarize.

- Deduct the summary from the available time the team has in the sprint.

- Is there still time left?

- Take a new user story and repeat the process until no available time is left.

- Summarize the number of story points from the stories that were included in the sprint.

- Now we have a theoretical velocity.

Now that we know the velocity of the team we can make a rough time plan for the entire (at that point known) project. This is good input for the PO in his or her discussions with stakeholders, and also input for ROI calculations.

The sprint planning process then continues all through the project and the theoretical velocity can soon be replaced with one based on experience instead. We will come back to this several times later in the book, starting in Chapter 8.

Backlog

When the initial user stories are in place and estimated with story points, the PO can start prioritizing the backlog. In scrum this is called ordering the backlog. Based on the business needs, the PO makes sure that the order of the backlog reflects what the business wants. In Figure 6-3 we can see the initial backlog we used for writing this book. We did a rough estimate on each backlog item and then a velocity planning. After that we could see what backlog items should be completed during which sprint (two-week sprints were used).

Figure 6-3.  The backlog we used for writing this book

The PO needs to keep the backlog in good shape all through the project. This means that it needs to be ordered. It also needs to have fine granularity at the top (maybe 3–4 sprints down the list) and rougher granularity further down.

The PO can also start to look at release planning at this point. It is important to get an overview of coming releases in the project especially if you have a larger project. Release planning can be done on the epics, the larger user stories. A good way is to look for themes among the user stories. What could be useful to release at the same time? If we find such features, we could make a theme of it and plan the theme for a certain release.

When this is done we could also do a very rough time estimate on the releases and suddenly we also have a rough time plan for the project.

Scaling Scrum   The PO is responsible for staffing the project. He or she should select an initial team that will help the PO during user story gathering and estimation. The team should consist of experienced persons (developers, business analysts, and so on) that ultimately should participate in the entire project.

Two to four persons in the team might be good at the initial phase, but when the project starts this should expand to the number needed (seven plus or minus two persons).

If need arises for more than one team, one or two from the initial team should participate in the new team. So, start small and expand as needed. We have seen projects starting too big and fail, so do not fall for that.

It is also good if the scrum master can join the team early.

Now we have as much information as we could possibly ask for this early in a project. The next step will be the sprint planning meeting when the team (as we showed earlier) has a planning meeting and selects the backlog items they feel they can commit to during the sprint.

Summary

We can now start setting up the project in TFS. Once we have a backlog, we can start adding the stories into TFS. We can also start adding the team members to the TFS project. We will have a closer look at this in Chapter 8.

Chapter 8 we also looks at how we can support the agile process during the sprint(s). We will see how not only the PO benefits from the agile project management features of TFS, but how the SM and the team also can use these features to enhance the work.

CHAPTER 7

Work Item Tracking and Process Customization

Traceability in our ALM processes is key to the successful delivery and maintenance of our applications and systems. In Chapter 2 we saw that traceability is one of the three cornerstones in a successful ALM solution.

- Traceability of relationships between artifacts. If not done correctly, traceability can be a major cost driver in any enterprise. There must be a way to trace the requirements all the way to delivered code—through architect models, design models, build scripts, unit tests, test cases, and so on. Practices such as test-driven development and configuration management can help, and these can be automated and supported by TFS.

- Automation of high-level processes. There are approval processes to control handoffs between analysis and design. There are other handoffs between build, deployment, testing, and so on. Much of this is done manually in many projects, and ALM stresses the importance of automating these tasks for a more effective and less time-consuming process.

- Visibility into the progress of development efforts. Many managers and stakeholders have limited visibility into the progress of development projects. Their visibility often comes from steering group meetings during which the project manager goes over the current situation. Other interest groups, such as project members, may also have limited visibility of the whole project even though they are part of it. This often occurs because reporting is hard to do and can involve a lot of manual work. Daily status reports can quite simply take too much time and effort to produce, for example, especially when we have information in many repositories.

Let’s now see in more detail how work items in TFS help us accomplish traceability in our projects and organizations.

Traceability

Unfortunately we have seen companies that stopped making changes to their systems just because no one ever knew where a change (or bug fix) might have its impact. This is not a situation any organization wants to end up with—yet it is quite common.

At the Swedish Road Administration some years ago, a new version of our system suddenly made old bug fixes disappear (commonly known in testing circles as a regression). The operators at the Traffic Management Center found themselves with no working phones because of an upgrade. This had the potential to make an accident worse than it already was, because the operators communicate with the rescue team and the police using phones. Having communications suddenly stop working can actually be a matter of life or death.

The vendor of that piece of software did not have control over its different software versions and did not have a good testing strategy. If the vendor had used automated tests, for instance, they would have discovered broken tests for the bug fix when the fix itself was not included in the next release. By checking which work items were associated with the failed test(s), the vendor would have been able to see which of these contained the problem. This would have indicated why they created the test in the first place, so they could have more easily fixed the problem. This traceability would have greatly improved their code.

Using a method like Test Driven Development (TDD), where developers write coded tests for their code, would help a lot in avoiding these situations. By automating these tests and running them with every build or check-in, we can achieve very good regression testing without having to break our backs doing manual labor-intensive testing every time we change the code. Furthermore, a model like CMMI (see Chapter 3) could also prevent the organization from loosing such good practices and knowledge, on the other hand, so could any agile practice as long as we have ways of dealing with quality issues.

And if they had used a good configuration management process, they would also have had the capability to trace all versions where the bug fix needed to be inserted, so they wouldn’t forget to include it in the coming releases.

Work item tracking in TFS can help us with traceability so we can avoid such problems. Let’s see how the work item tracking system implements traceability.

The TFS Work Item Tracking System

Sometimes it seems like we can have tons of Post-it notes on our monitors and desks—each one containing at least one task we are supposed to do. Often it just isn’t possible to track them with a tool. It could be that some tasks are connected with one project, others with another. We could try writing them all down in an Excel sheet and saving that to our computer. But soon we might find that the spreadsheet is located at our laptop, the customer computer, the desktop, at another customer computer, and so on. And we have no idea which one is the current version. This can be a real problem sometimes when we find we have no clue as to which version we should trust.

The same thing is often visible in projects. Project managers have their to-do lists for a project, and they all have their own way of keeping the lists updated. Let’s say a PM uses Excel to keep track of the tasks—the status of tasks, whom they are assigned to, and so on. How can the PM keep the team updated with the latest to-do list? If the PM chooses to e-mail it, chances are that some won’t save the new version to disk or will just miss it in the endless stream of e-mails coming into the mailbox. Soon there are various versions floating around, and things are generally a mess.

One way to solve this could be to use a project web site running on Microsoft Office SharePoint Server or some other similar tool. This could help, although we could still be in trouble if people forget to save changes or check in the document after they have updated it.

Another problem may occur if, for example, an Excel sheet is updated by a tester who discovers a bug and changes the status of one entry in the task list to indicate that a developer should look at the task again and solve the bug. How can we alert the developer that the bug exists? We would want this action to take place automatically, right? That would be hard if we used only an Excel sheet. The same thing occurs the other way around. When a developer has fixed a bug, we want the tester to be alerted that the problem has been resolved, so the tester can then check whether the bug can be closed.

What about requirements traceability? If the only place we keep track of the connection between requirements and the code is in a document, how do we know that the document is really updated? Can we trust that information?

Even if we purchase a separate tool to help us keep track of tasks, it would still be a separate tool for each category of team members. There are tools for bug tracking, requirements management, test management, and so on—the list can go on for a while. Chances are that someone will forget to update the tool because it takes too long to open, is too difficult to work in, or many other excuses for not doing the update. This could cost the project lots of money and time.

Work Items

In using TFS we have a task-tracking system at our service. The core of this system is represented by the tasks themselves, which are known as work items. A work item can be pretty much whatever we want it to be. It can be a bug, a requirement of some sort, a general to-do item, and so on. Each work item represents an object that is stored in the Team Foundation Server database. Each work item has a unique ID that helps us keep track of the places it is referenced (see Figure 7-1).

Figure 7-1.  Each work item has a unique ID

The ID lets us follow one work item, let’s say a requirement, from its creation to its implementation as a piece of executable software (component). Work item IDs are unique across all work item types in all team projects in a project collection. The work item type determines the work item fields that are available for tracking information, defaults defined for each field, and rules and constraints positioned on these fields and other objects that specify the work item workflow. Every change made to a work item field is stored in the work item log, which maintains an historical record of changes (http://msdn.microsoft.com/en-us/library/dd286718.aspx).

You can create and modify work items by using Team Explorer, Team Web Access, Office Excel, or Office Project. When creating or modifying individual work items, you can work in the work item form by using Team Explorer (see Figure 7-2) or Team Web Access. You can make bulk updates to many work items at a time by using Team Web Access, Office Excel, or Office Project.

Figure 7-2.  Creating a Work Item using Web Access

Work items provide a great way for us to simplify our task management in a project while at the same time enabling traceability. No more confusion as to which version of the task list is the current one. No more manual labor for gathering status reports on work progress that are used only at steering group meetings. Now we have a solution that lets us collaborate more easily with our teams and enables all members and stakeholders to view status reports whenever they want. We can also more easily collaborate with people outside the project group by adding work items via the Web.

TFS is so flexible that it lets us tailor the work items, as we want them to be. By installing TFS Power Tools we get an additional menu option called Process Editor under Tools in Visual Studio (see Figure 7-3), which simplifies editing the work items and the whole process as well. From this tool we can modify our work items in the project so they contain new information. Later in this chapter you will see more about how we can change our process template, including the work items. If you make a change to the current project (by modifying a work item, for example) this affects all new work items we create, not the existing ones. We only get the change in our current project as well. All new projects created with the same process template will not have these changes, unless we modify the process template on the TFS server.

Figure 7-3.  Modifying a Work Item using the Process Editor

The work items can contain information in different fields that define the data to be stored in the work item. This means that each field will have a name and a data type. Data types supported in fields are the primitive data types such as string, integer, and double, as well some complex types such as DateTime, PlainText, HTML, and others. System fields are one example of a field (or more correct, a label for a group of fields) that must be present in every work item type, and represent the minimal recommended subset of fields that any custom work item template should contain. Having such a common subset allows reusing basic Work Item Query Language (WIQL) queries or reports from predefined templates for your custom templates.

All work items can have different information attached to them. We can have information about to whom the work item is assigned and the status of the work at the moment (for example, a bug could be open, closed, under investigation, resolved, and so on). The State field can be modified (see Figure 7-3) so that each work item type can have its own state mechanism. This is logical because a bug probably goes through different states than a general task goes through, for instance. We can also attach documents to the work item and link one work item to other work items. We can create a hierarchy of work items if we want. Let’s say that we implement a requirement as a work item and this requirement contains many smaller tasks. Then we can have the requirement itself at the top and nest the other requirements below that so we know which work items belong to which requirement.

When a bug is discovered, for instance, we can quickly follow the original requirement by its work item ID and see in which places of the code we might have to make some fixes. We can also see the associated work items so that we can evaluate whether other parts of the code need to be changed as a result of this bug fix.

Because TFS saves information about the work item on the data tier, we can see the history of the work item. We can see who created it, who resolved it, who closed it, and so on. The information in the databases can be used for display on reports, allowing us to tailor these depending on our needs. One report could show the status of all bugs, for instance. Stakeholders can see how many open bugs exist, how many are resolved, and much, much more. It is completely up to us how we choose to use the work items.

Those of us familiar with pivot tables can use Excel to drill down into the information in the TFS data warehouse. There are people who think it is better to use Excel to directly connect to these tables and who use very detailed information in their reports.

The Work Item Form

The work items are defined in the project template in TFS. The template—and thus the work item types—are defined in a set of XML files stored on the TFS server. The XML file(s) for our work items define what information the work item will include on its form in TFS (see Figure 7-4).

Figure 7-4.  The Bug form in Microsoft Scrum

As you can see in Figure 7-4, the Bug work item type in Microsoft Scrum includes fields for many aspects of the bug. We can assign the bug to a specific person, set state (status), set severity, and much more. We can also add a description of the problem and attach files such as screenshots of the bug. There are other options as well, but we will not cover them here.

The fields on the work item form can have properties set for them. We set a field to be read-only, required, automatically populated, and so on. Because we can also change what information is included on this form by editing the XML, we can include the information that we want.

Some customers say they have had problems using the process templates that Microsoft provides because the information required to fill in the forms is not the information they want to track or record. Instead of changing the work item types, they have tried to adapt to the work items. Don’t make this mistake! If you need other information besides what is included in the templates, or if you need the information in another way, change the template. That’s the whole point of having an open and flexible solution such as TFS. You can adjust the tool to fit your needs. I have, for instance, seen the Bug work item that Microsoft uses and it looks nothing like what is included in any of the templates you get with TFS. Instead Microsoft encourages you to adjust the tool to your needs. This includes adjusting the work items.

Work Item Traceability

Let’s look at an example of how we can use Work Items to increase traceability. We start with a requirement in the form of a User Story (see Chapter 8 for more information):

- “As a Manager I want to Search Expense Reports so that I can easier get an overview of expenses”

This user story is entered into TFS and TFS assigns an ID to it (see Figure 7-5).

Figure 7-5.  Traceability starts with a work item in TFS

This ID will follow the work item all through its life. Figure 7-6 shows we can already associate the work item with storyboards, test cases, tasks. This means that we can get traceability from a requirement to test cases, to storyboards and to other work items. In fact, we can associate not only work items to other work items but to any work in our project, including documents and source code. Storyboards in this context are created using PowerPoint, and those files would typically be stored in a SharePoint document library. To link to the storyboard we only need to provide the link to the document in SharePoint.

Figure 7-6.  Linking work items to storyboards, Test cases and Tasks (other work items) enables us to reach great traceability

This is far better than keeping this information in our heads or on an Excel spreadsheet. Figure 7-7 shows how we can also create links to new items as part of the process.

Figure 7-7.  Linking work items to new or existing work items

We can also define check-in rules for our developers, which force them to associate a check-in/changeset with a work item(s), as can be seen in Figure 7-8. There should not be any need for a check-in unless the code change is associated with a work item. We should never do any code changes unless they are required to solve an issue, and this issue should always be documented as a work item.

Figure 7-8.  Linking a work item to a check-in can be required by a check-in policy

A changeset in TFS is a logical container into which TFS bundles everything related to a single check-in operation. A changeset consists of:

- Source file and folder revisions (adds, renames, edits, deletes, moves)

- Related work items (bugs, etc.)

- System metadata (owner, date/time, etc.)

- Check-in notes and comments

By associating a build with a changeset we can create traceability from the original requirement (user story in this case) to the built executable. We will see more about this later in this book.

This traceability can help us avoid problems like the ones described in the beginning of the chapter. By using the reporting functionality of TFS, we can quickly see what a work item is associated with and hence know that if we change some part of the code (like with a bug fix), this change affects a specific work item. Knowing this we can see that some test cases will be affected by the change and that we need to run those tests again to see whether the change broke anything. We can also get warnings from TFS that a check-in affects certain test cases, as we will see later in this book.

Work Item Queries

In Team Explorer, we can query the work item databases (see Figure 7-9) by using a new query language Microsoft provides: Work Item Query Language (WIQL), which has a SQL-like construct. Figure 7-3 shows an example of a query returning all active bugs, for instance. From Team Explorer or Web Access we can create new queries or modify existing ones.

Figure 7-9.  Work item queries in Web Access

Depending on the process template you use, the work item queries that are supplied differ quite a bit. Microsoft Scrum has different work item queries than MSF for Agile, for instance. When we have used the Agile template in some of our projects, we have found it necessary to add new work item types because the organization needed these for their ALM process. Queries to get information about these new work item types naturally don’t exist, so we have had to make these queries ourselves. Some of these queries have been built during the projects when the need arose, and many of these have later been included in the process template so they are now part of all new projects.

Conclusions on the Work Item Tracking System

The work item tracking system is one of the core components of TFS. This system enables us to create work items, or units of work, and can be used to enable traceability. We can use the work items included with TFS from the beginning, or we can choose to adjust these to our needs, or even create our own work item types. Each work item instance has a unique ID (as you saw earlier in Figure 6–8) that we can attach to the things we do in TFS. This enables us to follow one work item—let’s say a requirement, for example—from its creation to its implementation as a piece of executable software (component). We can also associate one work item with others and build a hierarchy of work items.

When a bug is discovered, we can quickly follow the original requirement by its work item ID and see in which places of the code we might have to make some fixes. We can also see the associated work items so that we can evaluate whether other parts of the code also need to be changed as a result of this bug fix.

If we implement a requirement as a work item, we can use the work item ID to track this requirement through source code and to the final build of the executable system. By requiring all developers to add one or more work item IDs to the check-in using a check-in policy, we can enable this traceability.

Our suggestion is that you look closely at the work item types supplied by Microsoft. Then you can decide which of these you can use for yourself and which you might adjust to suit your organization’s needs. If none of the ones supplied can be used, you have the capability to create your own work item types. Use this opportunity! Don’t adjust your way of working to the Microsoft templates. Adjust TFS to your needs instead.

Process Customization

As you have seen so far in this this book, it is essential to automate the ALM process to fully realize the benefits of it. TFS can help you quite a lot by letting you have one or more process templates on the TFS server that define the way you work with the ALM process.

In this section, we’ll take a look at the TFS process template, as well as the three process instances that Microsoft provides us with out of the box. You’ll learn about a template based on Scrum, another on CMMI, and yet another Agile process. The latter ones builds on Microsoft Solutions Framework, MSF, which is Microsofts project management framework.

The whole point of an extensible product such as VSTS is that we are able to customize it to our needs. One of the biggest advantages of VSTS is the capability to customize our process template so that we can realize our ALM process in the tools we use for our projects. Let’s take a closer look at how the process template is built up and how it can be changed by using the extensible features of VSTS.

Modifying the Process Template

There are two ways to modify the XML files for the project templates. We can use manual customization or we can use the Process Editor, which is a Power Tool from Microsoft.

If we are daring, we can manually edit the XML files. This can be done by exporting the files from the TFS server by using the Process Template Manager that ships with TFS and updating an existing template (see Figure 7-10). Or if we are even more daring, we can start from scratch. We suggest you use an already-existing process template and modify that.

Figure 7-10.  Exporting (Download) a process template from TFS by using the Process Template Manager

After we have the template in a folder on our computer, we can start modifying all aspects of it. We can use any XML editor to edit the XML files so why not use Visual Studio? In Figure 7-11, you can see an excerpt of what one of the XML files looks like when seen in naked XML. Now note the nice user interface we get with the Process Template Editor; we do not have to see the pure XML if we don’t want to.

Figure 7-11.  Example of process template XML file in Visual Studio

The Process Template Editor is a useful tool that Microsoft provides as an integrated part of the Team Foundation Server Power Tools for VSTS. You can find the Power Tools at this URL: http://msdn.microsoft.com/en-us/vstudio/bb980963.

Team Foundation Server Power Tools installs Visual Studio Team System Process Editor, which is a process template editor for editing TFS process templates inside the Visual Studio IDE (see Figure 7-12).

Figure 7-12.  Editing the process template by using the Process Editor inside Visual Studio

Common Adaptations of the Process Template

What are the most common things people change in the process template? That depends on your needs, so we recommend that you consider your needs carefully before starting to customize the templates. You should gather information to help point out these needs by doing an ALM assessment like the one outlined in Chapter 4. Because the process template is a representation of your ALM process, it makes good sense to understand your way of working. What are your organization’s needs? Which information is important in your bugs? How do you handle change requests? How do you handle requirements?

Do an assessment, run some workshops about the results, and talk about what your requirements are on the process template(s). Then select one project to use to pilot the process template and see the results. You will probably need to adjust your template after the pilot, but that is quite all right; that’s the purpose of a pilot.

The following are the most common parts of the template we usually update when working with our customers.

Work Item Types

You can use the work item types that Microsoft ships with TFS in the two MSF templates or the Scrum template. But as mentioned earlier, we think we should really consider our own needs in the organization and make adjustments to these. Your organization might need more work items or might need to extend the information required for them. If your project managers use Microsoft Office Project, you might want to change the mapping between fields in TFS against fields in Project. Another thing to consider is the workflow of the work items. How is the process in your organization? Which states can a bug transition between? Microsoft supplies a set of default work item instances when a project is created. These represent tasks that need to be done in all projects. Your organization might have different needs for default work items.

Work Item Queries

What information do you need to query about your work items? If you have made many changes to the work items, you might also need to change the queries so they reflect these changes. What queries does your ALM process need? In Figure 7-13, you can see the queries of the Visual Studio Scrum template.

Figure 7-13.  The work item queries in Visual Studio Scrum

Reports

This is something that most of our customers have modified. The reports in the Microsoft templates are very good. Figure 7-14 shows one of them, representing how much work is left in a project. This report and some others will be explained in more detail in other parts of the book. In choosing which reports and information you need, we once again come back to the fact that this is something that you need to discuss with your project teams and also with stakeholders and managers. What information is important to the various roles in your ALM process? What do the managers need to see? How can we provide great feedback on project status to the team?

Figure 7-14.  One of the reports in MSF for Agile showing remaining work in the project

Areas and Iterations

Areas and iterations are interesting concepts. Iterations are what the term sounds like: something done repeatedly, often taking as input the results of the previous pass. We use iterations to name the time dimension of our project (versions, projects, sprints, etc.). We can name them anything we want and we can nest these and build an iteration hierarchy as we see fit.

With Team Foundation Server 2012 we now also have dates on iterations! This may not sound like a big deal but having the system know about iteration dates makes it possible for TFS to automatically create a burndown chart or to show us work items for the Current iteration without having us specify what current is.

Areas are commonly used to represent product areas or features. One customer uses areas named after the windows or web forms in their projects. Another uses them for each component in their system.

We can use areas and iterations to tag specific parts of our projects. These concepts are flexible, and we have the freedom to use them as we want. All work items can later be labeled with both an area and an iteration. Depending on your ALM process, you might use this for various reasons. If you run a project using RUP, you might want to use the iterations by naming them after the phases of RUP. Then you can nest iterations below each phase depending on your need. Figure 7-15 shows an example of what this could look like. And if during the project we need more iterations in one phase, we can simply add them.

Figure 7-15.  Managing iterations

It is entirely up to you what you use these two categorizations for. In our opinion, they are very useful. They give you enormous freedom in setting up your projects, so I suggest you make good use of them.

Modifying Work Items

Microsoft encourages us to modify our process template. One important thing we have found worth modifying is the work items. Many organizations we have seen have needed information in their work items that is not available in the three Microsoft templates. In those cases, we have adjusted the work items to better fit in the organization. This has turned out very successful in all cases. One thing we have changed is the workflow of the work items.

How to Open the Process Template

We could start creating an entire new process template if we want, but it is far easier to start by modifying an existing one. First, we need to download the process template from our TFS Server. In Team Explorer go to settings and choose Process Template Manager below Team Project Collection (see Figure 7-16).

Figure 7-16.  Starting the Process Template Manager from Team Explorer

Select a process template for download, click Download in the Process Template Manager (see Figure 7-17), and select a location to download the process template. Close the Process Template Manager when you are done.

Figure 7-17.  Editing the process template by using the Process

To modify the process template you just downloaded, go to the Tools menu in Visual Studio and start the Process Template Editor (see Figure 7-18). You are given several options for what you can edit. As you see here, you can chose to edit the downloaded Process Template file(s) or select an item from the server. With the latter option you edit the current installed process template, changing all future projects created using that template.

Figure 7-18.  Starting the Process Template Editor from Visual Studio

After you have finished editing a downloaded process template, you can rename it and upload it to the server as a new process template that becomes available for all new team projects.

Work Item Fields

The default work items in TFS include a lot of information in their fields. But sometimes (quite often in fact) you need to include more (or maybe remove some) fields so that the work items fit your organization better. You do this by using the Process Template Editor. In Figure 7-19 you can see the fields from the Product Backlog Item in the Microsoft Scrum template. You can see their names, what data type they are, and also the Ref Name.

Figure 7-19.  The fields in the PBI WI from the Microsoft Scrum template

If you double-click on a field, you are presented with the Field Definition as seen in Figure 7-20. From this dialog you can change all aspects of the field itself.

Figure 7-20.  Field Definition dialog

You can add different kinds of rules to the field as seen in Figure 7-21.

Figure 7-21.  An example of a workflow for a Bug work item

So if you want to, you can control what values can be inserted into the field and add other constraints as well.

To change the layout of the work item, use the Layout tab (see Figure 7-22). This might look a bit complex at first, but once you start experimenting you will find that it is pretty easy to do a complete makeover if you want.

Figure 7-22.  The Layout editor for our work items

Select Preview Form to see your changes (see Figure 7-23).

Figure 7-23.  Previewing the layout

Work Item Workflow

You can add a workflow to the work items. A Bug work item has a State field, for instance, where the state flows through different levels. In this field, you can set the status of the bug. It can be active, closed, resolved, and so on. A typical workflow can look like Figure 7-24.

Figure 7-24.  An example of a workflow for a Bug work item

In this example, you can see the workflow for a Bug work item in Microsoft Scrum. This particular work item can have one of five states: New, Approved, Committed, Done, or Removed. The bug can transition through these states in the following ways:

- New to Approved

- New to Removed

- Approved to Committed

- Approved to Removed

- Committed to Done

- Removed to New

You can also let automatic transitions occur in the workflow. If a closed bug is reopened because of a new test showing there are still some errors in the code, you can automatically have the bug reassigned to the person who closed it. This way, you can save some work because you don’t have to hunt down that person yourself.

Web Access Customization

We have now looked at how to customize the process template and in particular the work item forms and workflows. If we make changes to the work item types we may want to reflect that in the Web Access UI as well. It is possible to change many aspects of the web UI and we will look at the most common scenarios in the following sections.

Using WITAdmin to Change the Web Access UI

To make changes to the Web Access you use the command-line interface to the work item system, witadmin.exe. witadmin.exe is installed as part of the Team Explorer and is located at %Program Files%\Microsoft Visual Studio 11.0\Common7\IDE (see Figure 7-25).

Figure 7-25.  The witadmin.exe command-line tool

Note  WIT Admin cannot currently be used against the hosted TFS Service (tfspreview.com). The cloud-based version of TFS will be covered in Chapter 34.

Changing the Product Backlog Add Dialog

Let’s start with the Product Backlog list in the Web Access. The default view in a Scrum template-based product has a quick add dialog (see Figure 7-26). But what if our process requires other fields than the Title to be entered or we just simply want to provide the most common values in the quick add form?

Figure 7-26.  The default Microsoft Scrum Product Backlog Add dialog

Well, as it turns out you can. So, let’s add the Effort field to the quick add dialog. First, run the following command to export the current definition:

witadmin exportagileprocessconfig /p:"expense reporting" /collection:http://localhost:8080/tfs /f: exportagileprocessconfig.xml

Then, edit the exported XML file by adding the Effort field to the AddPanel block in the ProductBacklog section:

<?xml version = "1.0" encoding = "utf-8"?>

. . .

<ProductBacklog>

<AddPanel>

<Fields>

<Field refname = "System.Title" />

<Field refname = "Microsoft.VSTS.Scheduling.Effort" />

</Fields>

</AddPanel>

. . .

</ProductBacklog>

. . .

Save the file and import the file using the reverse command:

witadmin importagileprocessconfig /p:"expense reporting" /collection:http://localhost:8080/tfs /f:exportagileprocessconfig.xml

Now the Web Access looks like Figure 7-27.

Figure 7-27.  A customized Microsoft Scrum Product Backlog Add dialog

Changing the Columns in the Product Backlog List

Next, we will look at adding more columns in the Product Backlog list. By default in the Microsoft Scrum template the list will look like Figure 7-28. Maybe you want to show the Business Value field, as well as the Effort field.

Figure 7-28.  The default Microsoft Scrum Product Backlog list

To change the list, use the same command as in the previous example, but change the Columns section:

<ProductBacklog>

. . .

<Columns>

<Column width = "400" refname = "System.Title" />

<Column width = "100" refname = "System.State" />

<Column width = "50" refname = "Microsoft.VSTS.Scheduling.Effort" />

<Column width = "100" refname = "Microsoft.VSTS.Common.BusinessValue" />

<Column width = "200" refname = "System.IterationPath" />

</Columns>

</ProductBacklog>

After importing the updated XML, the list will look like Figure 7-29.

Figure 7-29.  A customized Microsoft Scrum Product Backlog list

Adding States to the Board

Another area we may want to customize is the team Board. If we have added more states to our work item workflow, then it might make sense to show that as additional columns on the board. To add a column, do the following:

First, export the common properties from the TFS project:

witadmin exportcommonprocessconfig /p:"expense reporting"

/collection:http://localhost:8080/tfs /f:commonprocessconfig.xml

Then, change the XML file by adding the Ready for Test state:

<TaskWorkItems category = "Microsoft.TaskCategory">

<States>

<State type="Proposed" value="To Do" />

<State type="InProgress" value="In Progress" />

<State type="InProgress" value="Ready For Test" />

<State type="Complete" value="Done" />

</States>

</TaskWorkItems>

Finally, import the updated XML back to TFS:

witadmin importcommonprocessconfig /p:"expense reporting"

/collection:http://localhost:8080/tfs /f:commonprocessconfig.xml

Note  You must add the “Ready for Test” state to at least one of the work item types in the “Microsoft.TaskCategory” for this customization to work (or else how would the system know what to display in the “Ready for Test” column).

Now if you go back to the Web Access and refresh, the Board has an additional column for the Ready for Test state (see Figure 7-30).

Figure 7-30.  A customized Microsoft Scrum Product team Board

Summary

In this chapter we have looked at how the work item tracking system in TFS can be used as a generic tool to track things in a project. The work item system is part of the process mapping in TFS and the product comes with three different templates as a starting point. You can choose to use these templates as-is or you can customize the templates to suit the needs of your project. Finally, we took a look at how we can take the changes we make in our process into Web Access as well. We should take advantage of these capabilities to give our users an intuitive interface to the system.

Work Items are also the most important part in TFS for achieving traceability. They are the foundation for fulfilling the ALM cornerstone of traceability that we covered in Chapter 2. Using work items will give us the opportunity to have traceability from the requirement all the way to the executable code.

CHAPTER 8

Agile Project Management With TFS

In this chapter we complete a startup of an agile project using TFS. Many of the concepts covered in Chapter 6 are exemplified in this chapter so you can see how we move from planning to implementation. We also look at how TFS can support the agile project management process during sprints.

In this chapter and throughout the rest of the book, we will use a fictitious company in our examples. This way we try to have a red thread[1] in the things we present so we can more easily understand the process and how TFS supports our development organization.

This chapter focuses on the project management parts of a project. All support for agile development practices such as continuous integration, test driven development, test automation, and so on will be explained in subsequent chapters.

The main part of this chapter is written from the perspective of the PO, whom we will meet shortly. There will be a personal touch on some parts. The reason for this is that this part of a project focuses so much on collaboration and interaction between people. If you recall, the agile manifesto talks about the left side of the four pillars:

- Individuals and interactions over processes and tools

- Working software over comprehensive documentation

- Customer collaboration over contract negotiation

- Responding to change over following a plan

Case Study

Let’s start with the company we will use as an example. Any similarities to real companies are totally unintentional.

Company Background

Our company is called Kangreen and is located mainly in Sweden, where the headquarters are. The company is about three years old and hence relatively new on the market. Alice Miller, the CEO, got tired of her career as a psychologist and decided to open another business with a friend, Bob Peak.

Kangreen develops software for psychologists and has had a great start. Customers are located all over the world, and sales people travel widely to sell products. The sales organization consists of 12 persons but is growing as demand of the software increases.

Kangreen has about 60 employees worldwide. The development organization consists of approximately 30 people. Of these, 20 are located in Gothenburg, Sweden, and the rest in Seattle, Washington. All development is currently done using Visual Studio 2010, but the company is planning to start using Visual Studio 2012 for all new development. Kangreen uses TFS 2010 today, but mostly for version control. Figure 8-1 shows the high-level organizational chart of Kangreen.

Figure 8-1.  Overview of the Kangreen organization

There is a demand for Mac software on the market as well, so Kangreen is setting up a small Java development organization that will use Eclipse as the main platform for development. Recruitment is ongoing.

The great start and ever-increasing demand for Kangreen software hasn’t been without trials. For the development organization the need of a good ALM process is evident. There have been several cases where bug fixes have caused new bugs in the software that have not been found until the bug fixes were rolled out to customers. This lack of traceability could damage customer satisfaction and must be fixed quickly.

Another issue that has been troublesome is the collaboration between the team in Seattle and the team in Stockholm. There is a need for sharing code between these two parts of the organization, and they also need to have updated status reports on the progress in the different projects.

Kangreen development manager, Cindy Crafoord, has decided to implement a pilot project using the ALM features of TFS 2012 to bridge the gap between what they have today and what they can benefit from in TFS 2012. If the pilot is successful, Kangreen will migrate all their development to the TFS platform.

Cindy and Bob Peak have decided to use Scrum as the preferred project management method, and the developers agree on using XP practices to enhance quality of the software and therefore increase business value to the company.

The Pilot Project

The project Kangreen has decided to use as a pilot for the ALM implementation is an expense reporting application (Kangreen Expense Reporting). In the early days, expenses were handled easily by the admin staff, but since the company has grown quickly and sales persons are located and travelling all over the world, things have been a little more complicated. The admin staff wants an application that will make their jobs easier and at the same time make sure employees will get reimbursement for expenses quickly. The requirements for this application are covered in the section “Requirements” later in this chapter.

Because this project will be using Scrum as a project management process, Cindy and Bob have appointed Fiona Gallos as Product Owner for the application. Fiona is new in the company and has only been working with Kangreen for six months. She is experienced as a Product Owner because her previous employer used Scrum extensively.

Important stakeholders for the project are Bob Peak, Cindy Crafoord, and Karen Guckenheimer. Karen is manager for the admin department and will represent the end-users as well as the admin organization. Because the project aims to be a pilot for an ALM implementation, Dave Applemust from the infrastructure side and Harry Bryan from the development organization are also considered important stakeholders.

The People

- Alice Miller, CEO

- Bob Peak, IT manager

- Cindy Crafoord, Development manager

- Karen Guckenheimer, admin manager

- Dave Applemust, infrastructure specialist

- Eric Parrot, Business analyst

- Fiona Gallos, Product owner

- Guillio Peters, Scrum master

- Harry Bryan, Senior developer

- Mikael Persbrandt, developer

- Petter Ivarsson, User Experience, UX

Scrum TFS Template

Before we follow Fiona as she begins this project, we want to have a look at the Scrum TFS Template. This template is supplied out-of-the-box by Microsoft and is a good implementation of Scrum. For all examples in this book we will use this template for our sample project (Kangreen Expense Reporting).

Work Items

There are five work item types in the Scrum template (see Figure 8-2).

Figure 8-2.  The five work item types in the Scrum template

- Bug. Self-explanatory perhaps, but this is used to report bugs.

- Impediment. Based on the outcome of the daily Scrum, we may have one or more impediments blocking the team or a team member. This work item type is used for the impediments.

- Product Backlog Item (PBI). All requirements should be in a user story format and this work item type is used to document them.

- Task. This could be anything that needs to be done. If we use PBIs for user stories we could use tasks for all detailed work that needs to be done to solve the user story based on the outcome of the sprint planning meeting.

- Test Case. Just what it sounds like. Use these for documenting your test cases.

TFS Web Access and Portal

A team portal is created for all new TFS projects (see Figure 8-3). This is a customizable SharePoint portal that provides a useful overview of the project status.

Figure 8-3.  The TFS project portal

We can also use TFS web access (see Figure 8-4) if we want to let nontechnical users add bugs or new PBIs. From this we can view reports, create new work items, view builds, and much more. Often the PO or scrum master prefers to use the portal so that they don’t have to use Visual Studio to access these features. If we have a PO or SM who prefers a Mac, web access is the best way to access the power of TFS.

Figure 8-4.  TFS web access

We can set access control for the web access in Settings for the project. We can use the web access if we want to let nontechnical users add bugs or new PBIs. By controlling the access to the portal we can let certain users or groups of users only see (and do) what we want them to see. This way we can let customers (if we are consultants) into TFS with limited functionality. The portal is great for different kinds of collaboration.

Reports and Queries

There are numerous out-of-the-box reports with the templates in TFS. Chapter 9 covers this in more detail. Let’s just say that the Burndown chart (see Figure 8-5) is maybe the most common report that we use. The Burndown shows how many hours are still left in the sprint and is useful to see whether we are on track.

Figure 8-5.  The Burndown chart

There are also queries we can use to retrieve information from TFS: One query that is always supplied in the scrum template is the “Assigned to me” query (see Figure 8-6) that shows all work items assigned to me.

Figure 8-6.  The Assigned to me query in TFS

Using query language (see Figure 8-7) we can modify or write new queries that suit us better.

Figure 8-7.  Modifying the Assigned to me query in TFS

Another nice addition to TFS is the possibility to have an electronic board showing the state of the work items on the sprint backlog (see Figure 8-8). Here we can see what is done, what is in progress, and so on, instead of using yellow stickers on a wall. Having a big screen on the wall that shows progress gives all team members up-to-date information, not to mention the PO when he or she visits the team room.

Figure 8-8.  The task board showing sprint backlog items and their status in TFS

Project Startup Phase

This section follows the product owner, Fiona Gallos, during the startup phase of the project. We will see how TFS is used to insert the information that Fiona collects during this phase.

PO Sets Off to Work

The idea for this project started when Kangreen noticed that bug fixes created new bugs and that the new bugs sometimes appeared in parts of the system considered not to be affected by the original bug fix. Kangreen soon realized that they lacked traceability and had no way of knowing where a bug fix would have its impact besides the actual code change.

Fiona had just attended a conference and learned a great deal about ALM and TFS. She came up with the idea to get a better grip on the ALM process and at the same time start using agile practices at Kangreen. Both these efforts would greatly improve things at Kangreen so they could avoid embarrassing situations such as when customers find new bugs caused by bug fixes.

At the same time, Fiona saw that collaboration between the two developer teams could improve if they started to use TFS. Fiona wrote down a business case and presented it to the management team. After a few discussions they agreed to try this on a pilot. Because the Expense report project was in the pipeline they decided to use it for the pilot.

At this point it was hard to calculate ROI, but anything that could improve how the customers looked upon them would be worth going for.

Building the Initial Team

It is recommended that the PO starts with a small team during initial planning of the project. Fiona selected Cindy Crafoord, Harry Bryan, and Eric Parrot as they were experienced within the company and are also senior persons with experience from other companies as well. They are also available for the whole pilot project, which was an important aspect for Fiona. She knew the importance of having consistency among the team members during a project. Guillio Peters was going to be scrum master for the entire project, but he would join a week later because he was finishing another project at the moment. The rest of the team was going to be selected a bit later in the project. Fiona liked to start with a small team.

Fiona created the project in TFS (see Figure 8-9) from the web portal using the Scrum template. She named it Kangreen Pilot.

Figure 8-9.  Creating the Kangreen pilot project in TFS

Then she started to add the users to the project (see Figure 8-10).

Figure 8-10.  Adding users to the project

After adding the users and creating the TFS project, Fiona was ready to go. She had what was necessary to start requirements gathering.

Requirements

Requirements gathering was a fun part in Fiona’s eyes. Discussions with traditional project managers and stakeholders about requirements always came up and she enjoyed that. Traditionally all requirements had to be found at the beginning of the project and it was hard for many persons to accept that it is okay to start a project even without specifying everything. The fact that so many of these requirements were wrong or unnecessary in the end didn’t seem to bother traditionalists. They still went head first into projects that often failed or were flawed.

Fiona had run so many successful agile projects she knew that catching higher-level requirements in the beginning was okay. They could start without all details because they would be clarified at each sprint planning meeting and also during the sprints.

Fiona called the initial team for a requirements workshop. She also added Karen Guckenheimer to the workshop because she was one of the main stakeholders from the business side. Because Guillio (SM) was not present, Fiona explained what they were going to do. She stressed that they should look for higher-level requirements in the sense that they did not have to detail them yet. There was going to be no discussions about solutions or technicalities at this point. That was left for the development team to decide when the sprints started.

To avoid any confusion, she then explained the concept of a user story for the requirements team. Fiona wanted all requirements in this form:

- As a < type of user>, I want < some goal > so that < some reason>

Fiona had calculated three hours for this meeting, and booked a room with a large whiteboard. She also supplied Post-it notes and pens for everyone.

They started by brainstorming user stories and things were a bit slow to begin with. The meeting took off when Harry Bryan came up with two user stories:

- As a sales person I want to manage expense reports over the Internet so I can be more efficient.

- As a manager I want to search expense reports so I can easier get an overview of expenses.

Then suddenly they all started writing. After little over an hour the pace dropped again. They then spent another hour going over the user stories they had and any clarity with any of them was discussed. Fiona felt they had done a great job so far, and had a good foundation for the work ahead of them.

Building the Backlog and Prioritizing It

After the meeting, Fiona went to her desk and wrote down in a spreadsheet what they had come up with. She liked to use Excel, partly because she liked the tool and partly because she knew she could import the user stories into TFS if she wanted.

Fiona then started to order the list. She made an initial prioritization based on some assumptions:

- Initially they cannot know the actual cost of a work item

- All work items cost the same to develop

- Prioritization will be based on importance only

- After initial sprint planning and estimation Fiona will update the list again

It took Fiona roughly 30 minutes to complete the initial sorting. Now she really had something to start with. It was still early afternoon and she wanted to add the user stories to TFS before going home for the day.

Adding Backlog Items in TFS

Fiona opened the TFS project portal in Safari (see Figure 8-11) and felt a little bit of excitement as she saw the empty project that soon was going to be filled with activities. She had lots of input for the backlog.

Figure 8-11.  The project portal

She took a long look at the spreadsheet she had in front of her and started by going to the Work Items tab on the web page (see Figure 8-12). She could also have gone to the Backlog tab, which has a quick-add feature for adding work items (see Figure 8-13) that you can turn on or off.

Figure 8-12.  The Work Items tab

Figure 8-13.  Adding a PBI from the Backlog tab using the quick-add feature

From the drop-down menu on the left she selected New, Product Backlog Item (see Figure 8-14).

Figure 8-14.  The Work Items tab

This opened the form as seen in Figure 8-15. If Fiona had created a backlog item from the Backlog tab using the quick-add feature, the new backlog item would have ended up at the top of the backlog. Fiona would have had to click on it and then she would have had the work item form seen in Figure 8-15.

Figure 8-15.  The first PBI

She took the first PBI on her backlog and started filling in the fields. She left a lot as it was for now and only filled in the PBI name and description. For example, she needed a rough initial estimation before she could fill in Effort. She also wanted to wait until they had a clear Definition of Done ready and a risk assessment done before filling out things such as Acceptance Criteria and Test Cases.

Fiona then continued with the rest of the higher-level use cases until they were all in TFS.

Definition of Done

Before going home that day Fiona called a new meeting about the Definition of Done (DoD) with the team. She included infrastructure specialist Dave Applemust for this meeting as there are constraints from the infrastructure team when building and deploying new projects.

She wanted to discuss the DoD so that all had a common view on this before starting the actual coding. Many times she had seen the problems not having a DoD could cause in projects and she knew from experience this was important.

Two days later they met for the DoD meeting. Fiona explained the importance of this concept and spoke about issues she had experienced when not using a DoD. There were nods of recognition among the participants as she spoke.

She then let all participants write down the things they wanted to have on a DoD. After some discussion they agreed on the following list for an approved user story:

- All code is written and checked in (including tests)

- Coding conventions fulfilled (these are documented in a separate document and not included here)

- All unit tests passed (must be okay before check-in)

- Code is refactored (improved/optimized without change of function)

- All code has been reviewed by at least two persons (peer-programming or peer-review)

- User story is included in the build (build scripts updated, all new modules included)

- The user story is installable (build scripts updated so that story is included in automatic install)

- All acceptance tests are passed

- Acceptance criteria must exist

- Acceptance tests are implemented (automatic or manual tests)

- Backlog is updated by:

- All tasks remaining time is 0

- User story state is “Done”

- “Actual Hours” is updated

- All tasks are “Done”

- User story is installed on Demoserver

- User story is reviewed by PO

- User story is approved by PO

- Product documentation is updated and checked-in

- User manual is written

- Administrative manual updated

- Administration manual

- Help texts written

The team also came up with the following DoD for when the Sprint is done:

- All user stories in the sprint fulfill the DoD

- Product is versioned (release management/rollback)

- All accepted bugs are corrected

- New bugs that have been identified are closed and/or parked

- 80% code coverage from automated tests is fulfilled

- All chores are done and approved

- All integration tests passed

- Sprint Retrospective performed and actions for improvements identified

- Sprint Review, with PO present has been performed

- Performance test of the complete system has been done

Estimation

After DoD they had what they needed to do some initial estimation of the work. Fiona needed to come up with a rough budget for the project to show the stakeholders and also an initial release plan. She decided to use poker planning for this. She had used it previously and was happy with the result. She called for a poker planning meeting with the initial team.

Poker Planning/Story Points

Again they met in the same conference room they had used before. Fiona had purchased planning poker decks from Mountain Goat Software (www.mountaingoatsoftware.com) for everybody. She started by explaining the rules for everybody and then they began:

- Fiona started by reading the first user story.

- After a short time participants each selected a card without showing it to the others.

- When Fiona asked them to show their cards, they turned them over.

- Cindy and Harry were the furthest apart and they both explained their thoughts on the user story and then the team played again.

- This time they were closer to each other’s points (only one step apart) and the higher value was selected for the story.

- They continued through the user stories until they were finished.

Updating the PBI

After they were done Fiona went to her desk and started to update the PBIs. She now inserted the story points for each PBI into the work items (see Figure 8-16) in the Effort field. During sprint planning these would be broken down into more manageable pieces and each task would get a time estimate instead of story points.

Figure 8-16.  Updating the PBIs

So story points are done, but before continuing to sprint planning and time estimates Fiona wanted to do an initial risk assessment.

Risk Assessment

Risk assessment is part of all estimation in agile projects and should be done during the whole project. If any PBI could be considered as very risky, it might need to be prioritized higher on the backlog. It is always better to address high-risk items as early as possible to avoid surprises later. Fiona knew the surprises would come anyway.

There are different ways of performing risk assessments. We suggest you choose the one you are familiar with. Fiona chose to do a traditional risk assessment by using the following parameters:

- Severity (1-5)

- Probability (1-5)

- Risk

- Risk assessment score (severity x probability)

- Mitigations

- Probability after mitigation

- Risk assessment score after mitigation (severity x probability after mitigation)

For each user story on the backlog they went through this analysis. Fiona ended up with an Excel sheet looking like the following (see Figure 8-17).

Figure 8-17.  Risk mitigation

Updating the Backlog Order

The team found no risks that were exceptional at the initial risk assessment, so Fiona left the backlog almost untouched. In Figure 8-18 we can see part of the backlog before going into sprint planning.

Figure 8-18.  Part of the backlog for the Kangreen expense report application

Grooming the Backlog

All through the sprints the PO needs to groom the backlog. The PO does not do this work alone, so the team needs to be part of this as well. This is an excellent way to get the team’s views on the upcoming features and for them to give feedback and new ideas to the PO. Fiona decided to estimate about 10 percent of the team’s time for backlog grooming. This number had worked well in the past.

Building the Team

Now the team was close to getting started. Fiona had Cindy Crafoord, Harry Bryan, Eric Parrot and Guillio Peters in the team so far. She talked to the other team members and they decided they needed three more persons. They selected the following:

- Mikael Persbrandt, developer

- Ingrid Svensson, senior tester

- Petter Ivarsson, UX

Fiona then contacted each person’s manager and made sure they were available for the project. Luckily, they all were and when she approached the potential team members they were happy to come aboard.

Fiona was going to go for two-week sprints because that was a good time box based on her experience. Once she had a team that complained that they could not finish their PBIs during the four-week sprints they used. They always seemed to be late or not deliver everything they had committed to, complaining they needed more days in the sprints. She then said: “Okay, then we use two-week sprints instead.” The team was very confused, as Fiona had decreased the number of days in the sprints, not increased them. Once they started working on the two-week sprints however, they soon found they delivered more in two weeks than in four. The team was more focused and did not postpone anything until the end of the sprint, hence they were more effective.

Initial Velocity

Fiona needed a few more things before she would arrive at the time estimates for the project. She needed to know the initial velocity of the team. The velocity is nothing more than the speed of the team. How much work (user stories) can they take on in a given sprint? She also needed to know how many hours they actually have for work in the sprints.

Available Time

As usual, Fiona used Excel to calculate the available time of the team. She followed the rules outlined in Chapter 6:

- How long is the sprint? In this case two weeks.

- How many working days are available in the sprint? Fiona would have 10 working days.

- How many days does each team member work during the sprint? Planned vacation or other days off, planned meetings, and so on. She looked at each team member’s schedule and filled out the spreadsheet.

- Fiona deducted the time for Sprint Planning, Review, and Retrospective meetings, which would be eight hours per person for this sprint.

- Excel calculated the result of this. What we get is the capacity before “drag.” Drag is wasted time or unknown activities.

- Because the team was new and Fiona had not worked much with any of them, she used a standard 25 percent as she knew this could be a good landmark. Included in this is 10 percent backlog grooming.

Now Fiona had the available number of hours in the sprint: 289.5 (see Figure 8-19).

Figure 8-19.  The spreadsheet Fiona used for calculating available time in the project

Capacity Planning in TFS

You can use TFS for capacity planning as well. In the backlog section, in the current sprint, click the Capacity tab (see Figure 8-20). Here, you can set capacity, activities, and days out of office for vacations and holidays. As you set capacity, activity, and days off, graphical information about hours and capacity is automatically generated in the pane on the right (http://blogs.msdn.com/b/visualstudioalm/archive/2011/12/07/team-foundation-service-preview-get-started-with-agile-planning-tools.aspx).

Figure 8-20.  Entering capacity into TFS

Microsoft has introduced the concept of team in TFS 2012. We can create several teams if we want. So if we have a scrum project (or any other project of course) that needs to scale to more teams, we can create them and add the designated team members to each team. When calculating capacity we will get summaries for each member, team, and the project. This is a very useful feature, especially for the product owner.

In Figure 8-21 we can see that in the current sprint (Sprint 1) we have two tabs; one for contents and one for capacity. The capacity is shown also in the Contents tab in the far right and can be switched on or off.

Figure 8-21.  Entering capacity into TFS

The sprint planning features in TFS offers three different ways for the team to determine whether they have enough capacity: By person, by role (Activity), or at the whole team level. This is very useful information for the product owner.

Initial Sprint Planning

To calculate the initial velocity of the team Fiona usually did an initial sprint planning. This was exactly the same as any sprint planning except that it was performed before the actual sprints started. Often this sprint planning was used in the first sprint because they would be very similar.

During this meeting, the team estimates the tasks in hours so that they can plan the sprint and decide on how many user stories they can take on in their sprint. This is the way the Kangreen team performed this:

- Estimate the first user story in detail

- Break down what the team needs to do to deliver the story

- Estimate hours for each activity and summarize

- Deduct the summary from the available time the team has in the sprint

- Is there still time left?

- Take a new user story and repeat the process until no available time is left

- Summarize the number of story points from the stories that were included in the sprint. Now we have a theoretical velocity.

The first (highest prioritized) user story on the backlog was

- “As a sales person I want to manage expense reports so that I can be more efficient.”

The number of story points for this had been five from the planning poker session. This was broken down into smaller pieces:

- Create Expense Report

- Delete Expense Report

- Modify Expense Report

- Send Expense Report For Approval

- Logon to Expense Report System

Together with the team, Fiona prioritized these so they had a beginning for the sprint backlog. The sprint backlog looked like this after prioritization:

- Create Expense Report

- Send Expense Report For Approval

- Modify Expense Report

- Delete Expense Report

- Logon To Expense Report System

For each of these they continued breaking them down into smaller pieces and estimated them in hours. For creating the Expense Report they came up with the following.

- Create GUI

- Create Business Logic

- Fulfill DoD Requirement

- Write User Manual

The estimated number of hours for this user story was 137. With an available time of 289.5 hours they still had 152.5 left in the sprint. This meant that they still had room for more work, so they continued with the next user story on the backlog. This was

- “As a controller I want to be able to manage the users in the system so that I have full control over the users”

and was worth 3 story points.

After breaking this down they had 95.5 hours left so they continued with another user story worth 2 story points. When this planning was done there remained 23.5 hours of available time but Fiona and the team chose to not take on anything more in the sprint. The team was new and if there were problems, they wanted some space. It’s better to finish the tasks than to reach the end of the sprint and not be able to finish some of them. If the team had time left in the sprint, they could take on some more but they left that for later.

The total amount of story points for the sprint was now 10. This is the team’s initial velocity.

The sprint backlog now looked like this:

- Create Expense Report

- Send Expense Report For Approval

- Modify Expense Report

- Delete Expense Report

- Logon To Expense Report System

- Create User

- Modify User

- Delete User

- Create Customer

- Modify Customer

- Delete Customer

Each of these had tasks associated with them that are part of the complete sprint backlog and Figure 8-22 shows an excerpt of this backlog.

Figure 8-22.  Excerpt from the backlog

Updating Backlog/PBI

When the sprint planning was over Fiona updated the sprint backlog and inserted the new tasks into TFS. She associated them with the first sprint (see Figure 8-23). She also added the date when the first sprint would start under Manage schedule and iterations so that TFS was updated with this information.

Figure 8-23.  Associating task with sprint

One thing worth considering here is what to do with the epic user stories in the backlog after they have been broken down. Are they still valid in the backlog anymore at that point? In our opinion we can safely remove these backlog items (by setting the status to Removed) as long as we are certain that the content is covered in the broken-down tasks (see Figure 8-24). Fiona removed the epics and got fine granularity of the backlog items at the top of the backlog and larger epics the further down the list she came.

Figure 8-24.  Removing a work item by changing its state to removed

Note  An epic is a large user story that is so big that it is impossible to estimate how much effort it would take to develop it. You can compare an epic to this user story:

As a human I would like to have world peace so that we humans will not kill each other anymore.

Even though this example is farfetched, so are many epic user stories—at least until they are broken down into smaller, more manageable, user stories.

Forecast in TFS

There is a nice feature in TFS that will let you create a forecast on how much work you can have in each sprint. It requires that you fill in the effort estimate on each work item. In my example in Figure 8-25 we have story points estimated in effort. We can also see that forecasting is based on the velocity of five story points and TFS automatically draws the sprints and the work items that will fit into each sprint. The forecast can be switched off as well if we do not want to see it, by clicking on or off in the right top of the page.

Figure 8-25.  Forecast in TFS

We can use the Velocity Report in the upper-right corner of the product backlog to look at the historical velocity numbers and based on that, figure out a good velocity forecast number (see Figure 8-26).

Figure 8-26.  Velocity Report in web access

You can also base forecast on hours. Just change the values as you want. TFS uses story points or hours.

Release Planning

Based on the information she knew now Fiona could start planning the releases of the project. She knew the management team would like to know how many releases they were planned and she wanted to give them this information as soon as she could. The first thing she did was look for themes in the user stories.

Themes

Fiona looked at the backlog for quite some time and came up with several themes:

- Expense report management

- Search functionality

- User management

- Customer management

- Project management

- Smartphone availability

She quickly saw that three themes were going to be part of the first sprint. According to the initial sprint planning, Expense report management, User management, and Customer management were all part of the first sprint.

Considering that there were many chores in the first sprint, she knew that all three would not fit in. She aimed on getting the Expense report management theme done.

Fiona also knew the initial theoretical velocity (10 story points), which she used as an input for how much work she could expect in each sprint. With 44 story points total, the project would take 4.4 sprints to complete. She rounded this up to 5 sprints.

Note  A chore is just something a team needs to do. It could be setting up a build server, fixing the team room, fixing white boards, installing necessary software, and so on. Chores are never estimated. In the beginning the first sprints are probably filled with chores just to get started. This means that the velocity in the first sprints will be lower than in the coming sprints when most chores are complete. There is just not so much room left for estimated work in the first sprints.

So, a rough overview would give the following release plan:

- Expense Report Management in Sprint 1

- User Management, Customer Management in Sprint 2

- Project management, Search Management in Sprint 3

- Smartphone availability in Sprints 4 and 5, depending on smartphone

Estimated Time Plan

Fiona then used Excel to create a time plan of the project (see Figure 8-27). She knew this was going to be temporary and could change depending on what happened during the project so she would only show it to the stakeholders and not let them keep a copy of it.

Figure 8-27.  The initial time plan Fiona created

Estimated Project Cost

After all this was done, Fiona could come up with an initial estimate of the project cost. She knew how many weeks the project would take based on initial estimation, which was 10 weeks. With the help of the admin department she could calculate the weekly cost of each and every one of the team members. She then multiplied the weekly cost with the number of weeks and came up with a cost estimate. On top of this she added the hardware, software, and other costs she knew she would appear. She got an estimated project cost, which she used as input for the management meeting, where she was going to present the time plan and project budget. Luckily the management team approved of the project and she was good to go.

Fiona was now ready to start the project. She began by looking at the startup dates, confirmed again with all managers of the team, and then sent out the invitation for the sprint planning meeting for Sprint 1.

Running the Project

Let’s now briefly take a look at some things where TFS can help us during the sprints. Many of these features have been discussed previously in this chapter, but there are some new things that are handy as well.

During the Sprint(s)

We have gone through much of what Scrum says about sprint work in Chapter 3. In addition, here are some important meetings that the team needs to go through:

- Sprint planning. We have covered this extensively in this chapter. PO, scrum master, and dev team participate.

- Daily scrum. Each team member goes over what they have done since last daily scrum, what they will do until the next daily scrum, and if there are any impediments blocking their way. This is a stand-up meeting a maximum of 15 minutes long.

- Sprint review. This is where we demo the deliverables of the sprint and the PO can sign off on the approved user stories. PO, scrum master, and dev team participate.

- Sprint retrospective. Basically we write down what was good and what was bad during the sprint. Then we select some of the bad things and try to improve them during the next sprint. Scrum master and dev team participate.

Let us start by looking at them again.

Sprint Planning Meeting

Just for repetition, in Figure 8-28 we can see the steps performed during a sprint planning meeting.

Figure 8-28.  The sprint planning process

The outcome of the sprint planning meeting is the sprint backlog, filled with sprint backlog items. We have previously shown how that is inserted into TFS so we will not cover that again here.

Daily Scrum

Each day the team has a stand-up meeting to discuss three questions:

- What have you done since the last meeting?

- What will you do until the next meeting?

- Do you have any impediments stopping your work?

During the daily scrum we should use the task board (see Figure 8-29), which is an excellent way of displaying the work in the sprint and the status of each task.

Figure 8-29.  The task board in TFS

The agile template in TFS has a work item type for the impediments (see Figure 8-30). We use this to make sure that all impediments are captured and that we do not forget about any of them.

Figure 8-30.  The Impediment work item form in the Scrum template for TFS

Reports and Queries

During the sprint we use the reports and queries that we need. Some of these were shown earlier in this chapter. If there isn’t a report for what we want, we can use Excel or SQL Server Reporting Services to create one. The burndown chart is very much in use in all agile projects, whether or not we use TFS.

In Chapter 9, 14, 19, 24 and 29 at the end of each part of this book we will cover these reports in depth so please refer to those chapters for more information.

Backlog Grooming

As we have seen earlier, during the sprint the PO needs to groom the backlog so that it is in good shape. This means that the backlog should be ordered and that the top backlog items should be broken down into smaller, more manageable pieces. The team helps the PO with this and we estimate roughly 10 percent of available time for the team for this task.

The PO then updates the TFS backlog so it reflects reality. We can easily drag-and-drop items on the backlog to change the order, which is a nice feature.

Sprint Review

The sprint review is the meeting where the team shows the PO and any other stakeholder(s) what they have built during the sprint. Any working software should be demoed so that the PO can sign off on the user stories that have been delivered. Nothing of what is shown should be a surprise to the PO. He or she should have been such a part of the sprint that there should be no surprises here.

In the sprint review meeting we can use TFS quite a lot. There are many teams that use TFS for showing the PBIs that they have been working with during the sprint. This way we can see what was done (is everything right?) and what was not. There is also a great benefit using the reports in TFS so we can look at burndown charts and velocity. This information is useful as an input for the next sprint planning meeting.

Sprint Retrospective

During the retrospective we look at what was good and what was bad during the sprint. This is by far the most important meeting in scrum. Why? Because this is where we can learn how to improve. Constant retrospective and adaptation is essential for the team if they are to deliver quality software and business value.

We usually execute the sprint retrospective by using a white piece of paper and divide it with a marker pen. The left side is what was good (marked by a +) and the right side is what was not good (marked by a —). The team then calls out what their opinions are and the scrum master documents this on the paper. Sometimes there are very hands-on issues such as writing better comments during check-in, but there can also be softer issues such as “improve communications in the team.”

Based on this retrospective, the scrum master and the team select a few issues from the bad side and commit to improve on these. Issues that need to be taken care of are documented as tasks or impediments in TFS so that we can follow up on them and assign them to the correct person.

Summary

The goal for this chapter has been to show you how we can use TFS for agile project management. The Scrum template in TFS is a very good implementation of Scrum and we can recommend that you try it for yourself.

We have seen how the PO sets off to work on a new project from requirements to the actual start of the project. The focus has been on the PO because that role involves many traditional project management tasks, such as requirements, ROI, time plan, staffing the project, planning the project, and so on.

There are numerous reports and queries in TFS that are helpful to any PO or scrum master and worth the attention of the development team(s). These will be covered extensively in the rest of the book.

[1] In Greek mythology, Theseus used a red thread to mark his way through the Labyrinth. http://en.wikipedia.org/wiki/The_Red_Thread.

CHAPTER 9

Metrics and ALM Assessment for Agile Project Management

After most sections in this book, we will have a short chapter going over the metrics we can get from TFS and also show some examples of the ALM assessment questions in the online assessment Microsoft provides (see Chapter 4) so that you can see how these questions are connected to reality.

This chapter will cover the project management areas, and we will start by looking at some metrics you can use.

Metrics

A Key Performance Indicator (KPI) is a performance measurement and is used in most organizations to evaluate the organization’s success or the success of a particular activity within the organization. Often, KPIs can be used to measure the effects of a change project, for instance, implementing a good ALM process, or they can be used to evaluate the progress of a development project.

You can use the score from an ALM online assessment as a KPI and compare the assessment scores before and after the implementation of an ALM process improvement. This way, you will get indications whether you have improved or not by implementing a new process.

During your projects, you can also use the reports from TFS to see if you are constantly improving your work or not. Continuous improvement, in our opinion, is something to strive for. When it comes to the project management area, you can, for instance, look at the velocity (covered in Chapter 8) of a team and see if it is growing or decreasing. By using reports and metrics from TFS, you can choose the KPIs you want and how to evaluate them.

Standard Reports

To help you to get good metrics about the status of your projects, TFS has many reports available out of the box. The process templates of the reports differ, but when you look more closely at them, you can see that much of the same information is displayed in them. Let’s start by looking at the reports you can use for each process template.

Scrum

Basically, there are four important reports for the Scrum template in TFS:

- Backlog overview

- Sprint burndown

- Velocity report

- Release burndown

The backlog overview report lists all user stories, filtered by area and iteration and in order of importance.

We covered the sprint burndown (Figure 9-1) in our agile chapters. This report shows how much work there is left to do in a sprint. Using it you can predict when the team will be finished with the task, within the sprint or after the sprint is finished. Based on this information, the team and the Product Owner (PO) can take actions to make sure they deliver what they have committed to. The release burndown chart (Figure 9-2) shows the exact same thing as the sprint burndown but for the work included in a release.

Figure 9-1.  Scrum sprint burndown report

Figure 9-2.  Scrum release burndown report

Figure 9-3.  Scrum velocity report

Chapter 8 showed that velocity, that is, how much work a team can take on in a sprint, is important. Before any work is started, the PO calculates a theoretical velocity just for being able to start planning. As time goes by, however, this is updated with the team’s real velocity based on how much work they deliver in each sprint. This helps you estimate how much work they can take on in coming sprints. The velocity chart will help you easily retrieve this figure. Here, we will see how much effort the team has delivered for each sprint.

MSF for Agile

The following are the reports for MSF for Agile:

- Burndown and burn rate

- Remaining work

- Status on all iteration

- Stories overview

- Stories progress

- Unplanned work

Burndown and burn rate chart (figure 9-4). No surprises here; this is the same information as for Scrum. The burn rate provides summaries for the completed and required rate of work for a specified time period. You can also see the information for team members as well. You can choose to see the report based on hours worked or number of work items. As users, you must select the start and end date of the report yourselves; you cannot select a sprint like in the agile template.

Figure 9-4.  Burndown and burn rate report

Remaining work (Figure 9-5). This report can be used to track the team’s progress and identify any problems in the flow of work. You can view this report in either the Hours of Work view or the Number of Work Items view.

Figure 9-5.  Remaining work report

The status on all iterations report (Figure 9-6) will help you track the team’s performance over successive iterations. For each iteration, you can see information on how many user stories you have closed. You can see the progress of the team with information on the original estimate, how much effort has been needed, and also remaining work. This may not be agile in the strictest sense, since only work remaining is important, but it can serve its purpose anyway. This report will also show the bug status (active bugs, resolved bugs, and closed bugs).

Figure 9-6.  Status on all iterations report

The stories overview report (Figure 9-7) and stories progress report (Figure 9-8) list all user stories, filtered by area and iteration and in order of importance. You can see how many hours are completed for a user story and how much work is still remaining.

Figure 9-7.  Stories overview report

Figure 9-8.  Stories progress report

The unplanned work (Figure 9-9) report is useful when the team plans an iteration by identifying all work items that they intend to resolve or close during the course of the iteration. The work items that are assigned to the iteration by the plan completion date of the report are considered planned work. All work items that are added to the iteration after that date are identified as unplanned work.

Figure 9-9.  Unplanned work report

MSF for CMMI

Most of the same reports are available for MSF for CMMI as for MSF for Agile so we will not cover them again. There are, however, some different reports:

- Requirements progress report

- Requirements overview report

The requirements progress and overview reports list all requirements, filtered by product area and iteration in order of importance. You will see how much work is planned and how much is still left, very much like the user stories reports from MSF for Agile.

Note  The reports in this section are mapped to the particular process template it supports. If you find a report you like, it is possible to customize it to work with another template. In Chapter 32, we describe how to customize an existing report to do this among other things.

All the reports for all process templates that we have described in the preceding section will help you through the development projects, giving you the information you might need when you need it based on the project management process you have chosen. If you have customized your process, you need to make sure the reports are updated as well.

Custom Reporting

The reporting capabilities in TFS give you access to most of the information you manage in your ALM process. In the previous section, we showed how standard reports give you metrics for your project at a general level. By customizing, extending, and creating new reports, you can really get the intelligence to know what works well in your projects and what does not.

In this section, we will go through the relevant data warehouse models which you can use to create custom reports for project management. Understanding of these models is essential if you want to be able to gather metrics for your reports, so we recommend spending some time investigating the models.

Note  In Chapter 32 we will look at the details of reporting in TFS, including how to create custom reports based on the data models described in the following section.

Data Warehouse Model

The data warehouse for around-project management is mostly work item data and relationships to information such as associated changesets and test results. In the following sections, we will look at how these tables can be used to give current information as well as data[1] for trend analysis.

Current Work Item Tables

You can use the current work item tables to query for data about the current states of requirements, bugs, tasks, and other work item types.

Figure 9-10.  Current work item data model

Work Item History Tables

The work item history tables give information about historical data about the different work item types. You can use these tables if you want to create a report on how a field has been changed over time (for auditing purposes, for instance) or to create progress and burndown charts.

Figure 9-11.  Work item history data model

Work Item Link History Tables

You use the work item link tables to query for relationship information from the work item warehouse, for instance if you want to create a report over the status of tasks for a give product backlog item.

Figure 9-12.  Work item link history data model

Work Item Category Tables

These tables can be used to query for data about work item categories.

Figure 9-13.  Work item category data model

Work Item Changeset Tables

The work item changeset tables give data about the links between work items and files checked in to version control.

Figure 9-14.  Work item changeset data model

Work Item Test Result Tables

You use the work item test tables to query for work item data related to test results, for instance to get a report on which requirements have been tested and the outcome of the tests.

Figure 9-15.  Work item test result data model

Assessment Questions

In order to help us evaluate an organization’s maturity in different ALM areas, Microsoft has developed their ALM assessments, which may be read about in Chapter 4. Based on the score of the assessment, we receive a maturity level for a specific area. There are four levels:

- Basic, which means that the organization has brittle, disparate applications and platforms. Much is homegrown and processes are not documented. Much is performed ad hoc.

- Standardized, which means the organization has standards-based flexible business applications. Processes are starting to come into place but not every department follows them.

- Advanced, which indicates more adaptive application platform driving core applications and business processes. Here, processes are followed and maintained, best practices are used; this is what most companies might strive for.

- Dynamic, which is a fully dynamic application platform. Very few companies reach this level, but maybe it is not necessary to either.

So, based on the score, you can help the organization reach the maturity level they need for these areas. The following table (Table 9-1) list questions that can be used as a basis for an ALM assessment in the project management planning phase. These questions are taken from the online assessment. Keep in mind that some of these questions are asked based on a Waterfall approach to development. These are useful questions no matter if the organization uses an agile method or not.

Table 9-1. ALM Assessment questions

Area

Sample question

Discussion

IT Governance Maturity

Are IT governance frameworks (such as ITIL, CobiT 4.1, ISO17799) known and applied by the company?

What you want to find out with this question is if the organization uses a structured IT governance Process. If they do not have that, then maybe you need to look into that in your ALM improvement project.

Application Portfolio Management

Is the customer’s business case known and is it being delivered?

This is important to have. If you are planning an agile process implementation, make sure that the product owners are aware of the outcome of any improvement in this area.

Is there a consistent enterprise directive on how to align the vision of a project to business needs?

Really important to any successful project. Pay attention to the customer’s business needs and see how they can map onto the ALM vision.

Are any KPIs defined?

A product owner should have this information available for projects.

Do existing tools roll up into a broader IT Portfolio Management solution?

If not, suggest using Microsoft solutions      for this.

Compliance Management

Are compliance requirements identified and tracked by accountable resources within the enterprise?

This is not only an important question during an ALM assessment but also good input for the development team when it creates its definition of “done” (DoD) together with the product owner. Having a good definition of “done” is essential for all development processes, not just for Scrum.

Requirements Elicitation

Does a “vision” document exists to guide requirements?

Having some kind of vision of projects is always good. Formalizing it by using a vision document might not be necessary. However, it is essential that the team shares the stakeholder’s vision.

Is UML (or similar) used?

Are tools to standardize the capture and sharing of customer requirements in place?

Suggest that the customer can start using TFS and PowerPoint Storyboarding.

Requirements Analysis

Are modeling techniques used to extract product requirements from customer requirements?

Inquire whether there are modeling techniques in place that could be used to help the organization get started with storyboarding, for instance.

Are formal user acceptance tests used?

The answer to this affects the definition of “done.”

Requirements Management

Are requirements updated as they change?

If not, this is definitely something that most organizations would benefit from.

Do you capture requirements in an artifact (for example, use case, user story, wireframe, acceptance criteria, etc     .)?

If not, the customer needs help gathering requirements. Discuss with them and find out the best way to help them with this.

Are you able to adjust to the changing requirements of the customer?

If not, this is definitely something that most organizations would benefit from.

Do you manage requirements using a tool?

If not, help the customer and suggest using TFS for this.

Traceability

Is there traceability between requirement and product?

Traceability is a cornerstone in ALM. This is important, so do not let this question slip.

Project Initiation

Does the customer have a project champion or owner, and are they actively involved?

In an agile process, this is important. The product owner needs to be involved always. If the customer does not have this, help them realize why this is important.

Are methodologies and best practices being followed?

Most companies have processes and methodologies but many times they are not followed. Discuss and show the benefits of automating the processes using a tool like TFS.

Do team members have access to project documents and plans?

Transparency is important and is emphasized is the agile communities.

Is project management based on a formal methodology?

If the organization has no methodology, discuss with the customer and find out what process will benefit them the most.

Project Planning

Is the project plan complete and does it reflect reality?

Sounds really Waterfall to us. Planning is good, but if the plan does not change when reality changes, then what good is it? Discuss and suggest improvements.

Is formal resource estimation used?

Is the project’s vision and scope well defined and understood and does it map to the business problem?

Are project budgets formally approved?

Is project documentation versioned and archived?

Here, we readily suggest using TFS and SharePoint to improve this.

Are the internal success factors known and understood?

Project Monitoring and Control

Is project status tracked against the project schedule?

If you go for an agile approach, please discuss the metrics in TFS, like the burndown chart and other reports.

Are metrics used to manage the project?

Again, this is something that TFS can help the organization with. Discuss needs and suggest a way forward using TFS and maybe some custom reports.

Are all deliverables clearly defined and updated?

TFS certainly helps with this.

Are there unexpected changes in personnel?

Aren’t there always?

Do individuals work on several projects concurrently, often switching between them during any given week?

If this is the case, the customer will benefit from assigning individuals to one project at the time. This can sometimes be hard, but the context switching between projects takes its toll on productivity.

Is status reporting largely automated?

Here is another area where TFS will help the customer.

Is the customer able to make decisions in a timely way?

A development team needs to have answers to questions, and it is essential that the product owner is able to make decisions quickly.

Is the customer able to provide information in a timely way?

A development team needs to be able to answer questions quickly.

Are risks known and actively managed?

Risk management is essential. Ad hoc solutions rarely work. So, suggest that the organization use either Scrum, where risks are assessed and mitigated constantly, or another good risk mitigation strategy.

Is risk assessment carried out regularly?

Here, we always argue that by using an agile method you will manage risk constantly.

Stakeholder Management

Are all the internal stakeholders known when a project starts?

Are the external stakeholders known when a project starts?

Project Close

Have all the deliverables been formally accepted? Do the outputs match the defined acceptance criteria?

Here is another question where the suggested answer could be to have a good definition of “done” in place. Fulfilling the DoD means that the output matches the defined acceptance criteria.

Has a post-project review been held? Have lessons learned been identified, documented, and shared/published?

Doing a retrospective after a project or after a sprint is a good way to learn how to improve for the next project or sprint.

Has all the project documentation been archived?

Use TFS and SharePoint.

These tables also feature some input from us for some questions. We have chosen not to comment on all questions but will focus on some of the most important ones for each area.

Summary

In this chapter, we have looked at the concepts of metrics and assessment for agile project management. We have seen how you can use TFS to retrieve information for KPI assessment and also how you can see the project status using standard reports from TFS.

We have also shown how many of the assessment questions from the Microsoft online assessment can help us plan for successful implementation of project management practices.

The next section of this book will cover the architecture and design topics of development using TFS.

[1] If “data” not correct here, please clarify “as well as date for trend analysis.”

PART 3

Analysis, Modeling, and Design (Architecture)

Part III covers how we can plan for the development of our application or system using many of the tools in Visual Studio 2012. We begin by looking at how to use the PowerPoint add-ons that Visual Studio offers to create a storyboard. Storyboarding visualizes (or models) the flow of our application and can be a great way to sit down with a non-technical stakeholder and gather requirements and information. Storyboards are also great as a foundation for design discussions during sprint planning if you are using Scrum.

The Unified Modeling Language (UML) is a modeling method many people are used to. Visual Studio 2012 has UML support included and can be used in various stages of both planning and development. We see some basic UML modeling and how we can use Visual Studio to create these models.

Many developers find themselves with old code that they need to maintain. Poor documentation can make this task hard. Using the Architecture Explorer we can visualize existing code as well as new code, making it more readable and understandable.

Layer diagrams are part of the architecture tools in Visual Studio Ultimate and they are used to model the layers in a solution. Once the layers are defined we can define the dependency structure between the layers and map code to the corresponding layer. With this information in place, the model can be used not only as documentation but also to enforce that modules in the application do not take dependencies outside what is defined in the model.

CHAPTER 10

Prototyping Using Storyboarding and Feedback Tracking

As you saw in Chapter 2 there is more to ALM than the three pillars (Traceability, Visibility, and Automation) of high-level processes. You also saw that the following four topics are important:

- Collaboration

- Workflow

- Security

- Reporting and analysis

Of these four, the features of the new PowerPoint Storyboarding tool will help you to increase collaboration between stakeholders and the development team, as you can avoid confusion as to what the requirements really are. This chapter looks at how this feature works and how it can help to improve collaboration as well as traceability.

This chapter also covers the Microsoft Feedback Client, which is another part of TFS 11 that helps with collaboration and traceability in our organizations. The Microsoft Feedback Client allows us to launch an application, capture our interaction with it as video, and capture our verbal or type-written comments as well. The feedback is stored in Visual Studio 11 Team Foundation Server (TFS) so that we have good traceability.

PowerPoint Storyboarding

Requirements gathering is tough. Often stakeholders and end-users do not know what they want until they see a prototype or some early builds of a system. It’s then that the ideas and the creativity start flowing. User stories are a great way to capture high-level requirements without locking the development team into a particular technical solution. Using storyboarding, in addition to user stories, is a good way to add clarity to requirements by attaching visual drawings of the system. In TFS 11 we can create storyboards using Microsoft Office PowerPoint and attach (link) these to our User Story work item (Product Backlog work item). Of course we can use storyboards with the other process templates as well. These might not contain Product Backlog Items, but instead have other names for their work items. Providing a visualization of what the team is going to build allows us to more easily retrieve feedback from our team and stakeholders.

Note  The idea of storyboarding was developed at the Walt Disney Studio during the early 1930s. Disney credited animator Webb Smith with the idea of drawing scenes on separate sheets of paper and pinning them up on a bulletin board to tell a story in sequence, thus creating the first storyboard (http://www.instructionaldesign.org/storyboarding.html).

In more recent days, this technique has been applied to software development as well. A storyboard is a visual expression of everything that will be contained in a software application. It can show the flow of an application before it is developed, giving stakeholders and end-users the chance to experiment with the flow to see how it will affect the application or system.

Storyboarding minimizes the risks inherent in all software development projects: missed delivery deadlines, exceeded budgets, deliverable ambiguity, and client expectations so that we finally can end up with a product with minimum bugs. There are numerous techniques and tools for storyboarding. The aim of this book is not to teach storyboarding, but to show how TFS 2012 supports this way of working. Hence, we will not dive into storyboarding itself.

How to Do It

To start using PowerPoint Storyboarding go to the Start menu and click Microsoft Visual Studio 2012. Select PowerPoint Storyboarding and click (see Figure 10-1). Make sure you have Microsoft Office PowerPoint installed before you try this.

Figure 10-1.  Starting PowerPoint Storyboarding

If you do not want to use Visual Studio to start storyboarding you can use web access instead. Navigate to the project web access page in your browser. Create a new Product Backlog Item and go to the Storyboards tab in the lower left and click Start Storyboarding (see Figure 10-2).

Figure 10-2.  Starting PowerPoint Storyboarding from the project web access page

After PowerPoint has loaded you find a collection of predefined storyboard shapes that you can use to create your storyboard (see Figure 10-3).

Figure 10-3.  The storyboarding collection of PowerPoint shapes

To create your storyboard, you can drag and drop images from the Storyboard Shapes pane. You can also use all the features present within PowerPoint itself. These features include clipping and inserting screenshots. You can also hyperlink from one page to another, do animations, insert images and shapes, and align and group objects. These features are very powerful and will help you gather better requirements.

Here is a valuable tip. When you add placeholders for texts in the UI it’s common to use the “lorem ipsum. . .” Latin text.[1] There is even a macro in PowerPoint that helps you with this. Just type =lorem() in a textbox and it expands to a fancy Latin fragment! (See Figure 10-4.)

Figure 10-4.  Adding placeholders for text

After you are satisfied with the storyboard, save it. Keep in mind that you must save it on a network share, otherwise TFS cannot link a work item with the storyboard.

Now go back to the new Product Backlog Item (PBI) in your team project using either Visual Studio or web access. Go to the Storyboards tab in the lower left of the PBI. Click Link To and browse your way to the storyboard on the network share. After successfully linking to the storyboard, you will see the linked file in the PBI (see Figure 10-5).

Figure 10-5.  Sucessfully linked a PBI to a storyboard

When to Use Storyboarding

Okay, so now you know how to create storyboards using PowerPoint. When can you use this feature? From an agile perspective, you have several options. If you remember the requirements workshop our friends in Kangreen held in the beginning of their project in Chapter 8 they could definitely use this functionality during the workshop. They could have sat down and done the storyboards live during the meeting, coming up with new suggestions or improvements as they went along.

Another situation when this feature would be great is during the stakeholder meetings that the product owner holds. Together the PO and the stakeholders can discuss several options and agree on a design.

During sprint planning the PO could also quickly draw up a storyboard and discuss it with the team, clarifying any questions the team might have.

Storyboarding is a very useful technique that all projects can benefit from. Accessing a digital storyboard tool adds even more benefits.

Feedback Tracking Using Microsoft Feedback Client

Feedback can be hard to retrieve sometimes. You need to schedule meetings, find somewhere to meet, and preferably be located at the same location. To make feedback requests easier and more frequent, Microsoft has added a nice feedback feature to TFS 2012. This feature allows you to send feedback requests to other persons, such as stakeholders or users, via email. The person then gives feedback and returns their feedback.

Go to the web access page of your team project. Under Activities, click Request Feedback (see Figure 10-6).

Figure 10-6.  Request Feedback is available from the web access page

If you don’t see the Request Feedback link, then alerts have not been set up on your TFS server. To fix this, ask your TFS administrator to go to the TFS administration console and enable Email Alerts (see Figure 10-7).

Figure 10-7.  Configuring email alerts

Now fill out the form that appears (see Figure 10-8). First select which stakeholder should give the feedback. Then select how the stakeholder should access the application. There are three options:

Figure 10-8.  Sending feedback request

- Web application

- Remote machine

- Client application

In this case we have selected a remote machine and attached the link to our storyboard. Because the feedback is on a network share, the stakeholder can access it from his or her machine. We can also add login information or any information the stakeholder needs to access the application.

Next tell the stakeholder what to focus on during the feedback session. This could basically be anything, from compliance with company profile to whatever you need feedback on.

Note  You can send multiple items for review at the same time. You don’t need to create one request for each, although, from a traceability perspective this might be the best.

The stakeholder receives an email from the sender. This email includes a link to the feedback session. After the stakeholder clicks this link, the Microsoft Feedback Client starts. If the client is not installed, there is a download link for downloading and installing it (see Figure 10-10).

Tip  If your company policy includes firewall rules that do not allow external application downloads, you can change the link by following these steps:

- Use remote desktop or equivalent to access all servers in the TFS Application Tier

- Use Windows Explorer and go to C:\Program Files\Microsoft Team Foundation Server 11.0\Application Tier\Web Services\_static\tfs\11\_scripts\TFS (or the location where TFS was installed)

- Download the feedback client from http://www.microsoft.com/visualstudio/11/en-us/downloads#feedback-client and save it to a network share

- Edit the original TFS.Requirements.Feedback.Models.min.js (make sure you save a copy first, in case something goes wrong)

- Search and find “INSTALL_FEEDBACK_TOOL_URL”

- Replace the default value with a link to the saved feedback client exe file. Please note that this is a file we are referencing not a web site. It could look like this: file://MyCompanyShare/vs_feedbackclient.exe.

- Save the TFS.Requirements.Feedback.Models.min.js file

- Test that the feedback client link has changed

The Feedback client includes all information that you provided in the previous feedback request. For instance, any credentials you might have supplied for starting or accessing the application. When the stakeholder is ready he or she clicks Start (see Figure 10-9) to start the feedback session.

Figure 10-9.  Starting the feedback session

This starts the feedback client, which contains three steps as seen in Figure 10-10:

- Start, which starts the actual feedback session

- Provide, which gathers the feedback

- Submit, where the stakeholder submits the recorded feedback

Figure 10-10.  The three steps of the feedback client

By following the instructions in the feedback client the stakeholder has several options to record the feedback, including (see Figure 10-11):

- Recording screen interactions with an application

- Recording voice comments

- Using rich formatted text

- Including a screen shot or any other file

- Rating of the feature using stars (see Figure 10-10)

Figure 10-11.  Recording feedback in the feedback client

Once we are finished with the feedback session we can either submit it or save it for later additions. Before submitting feedback we have the option to review the feedback. If the stakeholder is satisfied with the session he or she can submit it. Next a feedback response work item is created in TFS and the feedback request creator can go to web access and look under shared queries. There is a Feedback Requests query that returns all feedback requests and responses (see Figure 10-12).

Figure 10-12.  Feedback requests query

When a feedback response work item is opened, all feedback given is attached as linked images (http://msdn.microsoft.com/en-us/library/hh301769(v=vs.110).aspx). To review the feedback:

- Choose the appropriate recording for the recording that you want to review.

- Select the link that appears after Navigate To at the bottom of the form.

- Click the Open button. The recorded feedback session is downloaded to your computer and opens in your default media player. You need the Microsoft Expression Encoder 4 installed as well.

- You can play, pause, and resume viewing the recorded session.

Summary

Gathering requirements is always hard. It is often problematic to have a stakeholder or end-user stating what they want unless they have something to relate to. Storyboarding is one way to make this process easier. By storyboarding the flow of an application or a system it is easier to get a feel of the system and thus the ideas and creativity seems to start flowing. The agile practices build on this fact, and by using agile practices in combination with techniques such as storyboarding, we can make the requirements gathering more efficient and hopefully capture the requirements more correctly.

The support for storyboarding in TFS and Office PowerPoint is a step forward in the requirements gathering process using Microsoft tools. This feature is really useful and we will definitely use it as a replacement for other storyboarding techniques in future projects.

Retrieving feedback from stakeholders is essential in any project. By using the feedback tracking features of TFS 2012 we have a solid way of both requesting and retrieving feedback.

Both these tool sets are valuable additions to TFS and Visual Studio that enable even better collaboration between stakeholders and development teams. Exactly in the spirit of ALM.

[1] Please see http://www.lipsum.com/ for more information regarding this Latin text.

CHAPTER 11

Top Down Design Studies (UML)

No matter how skilled you are at coding, you still need some structure during the process of building a system. If you want your new solution to be a success, you need to keep the quality high, and make sure it meets the needs of the users. If the designers, developers, and users do not speak the same language, you can be certain you will have problems in the end.

One of the worst things that can happen in a big project is when people think they mean the same thing, but they really don’t. This is something that could happen to us in our everyday life as well of course, but for a project it might make the difference between success and failure, because eventually this might lead to disagreement over which features to include in the solution, and thus make a perfectly functional system a failure.

As consultants, we cannot afford to let this happen. If our customers are not happy with the result, we will have difficulties getting a project from them again. This is why it is so important to agree about the requirements of the system at the moment you write them down. Of course you need to adapt as requirements change, but that’s what we use agile practices for.

In Domain Driven Design (DDD) there is a concept to solve just this problem (http://domaindrivendesign.org/node/132) and it is called ubiquitous language. Ubiquitous language is a language structured around the domain model and used by all team members to connect all the activities of the team with the software.

Having methods to help us describe the requirements is very helpful. For example, storyboarding describes the flow and UI of a system. The Unified Modeling Language (UML) is another method to describe the requirements at a detailed level.

Probably the best comprehensive[1] overview of UML is The Unified Modeling Language User Guide, 2nd Edition by Grady Booch, James Rumbaugh, and Ivar Jacobson (Addison-Wesley, 2005. ISBN: 0321267974)—unsurprisingly, as the authors collaborated on the design of UML. In this book, they state that four goals are achieved with modeling:

- Models help you visualize the system as it is or how you want it to be.

- Models permit you to specify the structure or behavior of a system.

- Models give you a template that guides you in constructing a system.

- Models document the decisions you have made.

As you can see, you have much to gain by using this technique—and large, complex systems are not the only ones that benefit from modeling either. Modeling helps even smaller systems, as many systems have a tendency to grow and become more and more complex as they go along. With a model, you can grasp this complexity, even after a long time has passed since the inception of the first version of the system.

A good set of UML diagrams can also help you find bottlenecks early in the design process. The sooner you eliminate these bottlenecks, the less they cost to get rid of.

You must also think about modeling at different levels. Sometimes you need a high-level view of the system, and sometimes you need a low-level view. You need, for example, one view when showing the solution to decision makers and another one when talking to developers.

Modeling can be difficult, however. If you aren’t careful when you choose what to model, your models might mislead you and make you focus on the wrong things. Because models are a simplification of reality, it might be easy to hide important details. To avoid this, you need to make sure you connect your models to reality. If you have a weaker connection in one place, you must at least be aware of it.

UML is a standard for writing software blueprints, and as the name implies, it is a language. UML can be used to visualize, specify, construct, and document the deliveries (or artifacts) of a software-intensive system. But remember that because UML is only a language, it is just one part of the development cycle.

One drawback of models is that they have a tendency to become obsolete at the same time as development starts. Nobody really focuses on keeping them up to date. And do we actually have use of them after we have developed the system (or application)? If we have used test-driven development, many problems such as seeing the effect of a bug fix or new version are eliminated. We have heard many developers argue that modeling is useful during sprint planning, but then they throw away the model, as it is no longer useful. Although this is very often true, the use of models and modeling during the design phase could be helpful. It really doesn’t matter whether or not we keep them; use them where you think they are appropriate.

UML Basics

If you have little experience in UML, here comes our crash course just for you. It is by no means a complete explanation, but it is important to have an understanding of these concepts because Visual Studio has tools supporting UML. At the same time, you will see some UML features in Visual Studio.

UML Support in Visual Studio 2012

Let’s take a look at which parts of UML are supported in Visual Studio 2012 Ultimate. Visual Studio (VS) provides templates for five of the most frequently used UML diagrams: Unified Modeling Language (UML)

- Activity diagram

- Class diagram

- Component diagram

- Sequence diagram

- Use case diagram

UML modeling diagrams and layer diagrams can exist only inside a modeling project. Each modeling project contains a shared UML model and several UML diagrams. Each diagram is a partial view of the model. The UML model contains all the elements in the UML diagrams and can be viewed by using UML Model Explorer.

This means that you must create a modeling project in your VS project and then start adding the UML diagrams you want (see Figure 11-1) by going to the Project menu and selecting Add New Item.

There are some exceptions to this according to http://msdn.microsoft.com/en-us/library/dd409445.aspx:

- Sequence diagrams that are generated from source code.

- Class diagrams that are created as views of the source code. These are not related to UML class diagrams.

- Dependency graphs.

- Diagrams that are not UML diagrams or layer diagrams, such as domain specific languages.

Figure 11-1.  The UML diagrams in Visual Studio

When a UML diagram is selected, a toolbox with the symbols needed for the specific diagram is presented in the left pane (see Figure 11-2).

Figure 11-2.  Toolboxes with symbols are available for all UML diagrams in VS

In the following sections, we’ll take a look at some of these diagrams and explain how to use them. This will be a simplified overview of how you can use them. Visual Studio, however, offers many complex UML scenarios if you want to create such.

Note  There is a Visual Studio feature pack that greatly enhances modeling and visualization in VS. We do not cover it here but take a look at http://msdn.microsoft.com/en-us/library/dd460723.aspx for more information.

Activity Diagram

An activity diagram shows the flow of control. Activities are action states that move, or transition, to the next state after completion. In Figure 11-3, you can see these activities as rounded rectangles. Our example shows a simple flow of control for the display of expense reports, which are also published on a web site. An arrow represents all transitions between the activities. To show activities performed in parallel, we use synchronization bars (Fork Node and Join Node in VS).

Figure 11-3.  An activity diagram showing activities as rounded rectangles

These diagrams are really flowcharts used early in the design process to show the workflow between use cases. You are not required to use them at this point in the process, however. You can use them where you feel they are appropriate, or when you think they explain a flow in a system so that a decision maker can understand what you mean.

In your activity diagrams, you can use something called swim lanes to show ownership. In Figure 11-4, we have added a swim lane to our activity diagram to show the responsibilities of employees and managers.

Figure 11-4.  A swim lane added to our activity diagram

The employee handles everything but the approval of the report. That task is left to the manager. So now we have shown the responsibilities of all parties in this extremely simple scenario.

Use Cases and Use Case Diagrams

When you create use cases, you first have to decide which actors will participate in your solution. An actor can be a person (or group of persons), but it can also be another system—that is, something that interacts with the system, but is outside the system. We represent our actors with stick figures, almost like the ones in The Blair Witch Project, if you are familiar with that movie (see Figure 11-5).

Figure 11-5.  Actors in a use case diagram represented as stick figures

Let’s continue our simple scenario from the previous section. We have already mentioned two actors: the employee and the manager. Do we have anyone else? One actor that immediately comes to mind is the administrative staff. Another one might be a PDF application that creates the PDF files of the reports

The next thing to do is find your use cases. The easiest way for you to do this is by looking at your actors and asking yourself why they want to use the system. A use case in itself is a description of actions that a system performs to give an actor the result of a value. In our example, they are portrayed as ovals, as you can see in Figure 11-6.

Figure 11-6.  Actions that a system performs for the actors

In our example, the employer creates an expense report. The manager approves the expense report so that the employee can get his or her money. The admin staff can look at the reports and perform any administrative tasks that they need.

Now, after you have identified some use cases, you can start documenting them. This is done by describing the flow of events from the actors’ points of view. You must also specify what the system must provide to the actors when the use case is executed. The use case should show the normal flow of events, but it should also show the abnormal flow—that is, when something goes wrong during execution and an alternate scenario takes place. (The scenarios are on a high level, so you cannot catch all possible actions.)

Often, use cases are great for showing the people paying for the system what actually will happen in it. This way, you can be assured early in the process that what you are building is what the customer expects.

Let’s take a look at the expense report creation process. This starts when the employee logs in to the system. The system verifies the password and then shows the form for the Create Expense Report form. The employee can choose from various alternatives what he or she intends to do:

- Create new expense report: Create new expense report subflow executes.

- Modify expense report: Modify expense report subflow executes.

- Delete expense report: Delete expense report subflow executes.

- Review expense report: Review expense report subflow executes.

- Exit: Use case ends.

When you’ve created your use case diagrams, you’ll have a good overview of the system. However this does not guarantee that no surprises will occur along the line.

Sequence Diagrams

Sequence diagrams are used to illustrate the dynamic view of a system. They show the interaction and the messages sent between objects in chronological order (see Figure 11-7).

Figure 11-7.  A sequence diagram of the addition of a new product to a campaign

First, you place the objects that participate in the interaction at the top of the diagram across the x-axis. The object that initiates the interaction is typically placed to the left, and the more subordinate objects to the right. The messages sent are placed to the left along the y-axis (time). Our example here shows a simplified version of creating an expense report.

Sequence diagrams are great for showing what is going on in a process. In collaboration with the customer, they can help in mapping out the requirements of the system. You can do a sequence diagram for every basic flow of every use case, and keep doing them until you think you have enough. Enough is basically when you cannot find any more objects and messages.

VS has different symbols for helping you create your sequence diagrams as seen in Figure 11-8.

Figure 11-8.  The symbols for sequence diagrams in VS

Class Diagrams

This is one of the most common diagrams used. Using class diagrams you can model the static design view of a system. A class diagram shows classes that are collections of objects with a common structure, behavior, relationships, and semantics. In UML, classes are represented by a rectangle with three compartments, as shown in Figure 11-9. The top compartment displays the name of the class, the middle one lists its attributes, and the bottom one indicates its behavior (or operations).

Figure 11-9.  A class diagram

You can choose to show one, two, or all three of these compartments. When you choose the names for your classes, try to maintain a standard throughout your project. You can, for instance, decide to use singular nouns, like Customer, with a capital letter at the beginning. How you choose, or what you choose, is not important. What is important is that you stick to your standard, so as to avoid confusion later on.

What UML modeling elements do you find in class diagrams? You will probably have many class diagrams in your model, because they show the logical view of your system. They show which classes exist, and the relationships between them. They also show associations, aggregations, dependencies, and inheritance. From your class diagram, you can see the structure and behavior of your classes. You can also see multiplicity and navigation indicators. Figure 11-10 shows the class Expense Report and some of its attributes. In the real world, we would find out these attributes by talking to our stakeholders and looking at our requirements. For the purposes of our example, let us say that the class has five attributes: ID, Report_date, Expense, Expense_amount, and Total_amount.

Figure 11-10.  The class Expense Report and its five attributes

Next, you need to find some operations for your class. Operations are the behavior of the class. In Figure 11-11, we have hidden the attributes of the Expense Report class and show only the class name and operations. Because this is a simplified view of a class, only three are specified: Add, Delete, and Modify. There could, of course, be many more, just as there could be many more attributes. It all depends on our requirements and the input from the customer.

Figure 11-11.  The Expense Report class and its operations, with the attributes hidden

After you have your classes, you can start looking for the relationships between them.

Relationships

Three kinds of relationships exist in UML according to the document “Introduction to the Unified Modeling Language” by Terry Quatrini (http://www.rational.com/uml/resources/whitepapers/index.jsp): association, aggregation, and dependency. They all represent a communication path between objects. One could argue that inheritance should be counted as a relationship; however, we chose not to here.

Association is represented by a line connecting classes, as you can see in Figure 11-12. This is a bidirectional connection, which means that one class can send a message to the other, because if they are associated, they both know the other one is there.

Figure 11-12.  An association between two classes

A stronger form of relationship is the aggregation. In UML, you show this as a line connecting the related classes. The line has a diamond on one end. An aggregation shows the relationship between a whole (represented by a diamond) and its parts. When you, as a developer, see this kind of relationship, you know this means there is a strong coupling between those object classes.

The third kind of relationship is dependency. It shows that a change in the specification of one class may affect another class that uses it. Keep in mind that the reverse may not necessarily be true. Use dependencies when you want to show that one class uses another class.

To find your relationships, start by looking at your sequence diagrams. If you find that two objects need to talk to each other, they must have a way of doing this. This way is the relationship. The deeper you analyze your diagrams, the more you will know what type of relationship to use. A parent-child relationship will probably be an aggregation, and so on.

Multiplicity

Multiplicity simply states how many objects participate in a particular relationship. It shows how many instances of one class relate to one instance of the other class. Based on this, you need to decide for each end of an association and aggregation what level of multiplicity is needed. Because multiplicity defines the number of instances, you will represent it in your diagram with either a number or an asterisk (*). The asterisk is used to represent a multiplicity of many (see Figure 11-13). In our example, we could say that one employee can own several expense reports. This is known as a one-to-many relationship, which would be represented as 1 - *.

Figure 11-13.  A multiplicity of one to many

You decide multiplicity by examining your business rules. For our example, we could have a business rule stating that one employee can own only three campaigns. Then we would have a one-to-three relationship.

Inheritance is the relationship between a superclass and a subclass. In our example, if we have a class called Employee, we can have other classes like Manager or AdminStaff that are separate classes but still fall under the Employee class. That is, a Manager is an Employee. This relationship is shown with a triangle (see Figure 11-14).

Figure 11-14.  Inheritance between two classes

Be cautious when using inheritance. You do not want to build too many levels, because if you do, a change in one class affects many others. This could bring disaster to your products, because you might have to make changes in many places because of this. Build another level only when you are sure there is an inheritance situation, and do not include it just for the sake of having it.

With the knowledge of the topics you have gleaned in this section, you have come quite some way down the path of understanding UML. A good source of information about UML, and the topics we did not cover here, is the book mentioned earlier, The Unified Modeling Language User Guide, 2nd Edition by Booch et al.

Component Diagrams

Component diagrams (see Figure 11-15) are used to visualize how components are wired together to form larger components or software systems. The component diagram’s main purpose is to show the structural relationships between the components of a system.

Components are connected by using an assembly connector to connect the required interface of one component with the provided interface of another component. This illustrates the service consumer - service provider relationship between the two components (http://en.wikipedia.org/wiki/Component_diagram).

An assembly connector is a connector between two components that defines that one component provides the services that another component requires. An assembly connector is a connector that is defined from a required interface or port to a provided interface or port.

When using a component diagram to show the internal structure of a component, the provided and required interfaces of the encompassing component can delegate to the corresponding interfaces of the contained components.

A delegation connector is a connector that links the external contract of a component (as specified by its ports) to the internal realization of that behavior by the component’s parts.

Figure 11-15.  A simple Component diagram

Integration with TFS

One cool thing in TFS 2012 is that you can link a model to a work item in TFS. For instance you can right click the model and select create work item from the menu (see Figure 11-16). This way you can bind the model to the backlog (or a backlog item).

Figure 11-16.  Linking a model to a work item

Summary

Visual Studio offers great support for UML out of the box. UML models can be a great addition to the sprint planning meeting when the team starts to break down the user stories into more manageable bits and the design process starts.

However, models have a tendency to be created and then forgotten about as the development process goes on. And if we do not use them more than during a limited time period, the might not be so necessary.

If we have good automated unit tests and a good stable continuous integration process we might not need to have any models at all. We will easily find where a bug fix or new feature implementation will have its impact with the help of the features TFS offers.

In the next chapter we will take a look at the architecture explorer and see some exciting features TFS and VS offers there.

[1] If you would prefer a shorter and more digestible introduction, try UML Distilled: A Brief Guide to the Standard Object Modeling Language (3rd Edition) by Martin Fowler (Addison-Wesley, 2003. ISBN: 0321193687). Whichever book you choose, it is important to get the latest edition.

CHAPTER 12

Using Architecture Explorer

Developers often find themselves having to fix bug or perform maintenance on an application that they have never been involved with before. This situation offers many challenges. For instance, the developer must understand the code. This can be tough if there is no documentation available or if there are no good automated tests for the application.

Similar problems also face developers who join a new project and need to get up to speed with development. Or perhaps applications have grown out of control over time and the developers not longer recognize what the solution and its dependencies look like.

To ease these pains, Microsoft has introduced Architecture Explorer, which visualizes our solution and the artifacts and dependencies it contains. By artifacts, we mean the classical code artifacts such as classes, interfaces, and so on, as well as whole assemblies, files, and namespaces. Architecture Explorer lets us select those artifacts, display graphs with dependencies, and even navigate along those dependencies and in and out of detail levels.

This chapter will show you the basics of using Architecture explorer. We suggest that you explore it yourself if you really want to experience the real power of it. http://msdn.microsoft.com/en-us/library/dd409453(v=vs.110).aspx is a great start for learning Architecture Explorer in depth.

Architecture Explorer

Visual Studio can help us visualize and understand an existing application or system. Using Architecture Explorer, we can see how our code is organized, what the relationships in the code look like, and how the code behaves. These are valuable for anybody who needs to quickly understand the code base and how changes to the code will affect the application. This will help us estimate how much work a change will require and also how much risk this brings.

Using dependency graphs, which we generate using Architecture Explorer, we can see patterns and relationships in the code. Let us take a look at these graphs and see a little bit of what they can do for us.

Dependency graphs are simply files in Directed Graph Markup Language (DGML), which is an XML-based file format for directed graphs (http://en.wikipedia.org/wiki/Directed_graph). This means that they can be produced using any other DGML tool and then be displayed in VS.

Dependency Graphs

To visualize code and its relationships, you can create dependency graphs (see Figure 12-1) for source code or compiled code in Visual Studio Ultimate in several ways Architecture Explorer

Figure 12-1.  A sample dependency graph

- From the Architecture menu, choose “Generate dependency graph”. This will show an overview of the solution.

- Select items in Architecture Explorer, and on the toolbar choose create a new graph document. This shows the specific code areas that you select. If you instead choose new dependency graph with ancestors, you will see the items’ parent containers.

- You can also drag items from Solution Explorer or Architecture Explorer to an existing dependency graph. If you use CTRL while dragging, you will include the items’ parent container.

You can create dependency graphs for the following types of projects and files:

- Visual C# .NET and Visual Basic .NET source code and compiled code, such as .NET assembly (.dll) files and executable (.exe) files

- Visual C and Visual C++ source code, header (.h or #include) files, and binary files (managed or native),but this requires the Visual Studio 2010 Visualization and Modeling Feature Pack (http://msdn.microsoft.com/en-us/library/dd460723.aspx).

Let us take a look at what dependency graphs can give us. In a dependency graph, items are represented as nodes and their relationships as links. Containment relationships between items are represented by groups,

which are nodes that contain other nodes. Groups can be expanded or collapsed to show or hide their contents. For example, when you generate a dependency graph from the Architecture menu, containment relationships are displayed as groups (Figure 12-2).

Figure 12-2.  Groups (Containment relationship) in dependency graphs

You can also choose to display containment relationships as links (Figure 12-3). On the shortcut menu of the graph, choose group, and then turn off grouping.

Figure 12-3.  Containment relationships as links

Identifying Complex Code

You can run different analyzers on your dependency graphs in Visual Studio to help you identify code that might be too complex or that might need improvement.

To get started:

Make sure the graph layout is set to tree layout or quick clusters layout. Select a task ,and on the graph toolbar choose the tree layout, which will show the nodes in a tree structure. The graph arranges the nodes so that most of the dependencies will flow in one of the following directions:

- Left to right

- Right to left

- Top to bottom

- Bottom to top

Check that the legend box is visible. If it is not, open the shortcut menu for the diagram from where you select show legend. In the legend box, choose add, choose analyzers, and then choose one of the following:

- Circular reference, which will identify nodes that have circular dependencies on one another

- Find hubs, which will identify nodes that are in the top 25 percent of highly connected nodes

- Unreferenced nodes, which will identify nodes that have no references from any other nodes.

Using these analyzers, we can see if we have loops or circular dependencies so that we can simplify them or maybe break the cycles. We can also see if we have too many dependencies, which could be a sign that they are performing too many functions. To make the code easier to maintain, test, change ,and maybe re-use, we need consider refactoring these code areas to make them more defined. Maybe we can also find code that performs similar functionality and merge with this. If the code has no dependencies, we should consider if we need to keep it at all. If no one is using that part of the code, maybe it isn’t necessary.

Note  When you generate a dependency graph for the first time, Visual Studio creates a code index for all the dependencies that it finds. Though this process might take some time, especially for large solutions or graphs that have many links, this index helps improve the performance of subsequent operations. If the code changes later, only the affected code is re-indexed. See http://msdn.microsoft.com/en-us/library/dd409453(v=vs.110).aspx for more information.

Analyze and Explore Code Using Architecture Explorer

We can use Architecture Explorer in many situations, not only when we need to understand unfamiliar applications or systems. In some projects, the development team has lost control over the code. It has grown out of control and it is hard to follow the structure and the dependencies. It might even be hard to understand what the code really does. Especially if the team does not use any XP practices—such as like test-driven development, automated unit tests, continuous integration, continuous build, and so on—it is easy to lose control.

We can use Architecture Explorer to browse and find source code in many Visual Studio projects and also in compiled code such as executables, assembly files, and binary files. In combination with dependency graphs, we can more easily get back control of our code.

To understand how we can use Architecture Explorer, let’s see what it looks like. On the Architecture menu, point to Windows, and then click Architecture Explorer. Open a solution (or compiled code), and the view we see in Figure 12-4 will appear.

Figure 12-4.  Architecture Explorer

In Architecture Explorer, structures are represented as nodes and relationships as links. Architecture Explorer displays nodes in new columns as we continue browsing. The first column shows the domains and views that we can browse among, and when we select a domain and a view, the available nodes in that view appear.

When a node is selected in a column, the next column will show the nodes that are (logically) related to that selection. An easy example is selecting namespaces. Doing this will show types in the next column (Figure 12-4). If we select a type, we will see its members in the next column.

The collapsed action column that appears along the right side of a node column identifies the default types or relationships of the nodes that the next column contains. We can filter the results of the next column by expanding the action column and selecting specific types of nodes or links.

How do we find source code in Architecture Explorer? We can use either of two views:

- Class View, which shows the logical view of a Visual Studio solution. We can browse by namespace, classes, members, and so on. (Figure 12-5)

- Solution View, which shows the physical view of a Visual Studio solution. We can browse by project, project files, and so on. (Figure 12-6)

We can use these views to browse source code down to the level of expressions.

Figure 12-5.  Class view in Architecture Explorer

Figure 12-6.  Solution view in Architecture Explorer

We can achieve the same result if we compile code that we want to investigate. The difference is that we do not open a Visual Studio solution; instead, we point Architecture Explorer to the compiled code by looking under file system, clicking select files, and selecting the file we want to look at.

Summary

This chapter has given an overview of Architecture Explorer and how it can help us. When we get our hands on an existing application or system (either source code/VS solution or compiled code), we can use Architecture Explorer to generate dependency graphs. These will visualize the code and its relationships so that we can understand the code more easily.

We can also use Architecture Explorer to find code by selecting different views (class view or solution view) of the code. In combination with dependency graphs, we can more easily get control of our code. Architecture explorer can be a powerful solution if we have no other way of understanding the code.

The next chapter looks at Layer diagrams, part of the architecture tools in Visual Studio Ultimate that are used to model the layers in a solution. Once the layers are defined, we can define the dependency structure between the layers and map code to the corresponding layer.

CHAPTER 13

Using Layer Diagrams

How often have we seen a well-thought-out design degrade because we lose control over the way our code evolves? One common reason for that is badly managed dependencies. In Chapter 12 we looked at how Architecture Explorer and the dependency diagram can be used to visualize code dependencies and to help us detect problem areas in our codebase. It would be better, though, if instead of having a tool to help us find issues in our code, we had one that would help us prevent the problems in the first place.

One such tool is the layer diagram. Part of the architecture tools in Visual Studio Ultimate, the layer diagram is used to model the layers in a solution. Once the layers are defined, we can define the dependency structure between the layers and map code to the corresponding layer. With this information in place, the model can be used not only as documentation but also to enforce that modules in the application do not take dependencies outside what is defined in the model.

The architecture archetype pattern we used for our solution unfortunately has a tendency to become a white-board product and not necessarily one followed by developers. Implementing the pattern as a layer diagram will help us maintain dependencies over time and prevent the code from becoming hard to maintain.

Patterns

Before we look at how layer diagrams work, we need a bit of background. Software patterns are commonly used in software development; one kind is the structural pattern for describing a software architecture archetype. We use these patterns to group components into logical layers in order to create a good separation of concerns in the application. Examples of architecture patterns are client-server and service-oriented architecture. Figure 13-1 shows an example of a service-oriented architecture.

Figure 13-1.  Model for a service-oriented architecture

A good start for more information about patterns for Microsoft architecture is the Microsoft Application Architecture Guide, 2nd edition (http://msdn.microsoft.com/en-us/library/ff650706.aspx). The guide covers architecture and design principles as well a good walkthrough of common application archetypes.

Using Layer Diagrams

Ideally, we use the layer diagrams throughout the development process. We begin by creating the diagram as part of the architecture and design phase. When we start development, we add components to the corresponding layer, and from then on we validate the code against the model to ensure we do not break the layering model. In the following sections we look at how to create a diagram, add components to it, and validate the model.

Creating a Layer Diagram

To create a layer diagram, we need to have a modeling project in our solution, and then we can add new layer diagrams to it. The modeling project is often a part of the main solution used for development, and keeping the layer diagrams in the same solution makes it easier to keep the model in sync with the code. But we can have the modeling project in a stand-alone solution and add references to project components as binary references.

The layer diagram is very straightforward to work with. Just add the layers from the architecture to the model. The layers can be nested, for instance, to have a presentation layer containing sublayers UI Components and Presentation Logic Components. We can also use colors to further improve the readability of the model. Figure 13-2 shows how to set the properties for a layer in the diagram.

Figure 13-2.  Layer properties

After creating the layer structure, we can define the relationships in the model. The core idea with layer diagrams is to define which layers depend on which others. For example, if we are building a system using a Domain Driven Design[1], a domain model component would typically be used by business services and perhaps UI components. We can make this explicit by drawing a dependency between the layers. Table 13-1 lists the elements available in the layer diagram.

Table 13-1. Elements available in the layer diagram

Element Name

Purpose

Layer

Defines a layer in the diagram.

Dependency

Defines a one-way dependency in the diagram.

Bidirectional Dependency

Defines a two-way dependency in the diagram.

Comment

Annotation to the model.

Comment Link

Associates a comment with a model element.

Mapping Code to Layers

When the layer structure is in place, we can start adding code to the layers. This is typically something that we continue to do as the project goes on. When new components or namespaces are added, we map them into the correct layer. A common problem is that the architecture diagrams become stale. To prevent this, we should add new components to the solution through the models first. This way we get the documentation up front, and in the case of layer diagrams, we get the benefit of dependency validation as a bonus. If this presents a problem, architecture diagrams should probably not be used in the project.

Next, after the layers are in place, we should define the dependencies between them. We can add dependencies manually by using the Dependency or Bidirectional Dependency association.

Should we come from a situation in which we are creating layer diagrams from an existing codebase, we can also let the tool generate the dependencies by using the Generate Dependencies command from the context menu in the diagram. Generate dependencies will inspect the associated components and create dependencies between the layers accordingly.

To add a component, we use Solution Explorer and drag elements (assemblies, classes, methods, and so on) to the corresponding layer. Figure 13-3 shows a complete example of a layer diagram with associated components.

Figure 13-3.  A complete layer diagram

Viewing Dependencies using Layer Explorer

The dependencies in the model are shown as numbers in each layer. Often you would want to see what is actually mapped to the layer. To do this, select the layer and open Layer Explorer, as seen in Figure 13-4.

Figure 13-4.  View links

Layer Explorer will display all the associated code elements, as shown in Figure 13-5, where we can view and modify the elements.

Figure 13-5.  Layer Explorer

Sharing Models

The layer diagrams can be edited only with the Ultimate version of Visual Studio, but a user of Visual Studio Premium can view them. You also can export images from diagrams to share with non-Visual Studio users—just open the model to export, copy the image using copy and paste, and paste it into another application as an image file.

Layer Validation

When we have the layer model defined, we can use it to validate the implementation. The validation helps us find discrepancies between the model and our code and tells us about differences between the design and the code, which helps us find code that is not organized as designed. The validation also helps us find unwanted dependencies—for example, calls to data-access code directly from the UI, where the design mandates always to use a service layer to abstract the data layer from the UI code.

We can also use the layer diagram to test a change in design, by remodeling to the new pattern we can map the existing code and validate which dependencies would be affected.

There are several ways to perform the validation, both manually and integrated, as part of the build process. Let’s look at how to run the validation and how to analyze the results.

Note  By default, the layer diagram understands only managed code. If you have an application build in C/C++, you can install the Visual Studio 2010 Visualization and Modeling Feature Pack[2] which will enable support for these languages as well.

Manually Validate Layer

We can start a validation manually from the diagram surface. Running the validation will show the result of the validation in the build output window in Visual Studio, as seen in Figure 13-6.

Figure 13-6.  Running layer validation manually

If the code and the model don’t match, we will get errors in the output window (Figure 13-7). You can analyze the errors just as with any other build error in Visual Studio.

Figure 13-7.  Layer validation error in Visual Studio

The error in this case is caused by ExpenseReportController makes a call directly to the Repository.GetExpenseReport, a call from the presentation to the data layer, which is not allowed in the model. To fix this, we could refactor the code so the presentation logic calls a business service that in turn would call the repository method.

Validate in local build

Once the layer model has been added to the solution, it makes sense to run the validation as often as possible. Validating often ensures that a change in code that breaks the design is identified immediately and can be corrected by the developer even before committing the new code to the repository. To make the validation run as part of the build, we need to set the following msbuild property in the modeling project:

<ValidateArchitecture > true</ValidateArchitecture>

Note  Running layer validation as part of a build will slow down the build process. If you want to automate layer validation, we recommend that you integrate the validation as part of your automated build process.

We can also set this property in the model project’s property window, as shown in Figure 13-8.

Figure 13-8.  Layer Explorer

When compiling the project as a developer, we will also get the layer validation as part of the build output (Figure 13-9).

Figure 13-9.  Validating the model as part of a local build

Validate in TFS Build

If we are using TFS Build to do server-side builds, we should of course validate the layer model as well. If we check in a model project, which is defined for local build, TFS Build will pick this up automatically and build using the same settings. If we want to validate the layer model as part of TFS Build, we must provide the /p:ValidateArchitecture = true MSBuild parameter ,as shown in Figure 13-10.

Figure 13-10.  Running layer validation as oart of a TFS Build

Creating an Architecture Template

If we want to reuse the architecture and design work, we can export projects into a new Visual Studio Template. To do this, open the modeling project in Visual Studio and use the Export Template feature under the File menu. This will create a .zip-file that we can distribute and install to use when creating new project.

Note  For Visual Studio 2010, Export Template Wizard will export and package the template as a setup file for easy distribution (http://visualstudiogallery.msdn.microsoft.com/57320b20-34a2-42e4-b97e-e615c71aca24).

Summary

In this chapter, we have looked at how the layer diagram can be used to describe the interation rules for components in different layers in a system and how the model can be used to enforce these rules over the lifetime of the product.

This concludes the architecture section of the book. We have looked at the new Visual Studio tools for requirement elicitation using storyboarding and feedback. We have looked at using UML for use-case modeling as well as architecture modeling. Last, we have explored the architecture exploration tools and the layer diagram—tools that can help us understand the structure of an existing application as well as help us make sure the application continues to be maintainable over time.

Next we will look at development practices and how we can apply ALM to the build phase of the development project.

[1] http://www.domaindrivendesign.org/resources/what_is_ddd

[2] http://msdn.microsoft.com/en-us/library/dd460723.aspx

CHAPTER 14

Metrics and ALM Assessment for Architecture, Analysis and Design

This chapter will cover the metrics and assessment questions for the architecture, analysis, and design areas in ALM. Let us start with the metrics and see what we can get there.

Metrics

There are not many metrics in TFS that we can use for KPI assessment for architecture, but we can use some taken from the development area. Using the code metrics, we can get information on how our architecture and design really are working, including:

- Lines of code. This is an approximate number based on IL code. A high count might indicate that a type or method is doing too much work and should be split up. This might also be a warning that code will be hard to maintain.

- Class coupling. Measures the coupling to unique classes through parameters, local variables, return types, method calls, generic or template instantiations, base classes, interface implementations, fields defined on external types, and attribute decoration. Strive for low coupling; high coupling indicates a design that is difficult to reuse and maintain because of its many interdependencies on other types.

- Depth of inheritance indicates the number of class definitions that extend to the root of the class hierarchy. The deeper the hierarchy, the more difficult it might be to understand where particular methods and fields are defined and/or redefined.

- Cyclomatic complexity is created by calculating the number of different code paths in the flow of the program and shows the complexity of the code. A high complexity makes the maintainability suffer, and it can also be hard to get good code coverage.

- Maintainability index is an index value between 0 and 100 that represents the relative ease of maintaining the code. The higher the better: A rating above 19 is good. Below that, maintainability suffers.

Using the architecture explorer, we can create dependency graphs (see Chapter 12) instead. Running analyzers on these graphs will give us useful information as well:

- Circular Reference, which will identify nodes that have circular dependencies on one another.

- Find hubs, which will identify nodes that are in the top 25 percent of highly-connected nodes

- Unreferenced nodes, which will identify nodes that have no references from any other nodes.

Using these analyzers, we can see if we have loops or circular dependencies so that we can simplify them or maybe break the cycles. We also can see if we have too many dependencies, which could be a sign that they are performing too many functions. In order to make the code easier to maintain, test, change, and maybe reuse, we need to look into whether we need to refactor these code areas to make them more defined. Maybe we can also find code that performs similar functionality and merge with this. If the code has no dependencies at all, we should reconsider keeping it.

Standard Reports

Unfortunately, there are no built-in reports that we can use for architecture analysis and design.

Assessment

To help us evaluate an organization’s maturity in different ALM areas, Microsoft developed its ALM assessments, which we discussed in Chapter 4. Based on the assessment score, we received a maturity level for a specific area that we can use in evaluating in which direction to take our ALM efforts.

Based on the score, we can help the organization reach the maturity level it needs for these areas. Table 14-1 lists questions that can be used as a basis for an ALM Assessment in the architecture, analysis, and design phase. The online assessment has very few questions covering this area, so you might want to use some of your own.

Table 14-1. Assessment questions for architecture, analysis, and design phase

Area

Sample question

Discussion

Architecture Framework

Does architecture definition follow a formal process?

Are there tools for documenting and sharing architecture models?

Using the features of TFS will help customers who don’t have good tooling for this. All architecture tools will help in this area.

Is the architecture well documented?

Using the features of TFS will help customers who don’t have good tooling for this. All architecture tools will help in this area.

Do major architectural decisions follow a defined process?

Analysis & Design

Do all team members have access to the design diagrams?

Using the features of TFS will help the customers who don’t have good tooling for this. All architecture tools will help in this area.

Are the diagrams updated throughout the project lifecycle?

In TFS, there is a flow back and forth between code and diagrams.

Are these diagrams stored and version controlled?

Of course they are, in TFS.

Is forward/backward engineering performed between the code and the diagrams?

In TFS, there is a flow back and forth between code and diagrams.

If using UML, are Sequence Diagrams created?

Can be created easily in TFS.

If using UML, are State Diagrams created?

Can be created easily in TFS.

Database Modeling

Do you use formal modeling methodologies?

Is your database being documented?

Summary

As we have seen in this chapter, we have only a few metrics that we can use for KPI analysis for the architecture and design area.

From an assessment point of view, the online assessment offers questions directed at this area that focus on establishing practices around the design and architecture process.

Let us now see what TFS 2012 offers in the area of developer practices.

PART 4

Building (Developer Practices)

Visual Studio 2012 and Team Foundation Server 2012 are all about development and making system development easier and more effective. Part IV takes us down the developer’s road and lets us have a look at what tools we can use to accomplish this.

We start our stroll by looking at the Team Foundation Server 2012 version control system. Having a good, robust, and reliable version control system is essential to the success of any development project. We also look at how we can manage our source code by looking at some branching strategies.

Test Driven Development (TDD) is an agile practice that has spread far beyond the agile world. In essence, it makes the developer write test code to verify the application logic throughout development. These tests are called Unit Tests – automatic tests that operate very close to the business logic. Their focus is to test individual rules in isolation to certify that the intended result is achieved and maintained over the lifetime of an application. Using these practices we can raise our code quality quite a bit.

As developers we want to create code of the highest possible quality. That often means a slower development pace because we need to go through more checks on our code. But does it really need to be a slow and burdensome process? Chapter 17 examines several powerful tools Visual Studio 2012 offers that we can use to get control of our code quality.

Performance analysis and tuning are often done only after software is released—but we do not have to wait until then. Instead, we should integrate profiling into our daily routines to make sure we always keep an eye on the performance of our applications. Visual Studio 2012 can help us implement good performance analysis both during development and after release.

CHAPTER 15

Version Control

Developers need good tools to work with code efficiently. A good version control system is necessary, and not only for managing versions of files. Many other practices, such as continuous integration and release management, rely on a solid version control solution to work.

Team Foundation Server is built from the ground up to be a scalable yet simple to use version control system. Just like the rest of TFS the version control system the data is stored in a SQL Server database that we access via a layer of web services. Changes are committed to TFS as a transaction so we can rely on SQL Server to guarantee consistency in the version control database. No more fear that some files get committed but not all in case of a network problem, with transactions all get committed or nothing at all.

We like to call TFS a centralized distributed version control system and by that we mean that it has a central repository that everyone connects to and collaborates in. But is also distributed in the sense that we can work efficiently wherever we are located, but we need to be connected to get updates or commit to the repository. With today’s highly connected work environments, it is rare that we need to work without a connection to our TFS server. If we do need to work disconnected, TFS has the capability to work offline and lets us sync up whenever we get back on a connection again. Figure 15-1 shows the logical topology for a version control user. Most users connect directly to the Team Foundation Server, but for scenarios with poor network conditions we can setup a TFS proxy on our local network that acts as a read cache for the remote TFS server. See Chapter 30 for information on how to use a TFS proxy.

Figure 15-1.  TFS version control local architecture

Getting Started with Source Control

Let’s look at how we as developers can perform common tasks when working with the version control system in TFS. The core tasks include setting up a mapping to the repository, getting the code checked-in, writing code, and committing to the repository. We may also need to switch context, for instance, to put work on a new feature aside if we must fix an urgent bug. Wouldn’t it be great if the environment knew how to do that for you? Turns out that TFS does just that, so let’s add that to the list of fundamental practices we need to be aware of.

Using the Team Explorer

The Team Explorer is the primary interface to TFS when working in Visual Studio. With Visual Studio 2012 the developer experience when working with TFS has been reworked to give us a better and more integrated experience. Previously we were often required to open different windows to get a task done, but now we will have most of the context directly available to us inside the Team Explorer. This may result in what feels like a cluttered UI with lots of details but once you get used to the new look it is great to have the relevant information and actions directly in front of you. Figure 15-2 shows the new Team Explorer in Visual Studio 2012.

Figure 15-2.  The Visual Studio 2012 Team Explorer

The Team Explorer is now even more task-oriented and lets us perform the following tasks:

- My Work: This is a new feature in the Premium and Ultimate editions of Visual Studio that brings together many of the source control tasks into the context of the developer. We will look at this later in this chapter. We can also start a Code Review session (a new feature in Visual Studio 2012, which is described in Chapter 17).

- Pending Changes: Takes us to the work in progress view in Team Explorer. We can also open the Source Control Explorer window to work with the repository.

- Work Items: From this node we can create new Work Items, as well as use and define queries to find sets of Work Items. The queries stored under Team Queries are common to all project members, whereas the ones under My Queries are private to the current user.

- Builds: This node holds the information about automated builds run on the server.

- Reports: Lists the available Team Reports, which then can be viewed inside Visual Studio.

- Documents: Lists the documents stored in the TFS project portal (SharePoint site). The documents can be directly accessed and modified from inside the Team Explorer.

- PreEmptive Analytics: This is a third-party extension for analyzing application issues. PreEmptive Analytics is described further in Chapter 18.

- Web Access: This link opens up the TFS Web Access client.

- Settings: Quick access to the team and project settings in TFS.

Workspaces

One of the first things we need to be aware of is the Workspace in TFS. Workspaces are used to map the TFS repository to the local folders used when developing. Often users are unaware of the capabilities of workspaces; they just do a “get latest”, set the workspace folder, and then live happy ever after. But there are some powerful capabilities in TFS workspaces to take advantage of.

The TFS server is aware of your workspaces and their contents, what files you have edited, added, and removed. All changes are stored locally as pending changes and are never introduced to the code in the repository until we commit them.

A workspace can contain multiple project mappings and we can have multiple workspaces for the same user. If we can map several projects to the same workspace why would we need multiple workspaces? The amount of pending changes can quickly become overwhelming if we do not separate the various projects and versions into separate workspaces. This is important because it is the workspace that determines what is visible when dealing with pending changes.

Note  If you have many branches in source control, it is recommended to create one workspace per branch.

Local and Server Workspaces

In Visual Studio 2012 we can create two types of workspaces, local or server workspaces:

- A local workspace is new in TFS 2012. With local workspaces a shadow copy for each file is stored locally which improves performance and the offline experience. For instance local workspaces let you compare files without accessing TFS and undo a checkout also without accessing the TFS server. Local workspace is the default when you create a new workspace.

- A server workspace is more or less the opposite. This is the behavior we have had in versions of TFS prior to 2012. Server workspaces are best suited for workspaces containing a large number of files because the cost for maintaining the local shadow copy increases with the number of files in the workspace.

In general, we recommend using a local workspace because it improves performance and offers better support.

Note  Pending changes for team members who use local workspaces are not visible to other team members.

Configuring a Workspace

Figure 15-3 shows the workspace configuration dialog that can be accessed from the Source Control Explorer or from File→Source Control→Advanced→Workspaces and then add or edit a workspace.

Figure 15-3.  Editing workspace settings

You can name the workspace something suitable (for instance, the Team Project it’s mapped to, plus the name of the branch if appropriate). The most important part is of course the Workspace folders where we map nodes in the repository to folders on our local disk.

Note  Folder mappings can be cloaked in a workspace. Cloaking is the term in TFS for hiding repository folders in the workspace. Cloaking is useful if you do not want to download some parts of the version control tree locally, something that will speed up “get” operations and save local disk space.

Clicking the Advanced button lets you configure all the settings shown in Figure 15-3:

- Location: Specifies whether this is a Local or Server workspace as discussed earlier.

- File Time: This is a new setting in TFS 2012 that allows you to control how files are time stamped when fetched from TFS. Check-in sets the time to when the file was committed to TFS; current sets it to the current time when the file is downloaded locally.

- Permissions: Can be one of the following:

- Private workspace: Can only be used by its owner.

- Public workspace (limited): Can be used by any user but only the owner can check-in or administer the workspace.

- Public workspace: Can be used and administered by any user.

Using the Source Control Explorer

The Source Control Explorer is the main window used to perform tasks against the Team Foundation Server version control system. The Source Control Explorer is opened from the Team Explorer window inside Visual Studio and is not available as a standalone application. There is a standalone version as part of the Team Explorer Everywhere. See Chapter 33 for details on Team Explorer Everywhere. As expected many commands are also available close to where they are used (typically on context menus and in the solution window tree).

The Source Control Explorer is used to perform tasks such as the following:

- Browse team projects and workspaces to identify what is under Team Foundation source control.

- Determine whether items are synchronized, or copied to your local computer’s workspace.

- View all pending changes.

- Undo or check in pending changes.

- Determine whether you have the latest version of an item copied to your local computer’s workspace.

- Get either the latest or specific versions of folders and files.

- Check out folders and files for edit.

- Lock and unlock folders and files.

- Delete, undelete, rename, and move folders and files.

- Resolve source control conflicts.

- Shelve source control items.

- View the history associated with source control folders and files.

- Compare versions of folders or files.

- Branch and merge source control team projects, folders, and files.

- Apply labels to changesets.

- View the properties associated with folders and files.

Figure 15-4 shows an example of the Source Control Explorer.

Figure 15-4.  The Visual Studio 2012 Source Control Explorer

When starting to use a project that is available in TFS, where the files have not yet been downloaded to the local machine, the Source Control Explorer has many grayed items. To resolve this, check whether the workspace mapping is correct (see the section on Workspaces in this chapter for more information) and then perform a Get operation.

Figure 15-4 also shows a situation where a workspace exists but the files have not yet been downloaded. We use the “Latest” column to get an understanding of the current state of files in the workspace. The states can be Not Downloaded, Not latest, or Latest.

Committing Work

Up until now we have looked at setting up the mapping to TFS and some of the fundamentals of working with the version control system. We will now take a look at what happens when we want to commit our work to TFS.

Pending Changes

Pending changes are really nothing new; they are simply the result of an operation in progress on an item in the repository. The difference between other version control systems and Team Foundation Server is the fact that pending changes are more visible because the Team Explorer window deals explicitly with the work in progress in a workspace. Adding and removing items in the repository is never done immediately. Changes are announced and stored locally until committed as part of a changeset when a check-in operation is performed.

Figure 15-5.  Pending Changes helps us understand what gets committed

Figure 15-5 shows how we can add a check-in comment and select files to include in the check-in. It is easy to connect the check-in with a work item from a work item query or just by typing in the work item id. We can also see which files are excluded from check-in. Using the context menu it is easy to compare changes, include/exclude a file from check-in, or to undo the checkout.

When working with local workspaces in Visual Studio 2012 changes in the file system are automatically detected and listed under Detected Changes in the dialog (see Figure 15-6).

Figure 15-6.  Promoting detected changes

Note  By default, multiple checkouts are enabled in Team Foundation Source Control, and this means that we are bound to end up in a conflict resolution scenario from time to time. To avoid this there are ways to lock a file explicitly: use the locking feature when you are performing complex changes to a file. But do not overuse locking because it slows down the speed of development.

Changesets

All operations on Team Foundation Source Control (TFSC) are atomic transactions, meaning that the selected pending changes when submitted to the repository either succeed completely or fail completely. Such a transaction is scoped as a changeset, which is the logical container that stores everything related to a single check-in operation.

Figure 15-7.  A TFS changeset

A changeset contains many different things, such as file and folder revisions, links to related work items, check-in notes, check-in comment, and other information, such as who submitted the change and when. We can use the context menu to compare the changed file against other revisions.

Working with Committed Code

Most of us don’t just write new code but rather spend much of our time analyzing code, improving existing features, or fixing bugs. To do this efficiently we need ways to trace changes, tag versions, and get to files in a previous state. We will look at how TFS supports these scenarios next.

History

History searching is pretty self-explanatory. We can query history on files as well as folders and in both cases lists of changesets affecting the search target are returned. You query the history by using the context menus in the Source Control Explorer or from File→Source Control→View History in Visual Studio. How great it is to always get the context of a change and not just the change in isolation. Without knowing what else has been changed when an item was modified can make it really difficult to trace the source of a problem.

Also note how much more useful a changeset history item is with a good comment; if well-documented, we save time by not having to open the changeset and look at the details.

Figure 15-8.  History in TFS Source Control

Annotate

Annotate is a great tool to help us understand why a section of code looks like it does. Annotate can be seen as a mix of the compare and history tools. We run annotate on a single file and the tool displays a view with the code annotated with the changesets behind each code block. Figure 15-8 shows that Eric and Harry have been working on a service. If we want to understand why the ID of the expense report is set to 1, annotate will show us that Eric added that line of code. If we click on the annotation, we get to the changeset and can perhaps find a work item which would explain the business reason for making the change.

Figure 15-9.  Using Annotate to understand changes

Labels

Labels are used to define a set of files presenting a known state that we may need to reference later. Typically labels are used to tag a version of a product so that we can get the corresponding files, for instance if we want to reproduce a problem. To create a label, select the node in the Source Control Explorer and select Advanced→Apply Label (see Figure 15-10). We can set the label based on different version specifications but latest would be the most common.

Figure 15-10.  Apply Label

We can modify the content of the label (if we have permission to do so) later if we need to, but be aware that labels are not versioned (see Figure 15-11).

Figure 15-11.  Edit Label

Note  Labels do not include information about deleted files, which is natural because the purpose of a label is to define a point in time. One place where this may matter is when we want to merge code from different branches. Using a label can be dangerous because we will not get the deleted files merged to the target branch.

Get Specific

If we need to get source code from a specific point in time we can use the Get Specific dialog (File→Source Control→Advanced→Get Specific Version). We can get files based on different version criteria (see Figure 15-12):

- Changeset

- Date

- Label

- Latest Version

- Workspace Version

Figure 15-12.  Getting a specific version of the codebase

We can also specify how local files should be treated when getting the new version.

Rollback

Sometimes things get checked in to TFS and we want to take them out. For traceability and auditability reasons it is not possible to remove changesets from the database, but we can fix problems by rolling back files to an earlier state. The check-ins are still there but the rollbacked changes are committed as a new changeset.

We have previously been able to do rollback, but only from the command-line. In Visual Studio 2012 we now have the option to rollback from a changeset (see Figure 15-13).

Figure 15-13.  Rollback a changeset

Enforcing Development Practices with Check-in Policies

Many of our practices we agree on when developing are not enforced by a tool, which unfortunately means that many times they are good ideas but not followed and therefore lose their value. Some rules around how we manage code can be enforced by using TFS check-in policies. TFS check-in policies can be seen as triggers that get called when a user wants to commit code to TFS. The code for a check-in policy runs locally and needs to be distributed to the client to work (a missing policy component results in a generic policy failure). We set the policies per Team Project from the Source Control Settings dialog (see Figure 15-14), accessed from the settings in Team Explorer or the Team→Team Project Settings→Source Control menu in Visual Studio.

By default Visual Studio includes the following check-in policies:

- Builds: Requires that the latest TFS build matches the check-in to be successful.

- Changeset Comments: A comment must be written before check-in.

- Code Analysis: Requires code analysis to be run before check-in.

- Work Items: One or more work items need to be associated with every check-in.

Figure 15-14.  Setting up check-in policies

Failing to fulfill a check-in policy results in errors like the one shown in Figure 15-15.

Figure 15-15.  A check-in policy violation

We can get more check-in policies from the TFS Power Tools (http://msdn.microsoft.com/en-us/vstudio/bb980963.aspx) or create our own check-in policies if we need to. See http://msdn.microsoft.com/en-us/library/bb668980.aspx for more information about how to create a custom check-in policy.

Putting Work Aside

Sometime we get in a situation when we need to switch context and set the current work aside for other tasks. Many times this is a time-consuming and error-prone process because we need to do this by hand. With TFS however we have shelving, a great tool to help us manage this problem.

Shelving allows us to bundle our pending changes and store them on the server without checking them in, resulting in what is called a shelveset. Shelving creates a space, the shelveset, on the server that is your own, containing pending changes, comments, and associated work items (in fact very similar to a changeset). Figure 15-16 shows an example of the shelving activity in Team Explorer.

Creating shelvesets is a fast and efficient way to back up any unfinished work at the end of the day or store away changes in progress if a task with greater priority (that involves the same set of files) requires your attention.

Figure 15-16.  Shelving changes

When you shelve, you can choose to move your changes out of your workspace or you can keep your pending changes. Moving your changes to the server is great when you need to stop working on your current changes, make a targeted fix, check in that fix, and then unshelve what you were working on before being interrupted. The reason for keeping your changes in your workspace is really only useful when working in a distributed environment and have the need to share your code or have another person review the changes.

Each developer can have as many shelvesets as needed. Other developers can see what shelvesets exist in the system. However, permissions for shelved changes are enforced according to the permissions for the items involved, so developers only have access to see the changes to the items to which they have permission.

Conflict Resolution

When working in projects where many people work with the same source code concurrently it is likely that there will be conflict situations. The following two behaviors add to make this a quite common scenario:

- TFS has the default behavior where it is allowed to check out a file multiple times to support good concurrency. This is a Team Project setting and can be changed as shown in Figure 15-17 (accessed from the settings in Team Explorer or the Team→Team Project Settings→Source Control menu in Visual Studio).

Figure 15-17.  Check-out settings

- Visual Studio 2012 has the default behavior to not prompt when files are checked out (this is a local user setting and can also be changed, see Figure 15-18, opened from the Tools→Options→Environment menu in Visual Studio).

Figure 15-18.  Visual Studio Source Control Environment Settings

As a result of these behaviors, we may need to handle conflicts during the following operations: getting files from TFS, checking in files to TFS, and when merging files between branches.

Note  Checking out a file does not get the latest version of it. This behavior should be handled by issuing a Get command before starting to work on a new feature.

When a conflict has been detected, TFS automatically tries to resolve the issue and if there are no problems the operation completes without interrupting the user. If the conflict is of a kind that needs the user’s attention, the view in Figure 15-19 is shown and we can resolve the issue in an appropriate way.

Figure 15-19.  Conflict resolution in TFS Source Control

If the user choose to use the merge tool, then a three-way merge UI is presented where it is possible to see the server’s and the user’s versions and then pick parts to manually perform the merge. It is also possible to edit the resulting file manually if that is preferred. See the section “Comparing Changes” later in this chapter.

Get Specific and Delete on Disk

As noted in the section about using the Source Control Explorer, the TFS uses a principle where the actions a user performs against the server are recorded. This recorded status is used to efficiently determine what needs to be done when the user performs new actions, such as a Get operation.

This behavior can cause some problems if files are changed in a way that the TFS server can’t see it, in particular if a file is deleted then a Get operation will not download the file again (because the TFS believes it still exists locally).

To resolve this problem, perform a Get Specific, By latest version from the Source Control Explorer. This downloads the files again and updates the TFS status to reflect the current state.

Checkout Does Not Equal Get Latest

It is important to remember that Team Foundation Server does not perform a get latest when you check out a file. The reason for this is that all changes are part of a changeset and are almost always dependent on some other change in the project checked in as part of the same changeset. This means that if we were to do a get latest on the file we check out we would end up with an inconsistent state when we modify the code. Therefore we are responsible for either doing a get latest on the project or perform a conflict resolution when we check-in the changes we made.

Comparing Changes

Another nice change in Visual Studio 2012 is the new diff and merge tools. The diff tool can be used on files as well as folders. Figure 15-20 shows the diff tool in the side-by-side view where it is easy to see what parts of the file has been changed.

Figure 15-20.  Comparing changes

If the built-in diff tool is not right for you, it is possible to change from the Visual Studio settings (Tools→Options→Source Control→Visual Studio Team Foundation Server and click on Configure User Tools).

Working with Concurrent Changes

When working with concurrent changes we will sooner or later end up with conflicting changes. Due to the nature of TFS there are many ways can get conflicts; when other users have changed the same files as we are working with, when files in one branch are merged to another or even when our own work has been changed in different contexts such as when shelving and resuming work.

In this section we will look at parallel development in TFS by creating branches for isolating work. Later we will merge changes and deal with conflicts that may end up during the merge.

We discuss strategies for parallel development and working with branches in Chapter 28 when we look at release management in general.

First let’s create a new branch; typically we do this from a top-level folder in our source tree. Branching in TFS does not create copies of files until they are changed in the child branch. This is something that makes branch and merge in TFS efficient to use both in terms of storage and a reduced number of items that need to be analyzed when merging.

Figure 15-21.  Creating a new branch

To understand the relationship between branches we can visualize the branch hierarchy (select a branch in Source Control Explorer and from the context menu choose Branching and Merging→View Hierarchy, see Figure 15-22).

Figure 15-22.  Visualizing a branch hierarchy

We can also use the hierarchy to analyze whether a changeset has been merged with the branches it is supposed to. Figure 15-23 shows how we can open a changeset and use Track Action to find where the changeset has been integrated.

Figure 15-23.  Tracking changes visually

Next, if we want to merge a branch or a changeset we use the Merge Wizard (see Figure 15-24). The wizard lets us pick a source and target branch to merge and then choose the kind of merge we want to do. All changes up to a specific version sync all changes up to a specific point in time; selected changesets let us cherry-pick specific changesets one by one.

Figure 15-24.  Using the Merge Wizard to integrate branches

As mentioned before any merge operation may result in conflicts. Figure 15-25 shows the Merge tool which lets us pick source and target changes and combine into the merge result.

Figure 15-25.  Merging changes using the Merge tool

Finally, we can re-run the changeset tracking to get a confirmation that the changeset indeed has been propagated to the intended branch. If you are tracking items over time, there is also a timeline view that shows when the different merges have occurred.

Figure 15-26.  Tracking a merged changeset

Working with My Work

My Work is a new, task centric experience in the new Team Explorer. The intent with this feature is to give us a collective view of the source control tasks from the perspective of work assigned to us. The tools are aimed to give us, as developer, a good flow through the development process. Figure 15-27 shows the My Work view of the Team Explorer.

Figure 15-27.  The Visual Studio 2012 Team Explorer − My Work activity

Note  The My Work panel in Team Explorer is only available in Visual Studio 2012 Premium or Ultimate.

The features in My Work are available as individual tasks, but My Work makes them simpler to use with the intentional flow. So let’s look at an example.

First, pick one of the tasks assigned to us, which is presented under Available Work Items. We can use the context menu or drag the item to the In Progress Work Items & Change pane to indicate that we are starting to work with it. As we continue to do work the changes are now tracked in association to the work item, something that helps us get good traceability in the work we do.

Figure 15-28.  In progress work

When we are finished with our work, we can easily move the change to the next state, which could be to check-in, shelve, or perhaps to request a code review.

If we want to suspend our pending work, use the shelving feature we looked at earlier in this chapter. To make it even easier to change context we can use the Suspend & Shelve action (see Figure 15-29) to put the current work aside. Suspend & Shelve will not only shelve the pending files, but also record the current state of the Visual Studio environment, including open files, window positions, and debugging breakpoint—all which are reset when we later resume work (see Figure 15-30).

Figure 15-29.  Suspend work in progress

Note also that it is possible to either resume to a suspended state or to take suspended work and merge it with the current changes. Being able to merge a shelveset with pending changes is new in Visual Studio 2012.

Figure 15-30.  Resume suspended work

Summary

In this chapter we have looked at the version control system in TFS. The source control system is a scalable centralized repository that integrates nicely with common developer tasks in Visual Studio; from simple activities such as changing code or looking at revisions, as well as more complex processes like branching and merging.

In the next chapter we will look at another important developer practice—unit testing.

CHAPTER 16

Unit Testing

Unit tests are automatic tests that operate very close to actual business logic. The focus here is to test individual rules in isolation to certify that the intended result is achieved and maintained over the lifetime of an application.

This increases the overall quality of the system and reduces the number of bugs created because all changes are guarded by the existing suite of unit tests that certify that changes do not break the existing system. The feedback on unit tests is instant because they trigger and run automatically each time a change is checked in to the version control system.

Unit tests are programs written to run in batches to test other code. Each test typically sends a class a fixed message and verifies it returns the predicted answer. In practical terms this means that you write programs that test the public interfaces of all the classes in your application. This is not requirements testing or acceptance testing. Rather it is testing to ensure the methods you write are doing what you expect them to do.

What Is Test Driven Development?

Test Driven Development (TDD) is one of the core practices in eXtreme Programming, XP. Even if we do not practice XP we can still use this practice as a way to help developers write better code. In TDD, you write the tests before you write the code. When all your tests are working, you know that your code is functioning correctly, and as you add new features, these tests continue to verify that you haven’t broken anything.

Instead of designing a module, then coding it and then testing it, you turn the process around and do the testing first. To put it another way, you don’t write a single line of production code until you have a test that fails.

By working this way you are using a process called Coding by Intention. When practicing Coding by Intention, you write your code top-down instead of bottom-up. Instead of thinking, “I’m going to need this class with these methods”, you just write the code that you expect to be there before the class actually exists.

In traditional software development, tests were thought to verify that an existing bit of code was written correctly. When you do TDD, however, your tests are used to define the behavior of a class before you write it.

Note  With TDD we want to run our tests frequently to get continuous feedback about the code we write. A change in code that breaks assumptions (tests) is something we should become aware of immediately. In Visual Studio 2012 we can configure the environment to run unit tests after build so that as soon as we compile the code we will also run all tests and get feedback on their result.

Principles for Unit Testing

A good unit test should adhere to these basic rules:

- Must be able to run without user interaction. Use assertions to automatically validate the test outcome.

- Must be repeatable. Use initialize and cleanup methods to prepare and cleanup the tests.

- Must be fast. Use mock objects to remove external dependencies during test.

We will explore these rules throughout this chapter.

Working with Mock Objects

One of the biggest challenges you will face when writing units tests is to make sure that each test is only testing one thing. It is very common to have a situation where the method you are testing uses other objects to do its work. If you write a test for this method you end up testing not only the code in that method, but also code in the other classes. This is a problem. Instead we use mock objects to ensure that we are only testing the code we intend to test. A mock object emulates a real class and helps test expectations about how that class is used.

To get this to work, we are reliant on either of the following patterns (see description that follow):

- Separation of Concerns

- Inversion of Control

Each of these patterns is in turn dependent of the fact that our classes are loosely coupled. To achieve this, we need to work with interface-based programming when we want to use mock objects.

The key benefit of working with mock objects is that we get deterministic behavior from the object being mocked and thus we can focus on the code implementing the tests we are running and not the mocked layer.

Separation of Concerns (Object Factories)

Object factories are all about abstracting away the definition of behavior from implementation. This means defining generic interfaces representing a contract with anyone who implements it.

Once we have a separation of behavior and implementation, we can create a factory implementation that can serve up multiple implementations of the same interface. Some examples of this pattern are

- COM: We have an abstract class factory called CoCreateInstance that is driven by metadata in the registry, which means we only need to be aware of our programmatic identifier (i.e. ProgID) and then the rest is dynamic information because we are simply asking for a known interface (IUnknown).

- ADO: The programming model doesn’t change when we change the database implementation. This is because we have abstract definitions about what is possible to do with a database, then we create data sources that define which database we should talk to (metadata once again) and simply provide the identifier for our metadata when we are to connect.

Inversion of Control (Dependency Injection)

Dependency injection is about creating plug-in based implementations, which leads to creating assemblers (i.e. components that put together an instance of an object based on pluggable parts).

There are many ways to implement this, but two common approaches are

- Constructor Injection: Using this approach you need to expose a constructor that takes an interface supplying the pluggable implementation, then it becomes possible to supply different implementations when instantiating an object of this type.

- Setter Injection: In principle the same thing as constructor injection. The difference between the two is that we can create our objects and then post-creation we can inject the implementations needed. It is a more powerful approach because it means we can change the behavior of our object dynamically during the execution of our code.

We will look at how we can work with dependency injection for unit tests later in this chapter.

Visual Studio Unit Tests

Now with the theory in place we can take a look at how to implement unit tests within our application. MSTest is the built-in test framework in Visual Studio that we can use for unit testing. MSTest can also drive other test types, as we will look at in Chapter 22. In Visual Studio 2012, the unit test framework has evolved to support other unit test and is now known as Visual Studio test (vstest).

Elements of a Unit Test

Unit tests are just code with some test specific context added to it. To declare code as unit test code we need to reference the Microsoft.VisualStudio.QualityTools.UnitTestFramework.dll, which imports the unit test framework into our project. Next, we can annotate the code with the following attributes:

- TestClass: Indicates that the given class contains test methods.

- TestMethod: Indicates a method as a test method.

A core Visual Studio unit test would therefore be declared like the following:

[TestClass]

public class ExpenseReportTest

{

[TestMethod]

public void CreateNewExpenseReportGiven2LunchesExpectPendingApproval()

{

}

}

Test Calling Order

Often we need to setup initial state before a test run or clean up state after it is complete. The test attributes shown in Table 16-1 can be applied to a method to control the calling order of a test.

Table 16-1. Test Attributes to Control Test Sequences

Test Attribute

Meaning

AssemblyInitialize

Method should be run before the first test in a given test assembly. The method signature for an assembly initialze method must be

[AssemblyInitialize]

public static void Init(TestContext context)

AssemblyCleanup

Method should be run after the last test in a given test assembly. The method must be declared as static void      with no arguments:

[AssemblyCleanup]

public static void Cleanup()

ClassInitialize

Method that should be run before the first test in a test class.

ClassCleanup

Method that should be run after the last test in a test class.

TestInitialize

Method that should be run before every test method.

TestCleanup

Method that should be run after every test method.

Assertions

A unit test involves more than just running a method under test—we also need to verify that the tested functionality behaves as expected. So the testing framework contains a number of test attributes (see Table 16-2) which we can use to check various types of expectations.

Table 16-2.  Help Classes for Unit Test Assertions

Assertion

Usage

Assert

The core assertion helper class containing a number of different methods to validate true/false conditions. The code below is a basic example of how to use the Assert      class:

[TestMethod]

public void CreateNewExpenseReportGivenProjectExpenseExpectPendingApproval()

{

// . . .

Assert.AreEqual(expected, actual);

}

We can use the Assert.Inconclusive      to indicate that a test method is not yet implemented:

[TestMethod]public void ChangeExpenseReportGivenRejectedExpectFail()

{

// . . .

Assert.Inconclusive("Test not implemented yet!");

}

CollectionAssert

The CollectionAssert      can be used to compare collections and to validate characteristics of the collection.

StringAssert

StringAssert      compares strings in our test.

ExpectedException

The ExpectedException attribute can be used to indicate that a test expects an exception to be thrown or else the test should fail. The example below should fail if an expense report being updated has already been changed by someone else and the code raises the application specific ExpenseReportValidationException     :

[TestMethod, ExpectedException(typeof(ExpenseReportValidationException))]public void ChangeExpenseReportGivenConcurrentUpdateExpectValidationException()

{

// . . .

Assert.Fail("Test should raise an exception!");

}

Creating a Unit Test

To put the preceding in context let’s look at the process of creating a good unit test in Visual Studio 2012.

1. Create Unit Test project in your language of choice. A good convention is to create a unit test project for each project you write unit tests for. Name the unit test with the same namespace as the tested project + .UnitTest as suffix.

2. Add a new Unit Test class to the project. Name the unit test after the class it is testing. Avoid including “Test” in the class name because it is already scoped by UnitTest by following Step 1.

3. Add a TestMethod to the test class. Name the test method using the Feature-Given-Expect format for good readability in the test lists. It is much better to learn why a test is failing from its name instead of having to read (or worse, debug) the test code. For example:

ApproveExpenseReportGivenInvalidProjectExpenseExpectRejected

This method tells us without reading the code that it is a test for the Approve Expense Report story, it tests what happens when approving an expense report with an invalid project code and expects that the expense report is rejected.

4. Add Asserts to verify the test outcome. Try to test only one thing in a unit test, again to make it clear what the intention of the test is. If a test can be created to test multiple data sets, we can use data binding to connect the test with a data source. This is still okay to the rule because the test only tests one thing, but with different conditions.

Figure 16-1 shows a complete Visual Studio Unit Test.

Figure 16-1.  A complete unit test

Context for the Test Run

We may need to control various aspects on how a test is run. To support this we have a context class in the framework, TestContext, which allows us to configure the test environment to affect the way the tests behave.

The TestContext contains static information that can be read from the test when run, including the following attributes:

- Owner. Specifies the person responsible for the test.

- DeploymentItem. Specifies dependent item for the test that gets deployed before the test is run. See the section “Managing Test Dependencies” later in this chapter for more details.

- Description. Documents the purpose of a test.

- Ignore. Marks that a test should not be run.

- Priority. Defines the priority for the test, which can be filtered on to run tests with a particular relevance.

- WorkItem. Specifies a work item associated with the test.

We can also use the TestContext in our tests to access the test execution environment.

- AddResultFile. Adds a file to the test result.

- StartTimer, EndTimer. Cannot be used with Unit Tests, these methods are only intended for creating transactions in load test runs. Use a System.Diagnostics.StopWatch and the TestContext.WriteLine to achieve the same result.

- CurrentTestOutcome. Gives us access to the outcome of the test.

- DataConnection. Gets the current data connection for the test.

- DataRow. Gives us access to the current data row for a data driven test. See the section “Data Driven Automated Tests” later in the chapter.

- DeploymentDirectory. Stores files deployed for the test run, typically a subdirectory of TestRunDirectory.

- FullyQualifiedTestClassName. Gets the full name of the class running the current test.

- Properties. Gives us access to test properties.

- ResultsDirectory. Gets the top-level directory that contains test results and test result directories for the test run, typically a subdirectory of TestRunDirectory.

- TestName. Gets the name of the test.

- TestResultsDirectory. Gets the directory for the test result files.

- TestRunDirectory. Gets the top-level directory for the test run that contains deployed files and result files.

- TestRunResultsDirectory. Gets the top-level directory for the test run result files, typically a subdirectory of ResultsDirectory.

- WriteLine. Writes messages to the test result.

To access the TestContext in a unit test we need to declare a property called TestContext wrapping a TestContext instance:

private TestContext testContextInstance;

public TestContext TestContext

{

get { return testContextInstance; }

set { testContextInstance = value; }

}

The test framework automatically sets the property for us. We can then use the property in our test code to interact with the TestContext. This example writes to the test output:

TestContext.WriteLine ("Expense Report updated to: {0}.", updatedExpenseReport.ToString());

The result is written to the output for the test run as shown in Figure 16-2.

Figure 16-2.  Using TestContext to write to the test result output

Test Explorer

In Visual Studio 2012 unit tests are managed from the new Test Explorer window. The Test Explorer runs tests from any test framework as long as they provide a test adapter for the Visual Studio framework.

Running Tests

When we build our solution and its test project, the unit tests are displayed in the Test Explorer (see Figure 16-3). If the Test Explorer is not visible we can open it from the Test→Windows→Test Explorer menu.

The tests are grouped by four categories; Failed Tests, Skipped Tests, Passed Tests, and Not Run Tests. We can run tests from the menus in Test Explorer or by selecting one or more tests in the list. We can also start a debug session from the tests the same way.

The Test Explorer shows the status of the last run and only for the tests that ran. The color bar at the top of the window is shown in the color of the last test run and gives direct feedback to the test status in a nice way.

Figure 16-3.  The Test Explorer

In true test driven spirit we can also select to run all tests after we build in Visual Studio. We can configure this behavior by clicking the Run Tests After Build button in Test Explorer or from the Test→Test Settings→Run Tests After Build menu (see Figure 16-4).

Figure 16-4.  Configure to run test after build

The test status is shown in color next to the unit test together with the call time for each test. We can also select a test and view the details from the test run (including errors from a failing test) as shown in Figure 16-5.

Figure 16-5. Unit test result details

If output from the test run is available, an Output link is shown that takes us to the output view for the test run (see Figure 16-6).

Figure 16-6.  Unit Test result output

Working with Code Coverage

Code coverage is a concept where components are instrumented with logging points. When the components are executed (typically through automated tests), the execution paths are logged. The logs are then used to create statistics over how the components are used.

Code coverage of 100% does not mean that the quality is perfect, but a low number tells us that we have insufficient testing. Code coverage somewhere in the range of 85% is a good number to aim for.

Note  Code coverage requires Visual Studio 2012 Premium or Ultimate.

Adding Code Coverage to a Project

In Visual Studio 2012 unit tests can create code coverage data for all assemblies without any explicit configuration. All solution binaries that get loaded during unit test runs are analyzed by default.

We can customize the way code coverage works in our project, typically by excluding some assemblies and including others. The way to do this is different in Visual Studio 2012 compared to Visual Studio 2010. Visual Studio 2012 supports different unit test frameworks and the .testsettings file can only handle MSTest tests.

Using Runsettings to Configure Code Coverage

Visual Studio 2012 customizes how code coverage is collected by adding a .runsettings XML file to the solution and setting it as the default setting using the Test→Test Settings→Select Test Settings File.

At this point there is no designer for the .runsettings file so it needs to be edited as XML in Visual Studio. Below is a fragment of a .runsettings file that can be used to configure code coverage. For a complete description on how to create a .runsettings file see http://blogs.msdn.com/b/sudhakan/archive/2012/05/11/customizing-code-coverage-in-visual-studio-11.aspx:

<?xml version = "1.0" encoding = "utf-8"?>

<RunSettings>

. . .

<Configuration>

<CodeCoverage>

<ModulePaths>

<Include>

<ModulePath > .*\\UnitTestProject1\.dll</ModulePath>

</Include>

<Exclude>

<ModulePath > .*CPPUnitTestFramework.*</ModulePath>

</Exclude>

</ModulePaths>

<UseVerifiableInstrumentation > True</UseVerifiableInstrumentation>

<AllowLowIntegrityProcesses > True</AllowLowIntegrityProcesses>

<CollectFromChildProcesses > True</CollectFromChildProcesses>

<CollectAspDotNet > False</CollectAspDotNet>

. . .

</RunSettings>

Using Testsettings to Configure Code Coverage

If we share the projects between Visual Studio 2010 and 2012 and use MSTest as the testing framework, we can still use the Code Coverage configurations in a testsettings file. Figure 16-7 shows how to configure Code Coverage from the Data and Diagnostics section of the Test Settings file.

Figure 16-7.  Enabling Code Coverage in a testsettings configuration

From the Code Coverage dialog we can then select configure and choose the assemblies to analyze code coverage for (see Figure 16-8).

Figure 16-8.  Configuring Code Coverage for selected assemblies

Analyzing Code Coverage Results

Regardless of how we gather the code coverage data we can analyze the result using the Code Coverage Results window in Visual Studio (see Figure 16-9). The result view shows us the assemblies analyzed and we can drill down into namespaces, classes, and methods.

Figure 16-9.  Code coverage results in Visual Studio 2012

The coverage data is shown by default as % Blocks measured. If you prefer % Lines covered, this can be shown by adding the additional columns to the result view (see Figure 16-10).

Figure 16-10.  Code coverage configuration

Finally if we want to understand why a section of code has a particular coverage number, the best way is to show the coverage data inline with the code. The Show Code Coverage Coloring button enables coloring so when we click on a code item, the coverage information is shown in different shades of blue and red (see Figure 16-11). This is great feedback in many ways; it show us which paths we missed to test, something that can be a result of missing tests as well as unused code.

Figure 16-11.  Code coverage result shown in source code

Configuring Code Coverage for a Build Definition

Code coverage is an important metric to track over time if you are interested in how the ratio of automated testing is compared to the amount of code being written. A good way to get the code coverage measured regularly is to include automated tests and code coverage in our automated build process.

We will look more at build and how to configure automated builds in Chapter 26, if you are eager to take a look feel free to jump over to that section right away.

Data-Driven Automated Tests

Visual Studio unit tests contain a concept of data-driven tests. This is not to be confused with database testing; instead data-driven tests are used to map parameterized values in an external resource to a unit test (also known as data pools). There are several uses of data-driven tests; the most common is to use data pools for functions that will require many test permutations.

MSTest supports a number of data sources, such as CSV files, XML files, Excel files, and database tables that we will look at later in this chapter.

Some tests can be run multiple times with different data sets. Instead of having to implement a number of different tests permutations, we can instead bind the test to a data source and then access the current data row from the unit test.

We connect a unit test with a data source by adding the DataSource attribute to the test method. The DataSource attribute lets us specify the following arguments:

- ProviderName. The provider for the data source to use. See Table 16-3 for examples of provider names.
Table 16-3. Data Sources for Data-Driven Tests

Data Source

Data Source Attribute

CSV

DataSource("Microsoft.VisualStudio.TestTools.DataSource.CSV", "|DataDirectory|\\data.csv", "data#csv", DataAccessMethod.Sequential)

Excel

DataSource("System.Data.Odbc", "Dsn = Excel Files;Driver = {Microsoft Excel Driver (*.xls)};dbq = |DataDirectory|\\Data.xls;defaultdir = .;driverid = 790;maxbuffersize =2048;pagetimeout = 5;readonly = true", "Sheet1$", DataAccessMethod.Sequential)

Test Case

DataSource("Microsoft.VisualStudio.TestTools.DataSource.TestCase", "http://tfs:8080/tfs/DefaultCollection;Agile", "30", DataAccessMethod.Sequential)      Note: This data source will create a strong dependency to TFS and may not be suitable for pure unit tests. But for other automated test it’s can be a good alternative since it allows us to edit the test data from the test case parameters.

XML

[DataSource("Microsoft.VisualStudio.TestTools.DataSource.XML", "|DataDirectory|\\data.xml", "Iterations", DataAccessMethod.Sequential)

SQL Express

[DataSource("System.Data.SqlClient", "Data Source = .\\sqlexpress;Initial Catalog = tempdb;Integrated Security = True", "Data", DataAccessMethod.Sequential)

- ConnectionString. The connection string to the data source (database, data file, etc.).

- TableName. Name of data table in the data source.

- DataAccessMethod. Specifies how the test data is drawn from the data pool. Can be either Sequential or Random.

To access the data row, we use the TestContext class. We get called by the test framework one time for each set of data in the DataSource so we only need to think about the current row when accessing data from the store. The code sample in Figure 16-12 shows a complete example of a data-driven unit test. The dependency to the data source should be handled using a DeploymentItem attribute so that the unit test can be run anywhere as long as the dependent item is deployed when the test is run. See “Managing Test Dependencies” in the next section.

Figure 16-12.  Data-driven unit test example

Working with External Files

In theory a unit-test should have no external dependencies. However, in some situations it may be very effective to use external files as input for tests; one such scenario is to capture actual messages processed in the system and in a test replay those messages. The files in this case would replace the need for setting up the test scene in the test implementation.

If external files are used in tests, they should preferably be deployed as embedded resources in the test assembly to simplify deployment so that the tests can run automatically anywhere.

If external files are used to setup the context for tests, it is important to keep versioning in mind. When test data is captured in the files, they represent a snapshot of how data is exchanged in the system. If the structure of the data changes, the test data will be invalid. Sometimes the changes are very subtle, like for instance if two string elements change order. It may be difficult to trace the source of these errors and it may also be time-consuming to maintain the changes if a large number of files are used.

Managing Test Dependencies

If a test has dependencies to files that do not come from the build process, we can instruct the framework to copy the dependencies when the test is run. By default the test assemblies and their dependent assemblies are created to the deployment folder for the test.

If we run tests in Visual Studio, a folder called TestResults is created directly under the solution folder. The TestResult folder contains subfolders for each test run. In the test run folders, two additional folders are created:

- Out: The out folder is the actual deployment folder and all files required by the test are copied here before the test is run.

- In: Certain test result files, such as code coverage files that are created as part of the test run, get copied to the in folder.

One option to deploy files to the test is to use a .testsettings file. Add a .testsettings file to the solution and open the designer. Switch to the Deployment tab and add files and folders as needed by the test (see Figure 16-13).

Figure 16-13.  Adding deployment items using a .testsettings file

We can also use the DeploymentItem attribute on the test method level to define which files should be copied out before the test is run:

[TestMethod, DeploymentItem("ExpenseReportsPendingApproval.csv")]

public void ApproveExpenseReportDataDriven()

{

// ...

}

Working with Other Test Frameworks

A new and interesting feature of the Visual Studio unit test framework is that it is now possible to mix unit test frameworks within the native tools in Visual Studio. We can even have multiple test frameworks in the same unit test project!

The unit test framework needs to implement a Visual Studio unit test adapter to become visible to the testing tools. If you are interested in authoring your own adapter, you can find a sample implementation here: http://blogs.msdn.com/b/bhuvaneshwari/archive/2012/03/13/authoring-a-new-visual-studio-test-adapter.aspx.

It is easiest to add more third-party frameworks from inside Visual Studio by using the Extension Manager (Tools→Extensions and Updates). Select the Online category and the Visual Studio Gallery and search for Testing (see Figure 16-14) and you will find many implementations, such as nUnit and xUnit adapters.

Figure 16-14.  Adding third party Unit Test frameworks to Visual Studio using the Extension Manager

Implementing Mock Objects

We can implement a mock implementation just like any other piece of code and include it as part of our unit test code base; however, the mock code most of the time just mimics the interface it replaces and contains very simple code. For example, a mock implementation of a fetch operation only needs to implement the method signature to be usable as a stand-in for the real object. If we want to be able to control what the mock implementation returns we can add a backing list field that we can control from the unit test and prepare the data to suit the purpose of the test. The following code would do fine for a simple mock implementation of a ListCustomers method:

public List < Customer > Customers = new List < Customer > ();

public List < Customer > ListCustomers()

{

return this.Customers;

}

We have several common options for how to create fakes for our unit tests:

- Hand crafted mock objects: This is the simplest approach as described previously. It should be noted that we are required to implement the complete interface to mock and not only the operations we want to test. This is of course time consuming and leads to more code to maintain.

- Mock frameworks: There are a number of good frameworks that help us creating fakes by dynamically substituting a type that we want to replace. Two popular open-source frameworks are moq (http://code.google.com/p/moq) and Rhino Mocks (http://hibernatingrhinos.com/open-source/rhino-mocks) which both offer dynamic mocking and fluent APIs.

- Visual Studio Fakes: Visual Studio 2012 comes with support for creating stubs to fake out an interface or abstract class.

Summary

In this chapter we have looked at unit testing as a development practice. Whether or not you choose to follow a test driven approach, a unit test should be the foundation of any test automation effort.

We may still benefit a lot from writing tests based on the unit test framework that is not strict unit testing but instead tests the code under the UI or against the APIs in the system.

See Chapter 20 for a discussion on agile testing and how a unit test framework can be used to implement different types of automated tests.

CHAPTER 17

Code Quality

As developers we want to create code with as high quality as possible. But high quality often means lower development pace because we need to go through more checks on our code. But does it need to be a slow and burdensome process?

In this chapter we will examine several powerful tools we can use to get in control of our code quality. The static code analysis framework checks our code for code-level correctness, over and above the syntactical checks the compiler does. We can also use Code Metrics to get feedback on the quality of our code and help to identify areas that need attention, often this is driven from a maintainability perspective. A common problem with maintainability is code duplication. To help us mitigate that problem we can use the new clone detection tool in Visual Studio that will analyze code for similarities which we then can refactor to get a better structure. Finally, the traditional code review workflow has been given tool support in Visual Studio and TFS; with Visual Studio 2012 we can now have a tracked conversation about our code without leaving the development environment.

Code Analysis

First we will take a look at how to setup and manage static code analysis in Visual Studio. The code analysis tool is an evolution of the community tool FxCop, which in essence is a framework for automating code analysis. Static code analysis works by parsing the source code and matching it against a defined set of coding conventions or code analysis rules. A simple example is a naming rule that checks that identifiers are cased correctly. Figure 17-1 shows how a violation of this rule is shown in Visual Studio 2012.

Figure 17-1.  Code analysis warning for incorrect identifier casing

The code analysis tool is intended to be used to complement manual code analysis, not replace it. The ambition should be to automate common static code analysis and use manual code review as group activities to share knowledge inside the team, to improve maintainability of the source code and so on.

The static code analysis tools in Visual Studio can be used with managed code as well as C/C++ code and is configured on a project per project basis.

It is recommended to create a baseline set of rules and share them between projects. This enforces usage of code analysis and makes sure everyone uses the same set of rules. To help us get started, Visual Studio comes with a set of predefined rule sets as listed in Table 17-1.

Table 17-1. Standard Code Analysis Rule Sets in Visual Studio 2012

Rule Set

Purpose

All

This rule set contains all rules. Running this rule set may result in a large number of warnings being reported. Use this rule set to get a comprehensive picture of all issues in your code. This can help you decide which of the more focused rule sets are most appropriate to run for your projects.

Microsoft Basic Correctness

These rules focus on logic errors and common mistakes made in the usage of framework APIs. Include this rule set to expand on the list of warnings reported by the minimum recommended rules.

Microsoft Basic Design Guidelines

These rules focus on enforcing best practices to make your code easy to understand and use. Include this rule set if your project includes library code or if you want to enforce best practices for easily maintainable code.

Microsoft Extended Correctness

These rules expand on the basic correctness rules to maximize the logic and framework usage errors that are reported. Extra emphasis is placed on specific scenarios such as COM interop and mobile applications. Consider including this rule set if one of these scenarios applies to your project or to find additional problems in your project.

Microsoft Extended Design Guidelines

These rules expand on the basic design guideline rules to maximize the usability and maintainability issues that are reported. Extra emphasis is placed on naming guidelines. Consider including this rule set if your project includes library code or if you want to enforce the highest standards for writing maintainable code.

Microsoft Globalization

These rules focus on problems that prevent data in your application from displaying correctly when used in different languages, locales, and cultures. Include this rule set if your application is localized or globalized.

Microsoft Managed Minimum

These rules focus on the most critical problems in your code for which Code Analysis is the most accurate. These rules are small in number and they are intended only to run in limited Visual Studio editions. Use MinimumRecommendedRules.ruleset with other Visual Studio editions.

Microsoft Managed Recommended

These rules focus on the most critical problems in your code, including potential security holes, application crashes, and other important logic and design errors. You should include this rule set in any custom rule set you create for your projects.

Microsoft Mixed (C++/CLR) Minimum

These rules focus on the most critical problems in your C++ projects that support the Common Language Runtime, including potential security holes and application crashes. You should include this rule set in any custom rule set you create for your C++ projects that support the Common Language Runtime.

Microsoft Mixed (C++/CLR) Recommended

These rules focus on the most common and critical problems in your C++ projects that support the Common Language Runtime, including potential security holes, application crashes, and other important logic and design errors. You should include this rule set in any custom rule set you create for your C++ projects that support the Common Language Runtime. This ruleset is designed to be configured with the Visual Studio Professional edition and higher.

Native Minimum Rules rule set

These rules focus on the most critical problems in your native code, including potential security holes and application crashes. You should include this rule set in any custom rule set you create for your native projects.

Native Recommended Rules rule set

These rules focus on the most critical and common problems in your native code, including potential security holes and application crashes. You should include this rule set in any custom rule set you create for your native projects. This ruleset is designed to work with Visual Studio Professional edition and higher.

Microsoft Security Rules

This rule set contains all Microsoft security rules. Include this rule set to maximize the number of potential security issues that are reported.

After we decide which rules apply to our code, we need to configure each project in the solution with the proper settings. We can assign the rule set on each project’s properties or on the solution level (see Figure 17-2).

Figure 17-2.  Configuring code analysis

Static code analysis is executed similar to a standard compilation of the code. When code analysis is enabled on build, the project will be analyzed after it has been compiled. To enable code analysis as part of the build process we set the Enable Code Analysis on Build flag on each project. We can always run the code analysis manually from the Analyze menu in Visual Studio (see Figure 17-3).

Figure 17-3.  Enable Code Analysis on Build

Note  Running Code Analysis as part of the build slows down the development experience. As a compromise we recommend manually running code analysis while developing and then integrating it with the check-in and server build processes to ensure code analysis is run.

After the code analysis has been run any errors found are shown in the Code Analysis window (see Figure 17-4). From the list you can select the item, which shows the documentation of the analysis rule, examples of why it happens, and suggestions of what to do to resolve the issue.

Figure 17-4.  Working with code analysis from the Code Analysis window

If you have rules that cannot be fulfilled, then in rare occasions you can choose to make suppression to the rules. In the Code Analysis window you will see the rule breaks as either global (no file and line reference) or local conditions. You can suppress the rule by just clicking on the Actions link on the item to suppress and select “Suppress Message.”

For global suppressions a separate file “GlobalSupressions.cs” is added to the project with the suppression to the rule added as an assembly level attribute. For local suppressions local attributes are added inline with the code as shown in Figure 17-5.

Figure 17-5.  Supressing a code analysis error

Creating Custom Rule Sets

We have seen how we can use the predefined rule sets to quickly get started with code analysis. Sometimes we want to exclude certain rules or use rules from multiple standard rule sets. If this is the case then we can create a custom rule set, save it as a .ruleset file, and reference it from the projects it applies.

To create a new rule set from scratch we create a new Code Analysis Rule Set from the General file templates in Visual Studio (see Figure 17-6).

Figure 17-6.  Adding a custom code analysis rule set

Next we select the rule groups that apply and can drill down and customize which rules should be applied as well as whether the rule should be treated as a warning or an error. Figure 17-7 shows an example of a custom code analysis rule set.

Figure 17-7.  Configuring a custom code analysis rule set

Having defined the rule set, we can now apply it to the projects as Figure 17-8 shows.

Figure 17-8.  Using a custom code analysis rule set

We can also create a custom rule set from an existing set by opening the code analysis properties in a project and saving the resulting changes as a new .ruleset file.

It is also possible to customize the code analysis dictionary used to check identifiers in the code. If we have terms in our code that generate analysis errors or warnings, we can customize the dictionary instead of suppressing the validation.

Integrating Code Analysis with TFS Builds

Running code analysis as part of the local build can slow down the development experience. As a compromise we can use TFS server-side builds to validate the code. Enabling this in a default TFS build workflow is as easy as setting the Perform Code Analysis property as shown in Figure 17-9. The following are our choices:

- Never: Never run code analysis as part of the build.

- AsConfigured: Run code analysis according to each project’s setting.

- Always: Always run code analysis.

See Chapter 26 for more information on how to setup TFS builds.

Figure 17-9.  Integrating code analysis with TFS build

Code Analysis Check-in Policy

A complement to running the code analysis as part of the build is to ensure we have run the code analysis before checking in the code (see Figure 17-10). If we want to do this, then we can just add the code analysis check-in policy to our TFS project. See Chapter 15 for more information on how to work with check-in policies.

Figure 17-10.  Configuring the code analysis check-in policy

Code Metrics

As our software evolves it is important to understand the complexity of the code. A common situation is that the older the code is, the harder it is to maintain. Often the reason for this is that it is easy to just add new features to existing code rather than refactor and redesign the solution. To avoid degraded code quality we can analyze the code and get a report on the code metrics. The code metrics we get from Visual Studio are the following:

- Lines of code is an approximate number based on IL code. A high count might indicate that a type or method is doing too much work and should be split up. This might also be a warning that code will be hard to maintain.

- Class coupling: measures the coupling to unique classes through parameters, local variables, return types, method calls, generic or template instantiations, base classes, interface implementations, fields defined on external types, and attribute decoration. Low coupling is better to strive for because high coupling indicates a design that is difficult to reuse and maintain because of its many interdependencies on other types.

- Depth of inheritance indicates the number of class definitions that extend to the root of the class hierarchy. The deeper the hierarchy the more difficult it might be to understand where particular methods and fields are defined or/and redefined.

- Cyclomatic complexity is created by calculating the number of different code paths in the flow of the program and shows the complexity of the code. A high complexity makes the maintainability suffer and it can also be hard to get good code coverage.

- Maintainability index is an index value between 0 and 100 that represents the relative ease of maintaining the code. A rating above 20 is considered good, below that maintainability suffers. It is however a very generous limit—a maintainability of 20 might seem okay, but in reality it is pretty horrible.

Note  An anonymous method is a method we can declare as a method without giving it a name and is typically used as delegate parameter. The metrics for the anonymous method is associated with the code declaring the method.

Generated code is generally excluded from code metrics. You can customize this using .NET attributes to get a more realistic coverage number. The following post shows some good examples of how to do this: http://geekswithblogs.net/terje/archive/2008/11/10/hiding-generated-code-from-code-analysis-metrics-and-test-coverage.aspx.

To calculate the code metrics for our code we can do so on all or individual projects by selecting the solution or project. Next, we run the analysis from the Analyze menu in Visual Studio and then Calculate Code Metrics for Solution or project. When the analysis is complete the result is shown in the Code Metrics Results window (see Figure 17-11). We can analyze the result by filtering for behaviors we are interested in as well as do a drill-down into the code we want to understand better.

Figure 17-11.  Code metrics summary

We can also export the result to Excel to do further analysis or just to save the result from the run as a way of documenting our code quality. If we find issues we want to address later, we can also create a work item from the result view.

Note  Currently code metrics is available in Visual Studio or as a command-line tool (http://www.microsoft.com/en-us/download/details.aspx?id=9422). If you want to include the code metrics in automated builds (which is a great idea by the way) you need to customize the build process and wrap the command-line tool. Chapter 26 covers build process customization. It also introduces the TFS Build Extensions, which is a community project for custom build activities, including one for code metrics.

Code Clones

Have you ever worked on a piece of code and thought you’ve seen it before? Wished you had a tool that would show you where the same or similar code exists in your code base? Copy-paste is probably the most common (anti) design pattern used by programmers and though quick to use to solve a problem, it will likely degrade your product in the long run.

In Visual Studio 2012 we now have a code analysis tool for detecting code with similar structure. We can search for code clones in a couple of ways. We can choose to analyze the entire solution for clones by selecting Analyze Solution for code clones from the Analyze menu.

Note  Only code fragments >10 statements will be analyzed when running code clone detection on the entire solution.

We can also select a section of code and have the tool find matching code. Select a code snippet you want to analyze for clones and choose Find Matching Clones in Solution from the context menu (see Figure 17-12).

Figure 17-12.  Detecting a cloned code section from selected code

The code clone analysis finds direct copies of code and also fragments that are similar but may differ in naming of variables or parameters.

The results are shown in the Code Clone Analysis Results window where we can analyze the detected clones. If we want to compare the differences we can select the original and the clone to compare. In the example in Figure 17-13 you can see how a Save method has been copied and the entity to save has been changed. The code structure is semantically the same, but as you can see the details are too different to find with a simple “find-in-files” match.

Figure 17-13.  Analyzing code clones

It is recommended to run the code clone analysis whenever we change existing code. When we modify code we can first run the analysis to learn whether the code we are about to change exists in other areas as well. We can also use the result from the analysis to consider whether we should refactor the code to make it more maintainable.

Code Review

Earlier we looked at how we can use static code analysis as a tool to help us create maintainable code that adheres to the development practices we have agreed on. But a tool is only a tool so if we want our software to be not only correct on the code level but also have a good design and structure we should perform code reviews as well.

There are many reasons why we would want to do a code review, for instance

- We are done with a feature and we want someone to review it before checking in.

- We find a changeset we do not understand and want someone to take a look at it.

- We notice a block of code we think is incorrect.

- We come across a brilliant solution in the code that we want to make others aware of.

Regardless of the reason, it should be easy to get the review done. A common reason to why code reviews get neglected is simply because we do not have good tool support for the process and therefore no common practice for how the review should be performed. In Visual Studio 2012 Code Review is built in to the platform which makes it very easy to use.

Note  The Code Review feature is only available in Visual Studio Premium and Visual Studio Ultimate.

A common way to initiate a code review in Visual Studio is to use the Team Explorer and request a review from the pending changes in My Work, for instance we want to run a code review before we checkin the code for a completed work item (see Figure 17-14).

Figure 17-14.  Initiating a code review session

We may also start a code review session from the source code history if we find a checkin that we want someone to take a look at as shown in Figure 17-15.

Figure 17-15.  Initiating a code review from source control history

Next the reviewers receive the code review request. As a reviewer, we accept or decline the code review from the Incoming Review queue in Team Explorer (see Figure 17-16).

Figure 17-16.  Code review incoming queue

Accepting to review takes us to the Code Review section where the details for the review session are outlined (see Figure 17-17). We can comment on the entire session as well as per file. We can even click on a file that opens up the file in the compare tool in Visual Studio. Here it is easy to review the changes made compared to the previous version. It is also possible to add review comments on code blocks by selecting a code section and add a comment.

Figure 17-17.  Responding to a code review

After the reviewers are done with the code review the person requesting the review can see that it has been completed (see Figure 17-18).

Figure 17-18.  Code review completed

We can then open up the review request to look at the details. If there are outstanding questions we can Reply and enter a comment on either the review session as a whole or on each file and code block in the review session. We can also use the checkboxes to work through the review items and mark the comment as taken care of. Figure 17-19 shows the Team Explorer in Code Review mode.

Figure 17-19.  Reviewing a code review session

Summary

In this chapter we have looked at how we can use Visual Studio to improve code quality in a number of different ways. We can use code analysis and code metrics to understand how well our code is written. We can use code clone detection to find redundant code that we can refactor to make the code base more maintainable. Finally, we looked at how the new capabilities for code review workstreams can be used to automate the manual code review process.

Next we will take the code quality concept even further and look at how we can work with quality issues. First we need to know if and where we have a problem. The profiler in Visual Studio helps us understand how our system behaves when it is run and if we have any performance or memory issues we need to deal with. Finding the problems early during development is of course ideal, but often the problems are found during testing or in production. Troubleshooting in these environments are by nature more complex so we need good tools to help us work efficiently with the issues found. To help us out we can use IntelliTrace and Preemptive Analytics.

CHAPTER 18

Performance and Profiling

Performance analysis and tuning are often done after release—but we really do not have to wait until then. Instead we should integrate the profiling into our daily routines to make sure we always keep an eye on the performance of our applications.

In software engineering, profiling is a form of dynamic application analysis that measures, for example, the memory usage, the usage of particular instructions, or frequency and duration of function calls. The most common use of profiling information is to help developers optimize their applications.

Profiling is achieved by instrumenting either the application source code or the binary executable. TFS and Visual Studio include profiling tools that will help us with this task.

Profiling Overview

There are many ways a profiling tool can gather and output data. The profiling tools can collect different kinds of information like performance counters, hardware interrupts, code instrumentation, and operating system information just to mention a few. Let’s take a brief look at some ways the profiling tools can collect data:

- Event-based profilers: Some programming languages offer an event-based profiler. In Java we have the JVMTI (JVM Tools Interface) API, formerly JVMPI (JVM Profiling Interface), which provides hooks to profilers, for trapping events like calls, class-load, unload, thread enter leave. In .NET we can attach a profiling agent as a COM server to the CLR using Profiling API. Like Java, the runtime then provides various callbacks into the agent, for trapping events like method JIT / enter / leave, object creation, etc.

- Statistical profilers: Some profilers operate by sampling where the profiler probes the target application’s program counter at regular intervals using operating system interrupts. Sampling profiles are typically less numerically accurate and specific, but allow the target program to run at near full speed, which helps the profiler detect issues that would be hard to catch otherwise. Often a drawback with some profiling methods is that the application takes a performance hit during profiling. The resulting data from statistical profilers are not exact, but a statistical approximation.

- Instrumenting profilers: Some profilers instrument the target program with additional instructions to collect the required information. This profiling method can affect the performance of the program, which could cause inaccurate results. Instrumenting will always have some impact on the program execution, typically always slowing it. However, instrumentation can be very specific and be carefully controlled to have a minimal impact. The impact on a particular program depends on the placement of instrumentation points and the mechanism used to capture the trace.

There are two major outputs a profiler might produce:

- A statistical summary, or profile as it is also called, of the events the profiler has observed. Summary profile information is often shown annotated against the source code statements where the events occur, so the size of measurement data is linear to the code size of the program.

- A trace, which is a stream of recorded events. For sequential programs, a summary profile is usually sufficient, but performance problems in parallel programs (waiting for messages or synchronization issues) often depend on the time relationship of events, thus requiring a full trace to get an understanding of what is happening.

Let’s now take a look at what Visual Studio has to offer when it comes to profiling.

Profiling In Visual Studio 2012

Visual Studio 2012 offers some great tools when it comes to profiling. These tools let us measure, evaluate, and find performance-related issues in our code. The tools are fully integrated into the Visual Studio IDE to provide a seamless and approachable user experience; however the profiling tools are also available from the command line if we would like that approach.

The profiling tools in Visual Studio offer five ways to collect and analyze data:

- Sampling: This is the recommended way to start exploring application performance. The sampling method is nonintrusive and has little impact on the execution of the application being monitored. Sampling profiling interrupts the processor at given intervals and collects the executing call stack. Based on these statistics, call counts are calculated so we can analyze execution patterns.

- Instrumentation: When using the instrumentation the profiled code gets injected so exact call times can be recorded. In the analysis report we use the following values to understand the application performance:

- Elapsed Inclusive: Total time spent executing the method (including calls to other methods).

- Application Inclusive: Same as Elapsed Inclusive but with calls to the operating system excluded.

- Elapsed Exclusive: Time spent executing code in the method.

- Application Exclusive: Same as Elapsed Exclusive but with calls to the operating system excluded.

- Concurrency: This profiling mode collects data about multithreaded application. The analysis reports resource contention and there is also a visualization of how the application executed which we can use to trace bottlenecks in the system.

- .NET Memory: The .NET memory profiling collects information when .NET objects are allocated and returned. Memory profiling can be used together with sampling or concurrency proofing.

- Tier Interaction: This mode collects information about ADO.NET calls made to a SQL Server database. The analysis data can help us understand database performance from an application perspective. Tier Interaction can be used with any of the other profiling modes.

If we do not want to manually set up our profiling, we can use the Performance Wizard (see Figure 18-1) and start with the default settings. By analyzing these results we can decide whether we need to change any settings or setup some manual profiling.

Figure 18-1.  The Performance Wizard

We run the Wizard from the Analyze→Launch Performance Wizard in Visual Studio.

The output of the Performance Wizard is extensive. Performance warnings that alert us to common coding issues in the VS Errors window, for instance. From these warnings we can navigate to the source code and to help topics on how to write more efficient code.

Other reports give us views on the different levels of application structure. We get this information from source code lines to processes. We also get program execution data from the calling and called functions of a specific function to the call tree of the entire application. This is very useful information to assess our application performance data.

Let’s now take a look at how we can set up profiling in Visual Studio 2012.

Setting Up Profiling

Setting up a profiling session can be as simple as running the Wizard shown in Figure 18-1. We can also create and configure a profiling session manually by adding a new session from the Analyze→Profiler→New Performance Session (see Figure 18-2).

Figure 18-2.  Performance Explorer

We can change profiling mode, add or remove profiling targets and other settings from the properties of the performance session (see Figure 18-3).

Figure 18-3.  Performance session properties

Running a Profiling Session

When the requirements for the profiling have been set up we can start a profiling session. We can use the Performance Explorer as shown in Figure 18-4 to start a new session. The profiler runs until we shut down the application under test after which the profiling data is collected for analysis.

Figure 18-4.  Starting a profile session from the Performance Explorer

There are also other options to start a profiling session available from the Analyze menu in Visual Studio.

Examining the Results

After a performance run has completed, the profiling report is presented (see Figure 18-5). The report has several views to help analyze the result. Typically we would start from the Summary view to get an overall picture of the session. Often the Hot-path feature highlights the most expensive path of calls in the session—this is a good way to begin the analysis.

Figure 18-5.  Performance analysis report

Next, use the tools to zoom in and filter on the data in the report. Clicking on a function takes us to the function details where we see details such as the call tree (annotated with time spent/method). We even get the most expensive lines of code highlighted in the code editor to help us quickly analyze the problem (see Figure 18-6).

Figure 18-6.  Performance session function details

Comparing Results

Part of any performance optimization effort is making adjustments and comparing the results with previous runs. The profiling results can be saved and used as a baseline for a later comparison. Or we can use the Performance Explorer and select two reports to compare as shown in Figure 18-7.

Figure 18-7.  Comparing two performance reports

The comparison report in Figure 18-8 lets us filter the result using the collected values. The result is then shown with indicators if the time has gone up or down since our last run, which helps us find performance improvements or regressions quickly.

Figure 18-8.  Performance comparison reports

IntelliTrace

IntelliTrace is a major enhancement made to the Visual Studio debugging environment. Usually when an exception occurs, you start debugging with breakpoints set in suspected areas and check variable/object values for correctness to see whether you can find what went wrong.

If we enable IntelliTrace, the difference will be that if an exception occurs IntelliTrace allows us to just go back in time of our application execution and find out what events got raised and what the values were of various variables, etc. This helps us inspect all the events that finally triggered the exception without the need for breakpoints.

Traditionally a debugger gives us the state of our applications at the current time, and the information about events is limited especially if they occurred in the past of the execution. Using IntelliTrace in Visual Studio 2012 enhances the picture of our application execution so that we do not have to restart the application to re-create past events. We do not have to make educated guesses of what happened prior to an application exception. We can actually see events that occurred in the past and the context in which they occurred. This eases our debugging and makes us find and fix bugs and errors more efficiently.

IntelliTrace helps by recording the following information:

- Debugger events: These events occur within the Visual Studio Debugger while you debug your application. The startup of your application is one debugger event for instance.

- Exception events: These occur for handled exceptions, at the points where the exception is thrown and caught, and for unhandled exceptions. IntelliTrace collects the type of exception and the exception message.

- Framework Events: These occur within the .NET library. The data collected by IntelliTrace varies by event. For a File Access event for instance, IntelliTrace collects the name of the file.

The point of IntelliTrace is that we should be able to debug our applications quicker than before. It also helps us find and fix problems with errors that we cannot reproduce easily. It is much easier to use IntelliTrace than to try and reproduce the error.

IntelliTrace runs in the background and records the debugging information. Once we want to look at the past state of an application execution we enter IntelliTrace and then we can navigate to the points in time we are interested in.

We can create and save our IntelliTrace output as an .iTrace file in either Microsoft Test Manager or Visual Studio. This lets us perform post-crash debugging without losing any of the features of IntelliTrace.

Note  IntelliTrace analysis in Visual Studio requires the Visual Studio Ultimate edition.

Using IntelliTrace has normally very little effect on application performance and is thus turned on by default when debugging in Visual Studio. This way we have the option to use IntelliTrace even if we did not plan to from the beginning. You can turn off IntelliTrace as seen in Figure 18-9. Follow these steps to enable IntelliTrace:

- Select Tools→Options→ IntelliTrace menu in Visual Studio.

- Use the Enable IntelliTrace check box to turn on IntelliTrace.

Figure 18-9.  Even though IntelliTrace has little impact on performance, we can turn it off if we want

The other tabs for configuring IntelliTrace let us setup log location, events to capture, and modules to exclude from profiling.

IntelliTrace supports debugging Visual Basic and C# applications that use any .NET version from 2.0 to 4.5 (http://msdn.microsoft.com/en-us/library/dd264915(v=vs.110)). You can debug most applications, including applications that were created by using ASP.NET, Windows Forms, WPF, Console Applications, Windows Workflow, and WCF. IntelliTrace does not support debugging C++, script or other languages. Debugging of F# applications is supported on an experimental basis. Metro style applications are supported for events only.

Working with IntelliTrace

After configuring IntelliTrace, using it is automatic inside Visual Studio. Once the application runs in debug mode, the IntelliTrace log is recorded. At any point we can open the IntelliTrace window and pause the debug session to look at the IntelliTrace events. Figure 18-10 shows the IntelliTrace Explorer which can be used in these two modes:

- IntelliTrace Events View: This is the default view that lists the events from the trace session.

- Calls View: In this view the events are listed in a hierarchical view similar to the call stack in the normal debugger.

We can also use the IntelliTrace Explorer to get to the configuration dialog for IntelliTrace or save the iTrace file for later analysis.

Figure 18-10.  IntelliTrace Explorer

Selecting an event gives detailed information about the event as well as access to the Local variables collected and the Call Stack (see Figure 18-11).

Figure 18-11.  IntelliTrace Call Stack

The code behind the event is automatically shown in Visual Studio (if the debugger has access to the source code) and we can navigate in the debug history using the controls beside the code (see Figure 18-12).

Figure 18-12.  Using the call naviation in an IntelliTrace session

Chapter 21 looks at how we can use Microsoft Test Manager to collect IntelliTrace in test environments as part of manual or automated testing outside Visual Studio.

Using IntelliTrace in Production

New to IntelliTrace in Visual Studio 2012 is running the IntelliTrace data collection standalone. It is now possible to deploy IntelliTrace to an environment simply by copying the runtime files. Because the technology solves a problem that often only occurs in environment where we do not have development or testing tools installed this is a great enhancement.

To enable IntelliTrace in production see http://msdn.microsoft.com/en-us/library/hh398365(v=vs.110).

PreEmptive Analytics

One way to further enhance and improve your performance analysis is to use the free tool PreEmptive Analytics (PA) from PreEmptive Solutions that comes on the TFS installation. PA aggregates and analyzes exceptions and creates work items from this information based on the rules and thresholds we choose. PA is a very useful tool when an application has been deployed into operations (as well as during development) as it works in the background. PA for TFS can respond in near real-time (matter of seconds).

PA for TFS collects runtime data transmitted from instrumented applications, analyses the aggregated production incidents against rules and thresholds that you can specify yourself. Your applications automatically send back exception report data to the PreEmptive Analytics endpoint service as errors occur during their execution and the PreEmptive Analytics aggregator service collects these error reports and automatically, when appropriate, creates and/or updates TFS work items.

PA allows you to instrument your application(s) and receive reports from your users on any crashes they experience. The reports are analyzed, correlated with other reports, and distilled to a set of production incidents that appear to be the same underlying cause. These show up as work items in your Team Foundation Server database.

Note  Feedback Driven Development is a set of techniques for measuring progress based on customer interaction. Progress could be anything from “Are we completing our user stories?” to “Are we making profit?” Learn more about Feedback Driven Development and PreEmptive Analytics at http://www.preemptive.com/images/stories/ri_documentation/preemptive_analytics_tips_dec_2011.pdf.

Installing PreEmptive Analytics

PreEmptive Analytics consists of a service and a client-side component. The service is setup as an endpoint that clients connect to and upload diagnostic data for analysis. The client-side component is instrumented into the application at compile time and calls the service to log exceptions. The PreEmptive Analytics Community Edition is bundled with the TFS 2012 installation and no additional downloads are needed to setup the service. We start the installation from the TFS administration console (see Figure 18-13). The setup walks you through the process of setting up an aggregator service which is the endpoint our application connects to in order to report exceptions to the system.

Figure 18-13.  Installing PreEmptive Analytics from the TFS admin console

Configuring Your Team Projects for PreEmptive Analytics

Next we need to provision PreEmptive to each Team Project in which it should be used. We manage this configuration from the PreEmptive Analytics Aggregator Administration Console (see Figure 18-14) by selecting the Team Project and Apply. In effect this action will import the required PreEmptive work item types and SQL Server reports to the Team Project.

Figure 18-14.  Configuring PreEmptive Analytics from the PreEmptive Analytics Configuration Utility

Adding Exception Reporting to a .NET Application

After the core setup, we can configure our system for analytics reporting and TFS to receive the data.

Configure an Application for Analytics Reporting

An application needs to be instrumented with logging calls to PreEmptive Analytics for data to be sent to the service. To set up this we use the Tools→PreEmptive Dotfuscator and Analytics. This opens up a configuration application where we add Inputs and Analytics attributes (see Figure 18-15). There are several attributes we can use to configure the reporting to the PreEmptive service; the most essential are

- ApplicationAttribute: Defines the application to trace.

- BusinessAttribute: Defines the company owning the application.

- SetupAttribute: Indicates tracing should start.

- TeardownAttribute: Indicates that tracing should stop.

For details on how to configure PreEmptive Analytics application attribute see the product documentation (http://www.preemptive.com/products/patfs/overview). Finally, we build the PreEmptive project to produce the instrumented assemblies we will use to get the data collected.

Figure 18-15.  Configuring an application for PreEmptive Analytics

Configure TFS to Receive PreEmptive Analytics Data

The second part of the configuration is to set up TFS to collect the data and create work items for the result. To set up TFS we use the Team Explorer and the PreEmptive Analytics activity (see Figure 18-16).

Figure 18-16.  Configuring PreEmptive Analytics for a Team Project

In the configuration dialog we specify the company and application ids (from the attributes defined in the PreEmptive application file). See Figure 18-17 for an example.

Figure 18-17.  Adding a PreEmptive Analytics application

Examining Incidents in Team Foundation Server

Finally, we can analyze the incident work items in TFS. Because the PreEmptive Analytics incidents are reported as standard TFS work items we can manage them using any of the standard TFS tools. From the PreEmptive Analytics view in Team Explorer we can use the predefined queries and reports as shown in Figure 18-18.

Figure 18-18.  Viewing PreEmptive Analytics incidents in TFS

Summary

Good application performance is essential to the success of most software. Unfortunately, performance analysis is something done very late in the process, often as late as in production when the issue is reported by a customer.

With the built-in application profiler in Visual Studio we can start analyzing performance while developing. It is easy to run a profiling session at any time; we can even use a unit test to drive the profiling session. So with this tool we have little excuse not to deliver well-performing applications.

In this chapter we have also looked at two related tools focusing on helping us understand problems in the environment in which they were found. It is common to spend lots of time analyzing problems that we cannot reproduce in development environments.

With IntelliTrace, we can get an execution log from any environment, including production, which we can open in Visual Studio and replay the events from the environment where the problem occurred.

The second tool, PreEmptive Analytics, lets us instrument our code with rich error logging. When a problem occurs in the application, the data is sent to a service in TFS which logs the problem as a Work Item for analysis. We can then address the issue just like any other bug report in TFS with rich execution context attached to the bug.

CHAPTER 19

Metrics and ALM Assessment for Developer Practices

Developer practices are really what Visual Studio 2012 is all about. Microsoft offers several reports and metrics we can use for analyzing and assessing our code base. There are also many assessment questions in the online assessment targeting this area. Let us start with the metrics.

Metrics

Using the metrics from TFS we can find KPIs that will help us understand if we are successfully working to improve our code. These are useful from both architecture and design viewpoints as well as from a developer viewpoint. Using them will help us improve how we design our application or system.

There are several important metrics we can get automatically from Visual Studio and TFS to get a good understanding about the quality of the development work, including

- Code coverage

- Code metrics

- Compiler warnings

- Code analysis warnings

Code coverage

Code coverage shows us how much of the code has been covered by our automated unit tests. We get the value as a percentage of the whole code base. The difficulty is often to decide what percentage is enough. Should we always strive for 100%, or is 80% enough? This is something the team has to discuss with the product owner in Scrum or any other similar decision maker in other processes. This value is input for the Definition of Done (DoD).

Code metrics

There are different code metrics we can get:

- Lines of code is an approximate number based on Intermediate Language (IL) code. A high count might indicate that a type or method is doing too much work and should be split up. This might also be a warning that code will be hard to maintain.

- Class coupling measures the coupling to unique classes through parameters, local variables, return types, method calls, generic or template instantiations, base classes, interface implementations, fields defined on external types, and attribute decoration. Low coupling is better to strive for because high coupling indicates a design that is difficult to reuse and maintain because of its many interdependencies on other types.

- Depth of inheritance indicates the number of class definitions that extend to the root of the class hierarchy. The deeper the hierarchy, the more difficult it might be to understand where particular methods and fields are defined and/or redefined.

- Cyclomatic complexity is created by calculating the number of different code paths in the flow of the program and shows the complexity of the code. A high complexity makes the maintainability suffer, and it can also make it hard to get good code coverage.

- Maintainability index is an index value between 0 and 100 that represents the relative ease of maintaining the code. The higher the better. A rating above 19 is good. Below that, maintainability suffers.

Compiler warnings

Errors and warnings should be avoided in a project. Allowing more than zero errors or warnings tends to result in the team accepting a lower quality on the codebase, which over time will cause the code to lose maintainability (commonly known as the broken windows theory [see http://en.wikipedia.org/wiki/Broken_windows_theory]).

Track this metric to make sure the number of errors is zero. This should ideally be enforced by automatic build policies (as described in Chapter 26).

Code analysis warnings

Code Analysis in Visual Studio performs static analysis on code, which will help developers identify potential design, globalization, interoperability, performance, security, and a bunch of other categories of potential problems.

The Code Analysis tool provides warnings that indicate rule violations in managed code libraries. The warnings are organized into rule areas such as design, localization, performance, and security. Each warning signifies a violation of a Code Analysis rule.

Code analysis can be used to enforce company policies on the code developers write. We can extend the ones Microsoft offers by writing our own rule set or we can suppress the ones we do not want. Definitely discuss this with your development team and the product owner, as the warnings will have an impact on the effort required before the Definition of Done is fulfilled.

Standard Reports

The Quality Indicators report (Agile, CMMI templates) (see Figure 19-1) gives us an overview of the code quality metrics over time. Based on automated builds, the report shows how code churn, code coverage, test results, and the bug count vary over time.

Figure 19-1.  Quality Indicators report (from http://msdn.microsoft.com/en-us/library/dd380683(v=vs.110).aspx)

Note  The reports in this section are mapped to the particular process template it supports. If you find a report you like, it is possible to customize it to work with another template. In Chapter 32, we describe how to customize an existing report to do this among other things.

Custom Reporting

The reporting capabilities in TFS give us access to most of the information we manage in our ALM process. In the previous section, we have seen how standard reports give us metrics for our project at a general level. By customizing, extending, and creating new reports, we can really get the intelligence to know what works well in our projects and what does not.

Note  In chapter 32, we will look at the details of reporting in TFS, including how to create custom reports based on the data models described below.

Data Warehouse Model

Code Churn Tables

The code churn tables can be used to query for data about code changes made to code under version control and gives us information such as the number of lines changed, along with when and by whom the changes was made.

Figure 19-2.  Code churn data model

See also the Work Item Changeset tables in Chapter 9 as well as the Build Details tables in Chapter 29 for more warehouse data useful for analyzing the development practices.

Assessment

In order to help us evaluate an organization’s maturity in different ALM areas Microsoft has developed its ALM assessments (see Chapter 4). Based on the score of the assessment we received a maturity level for a specific area which we can use for evaluating which direction to take our ALM efforts.

So based on the score we can help the organization reaching the maturity level they need for these areas. The following table (Table 19-1) lists questions that can be used as a basis for an ALM Assessment in the developer practices. The online assessment has many questions covering this area but you might want to consider using some of your own as well.

Table 19-1. ALM Assessment questions

Area

Sample question

Discussion

Code Writing

Are there standards in place for writing secure code?

This is implemented using TFS and Visual Studio by adding custom code analysis rules (or use existing)

Are there standards for writing coding that ensure comments, variable/function names are free from profanity, political, religious statements?

Is there a well-defined general coding practice for namespaces, function and variable names?

Code Analysis

Is there good static code analysis?

If not TFS will help the organization with this.

Is there good performance testing?

If not TFS will help the organization with this.

Is there good stress testing?

If not TFS will help the organization with this.

Are there standards for code coverage of tests?

TFS will give these statistics for the development team, thus we can use TFS to implement these standards.

Code reuse

Are patterns & practices established for code reuse?

Code reuse is a good thing. Unfortunately so far we have not seen an organization that has succeeded with implementing this in a good way. Ambition is often high but somehow this never seems to work out.

Are Frameworks used?

xs

Are Code Snippets used?

Code reviews

Are effective code reviews carried out?

If not in place you can suggest that the developer use peer programming which gives code reviews during code writing.

Is code often checked in that fails unit tests?

Using automated unit tests and gated check-ins will prevent this.

Quality Metrics

Is there a well-defined and thorough check-in process which includes quality checks?

Here we can definitely use TFS to help us with this. We can set different check-in policies that help us improve quality.

Are unit failures measured?

TFS will give us reports of this.

Collaborative Development

Is there an effective versioning and branching strategy?

Most of these topics below are covered by TFS version control system.

Is there an effective way source can be retrieved for debugging a deployed product?

Is there an effective backup and disaster-recovery mechanism in place for source code?

Does the source control system allow for development activity at different geographical sites?

Does your source control system support atomic check-ins?

Does your source control system support branching, merging, diffing, labeling?

Is there a policy governing multiple checkouts?

Does the source repository structure and permissions allow for parallel development?

Version Control Repository

Is all code under effective source control?

Most of these topics below are covered by TFS version control system.

Is the source repository well structured?

Is there a consistent labeling policy?

Is the source control properly secured?

Are the source control policies well documented?

Are all of the organization’s intellectual property (source code, documentation etc.) under effective, secure source control?

Change Management

Are there formal checks-in criteria governing source code changes?

Is there effective auditing of who makes changes to source control?

Is there effective auditing of why changes are made to source control?

Implement a check-in policy that requires comments to a check-in. We can also implement a policy that requires an association with a work item.

Is there effective tracking of builds to source control versioning?

Summary

In this chapter, we have seen how we can use TFS to retrieve information for KPI assessment and also how we can see the development status using standard reports from TFS.

We have also shown how many of the assessment questions from the Microsoft online assessment can help us plan for successful implementation of developer practices.

The next part of this book will cover how we can use TFS to test our system or application.

PART 5

Validating (Software Testing)

Testing our software is always important. Historically, testing has been done only at the end of a project and not continuously. Many times the result has been that if development takes longer to complete, the test phase gets cut down in time in order to meet the release date. Obviously this has affected the outcome and quality of many projects. Luckily this has changed recently, especially with the popularity of Test Driven Development (TDD) and other agile practices.

In this part of the book we look at how Visual Studio 2012 and Team Foundation Server 2012 can help us improve our test efforts through the use of tools and manual testing. We learn how to integrate manual testing with our test process and store the result in TFS so that we can get a good overview of the test status in our projects.

We also learn how to create and handle automated tests, which are great for doing regression testing during our daily development and build efforts.

The Test Lab Management features of Visual Studio 2012 offer powerful tools that help us build a robust test process to increase quality of our products.

CHAPTER 20

Overview of Software Testing

It’s now time to take a look at how test and quality assurance teams can use application lifecycle management ideas to improve their process.

First we will look at the core elements of traditional software testing; test planning, test design, manual test execution, and bug tracking.

When we know what to test and how to do it, we can start looking at automating our tests.

Finally when we have automated all or parts of our testing effort we need a way to integrate the automated tests into our daily work.

But before we dive into the tooling, let’s take a look at some ideas to help us work efficiently with testing in our projects.

Agile Testing

Agile projects are challenging. With a common mindset in which we embrace change and want to work incrementally and iteratively, we have good conditions to deliver what our customer asks for on time.

To get testing to work in an agile environment we need to rethink the testing approach we use. Working with incremental development typically means we need to do lots of regression testing to make sure the features we have developed and tested still continue to work as the product evolves. Iterative development with short cycles often means we must have an efficient test process or else we will spend lots of time in the cycle preparing for testing rather than actually running the tests.

We can solve these problems by carefully designing our tests; this helps us maintain only the tests that actually give value to the product. As the product evolves through increments, so should the tests and we can choose to add only relevant tests to our regression test suite. To make the testing more efficient, we should automate the tests and include them in our continuous integration scheme to get the most value from the tests.

We will now look at ideas to help us design our tests and in the coming chapters we will look at how we can improve our testing process to help us do testing in an agile context.

Defining Tests

To define tests we need to think about what we want to achieve with the tests. Are we testing requirement coverage? Are we testing to make sure the software performs according to our service level agreement? Are we testing new code or re-testing working software? These and other aspects affect the way we think about tests.

Brian Marick has created the model shown in Figure 20-1 that is excellent when reasoning about what kind of tests we should create, when, and for what purpose. Let’s take a look at the model and how it can be used to help us define our tests in a suitable way.

Figure 20-1.  Agile testing quadrants help define our tests

Q1 – Unit and Component Tests

Unit and component tests are automated tests written to help the team develop software effectively. With good suites of unit and component level tests we have the safety net that helps us develop software incrementally in short iterations without breaking existing functionality. The Q1 tests are also invaluable when refactoring code. With good test coverage a developer should feel confident to make a change without knowing all about every dependency. The tests should tell us if we did wrong!

Q2 – Functional Tests

Functional tests are mainly our traditional scripted system tests in different flavors. It is hard to avoid running these tests manually at first, but we should try to find ways to automate them as we learn more about our product and how it needs to be tested. Many functional tests can be automated and then we can focus on early testing for the manual tester.

Q3 – Exploratory Testing

Exploratory testing is a form of software testing in which the individual tester can design and run tests in a freer form. Instead of following detailed test scripts, the tester explores the system under tests based on the user stories. As the tester learns how the system behaves the tester can optimize the testing work and focus more on testing than documenting the test process.

We should leave this category of tests as manual tests. The focus should be to catch bugs that would fall through the net of automated tests. A key motivation for automated testing is to let do more of exploratory and usability testing because these tests validate how the end-user feels when using the product.

We will look more at exploratory testing in Chapter 21 when we look at how we can plan and test using the Microsoft Test Manager product.

Q4 – Capability Testing

Lastly we have the capability tests. These tests are run against the behavior of the system; we test non-functional requirements, performance, and security. These tests are generally automated and run using special purpose tools, such as load test frameworks and security analyzers.

Acceptance Criteria

Acceptance criteria are to testing what user stories are to product owners. Acceptance criteria sharpen the definition of a user story or requirement. We can use acceptance criteria to define what needs to be fulfilled for a product owner to approve a user story.

Chapter 6 looked at the agile planning process and how product management can use user stories to define the product.

With acceptance criteria we have yet another technique to help us refine the stories.

A user story can be stated as simply as

- As an Employee I want to have an efficient way to manage my expenses.

The conversation around this statement between the product owner and the development team can raise questions such as:

- Who can submit expense reports?

- What states can an expense report be in?

- When is it possible to change or remove an expense report?

- What data is required in an expense report to register it correctly?

- Where are the expense reports stored? For how long?

We use this information to formulate acceptance criteria. Take for instance the question “what states can an expense report be in?” From this we can formulate acceptance criteria such as the following:

- An expense report has the following state model:

- New when created

- Pending after submitted for approval

- Approved

- Rejected

This exercise then leads to more questions to the product owner, for instance should an employee be able to change an expense report that has been refused? So having this type of conversation not only helps us know what to test but also helps define the product.

When working with TFS we can capture all important pieces of information from this process in the product backlog work item. The PBI gives good traceability to follow the user story (requirement) to its acceptance criteria. Figure 20-2 shows an example of how the Visual Studio Scrum template shows this information side by side.

Figure 20-2.  A product backlog item with acceptance criteria

Planning

Test planning typically involves creating a test plan. The test plan captures the requirements for a period of testing. The test plan includes information about aspects of the quality assurance process, including

- Scope of testing

- Schedule

- Resources

- Test deliverables

- Release criteria

- Approvals

The IEEE 829-2008[1] standard can be used as a reference to create the test plan documentation structure for a project. We can create this as an overview document with links to the details either in TFS or in other documents.

Test Specification and Test Matrix

A big part of a test plan is the list of tests to run during the project. This list is often referred to as the test design specification. The test specification contains the details about which features to test, test cases, grouping of tests into categories, their priorities, and so on. Most of this can be captured in TFS by using product backlog items and test cases, but to get a good overview we should work with a test matrix to summarize the test cases. The test matrix helps us find relevant tests when a requirement is changed, find all tests of a particular priority, or get a list of all automated tests.

The table in Figure 20-3 shows how to define a test matrix. In Chapter 21 we come back to this matrix and look at how we can map this typical table into the test plan in Microsoft Test Manager and use TFS to store our test plans and test cases and at the same time get the metrics automatically calculated for us.

Figure 20-3.  Basic test matrix to track test cases with priority and status

Evolving Tests

As a part of the agile process we need to deal with an incremental and iterative development of test assets. As the product goes through the specification-design-implementation-release cycle the test cases also need to adapt to this flow as well. Initially we know very little about a new feature and we typically need to run tests against all acceptance criteria defined for the requirement. When a feature has been completed we should be confident it has been tested according to the test cases and that it works as expected. After that we only need to run test to validate changes in the requirement, which means we must have a process for how we know which tests to run.

Another side of the agile story is to look at how to speed up the testing process to keep up with short iterations. If we follow the preceding ideas we can have techniques to know more about which tests to run. But running all tests manual will probably not be feasible so we need to rethink how we design these test cases.

One way to think about how we can structure our test base is to think of it as a pyramid. Figure 20-4 shows how the types of tests from the testing quadrant can be put in proportion in our specific case.

Figure 20-4.  Proportions of types of tests

Typically we would focus on a big part of unit and component tests because these are the cheapest to implement and maintain. But these tests do not test the system as a whole so we need to add regression tests to run end-to-end tests as well. Some of the regression tests should be implemented as user interface tests to really simulate how an end-user would use the system, but UI tests are more complex to design and maintain and it is often not practical to have more than a small set of these tests. Most of these tests can and should be automated to give us an efficient way to keep up to the changes in the product.

Strategy for Automated Testing

Figure 20-5 shows an approach for evolving tests that we have found practical to use as a model.

Figure 20-5.  Strategy for evolving automated tests

The principle here is that when we start with a new requirement we know very little about how the feature behaves and how it needs to be tested. So in order to test it, we typically run manual tests against all acceptance criteria.

When we have verified the new feature and know that it works, we can look at the tests used to achieve this. Based on this knowledge we can do two things; select which test cases we should keep as regression tests and also select a set of test cases to implement as automated acceptance tests.

In coming sprints we then run all automated tests as well as the manual regression tests, which should give us confidence for the quality in the product and help us avoid regressions when evolving the software.

If we look at the automated regression tests, we often find that these represent realistic end-user scenarios. Wouldn’t it be great if we could take those scenarios and use them for performance testing as well? Well, it turns out we can! By using the load testing capabilities in Visual Studio we can plug in the regression tests in performance test suites. We can also start running the performance tests early in the development process because the team can have all tools needed installed and ready for use.

Platform Support for Testing Practices

Finally let’s take a look at how the concepts we have looked at fit together. Figure 20-6 illustrates how we added testing to the shared ALM platform. The traditional tester focuses on test case management (planning, designing, and running manual tests). The technical tester works on automated tests—building on the initial work from manual testing. Developers create unit tests as part of the development process. Regardless of who does what we now have a platform for testing to work on and all is available to us as part of the Visual Studio platform!

Figure 20-6.  Overview of Microsoft Test Manager and Team Foundation Server

Summary

In this chapter we looked at some of the key concepts in agile testing and how this changes how we do testing in an agile project. Agile testing requires us to start testing early because we aim to deliver working software in every sprint. This means we need to define tests in parallel of feature development and run the tests again as the software evolves. Acceptance criteria is a powerful technique to use to sharpen the definition of user stories and at the same time get the foundation for testing the stories in place. To manage this process we need a solid model for tracking the test cases to know when to re-run tests to avoid regressions in our software.

In the following chapter we will look at how we can apply these principles to our testing process using Visual Studio 2012. First we will create the test assets and run manual tests and then we will look at how we can automate the test cases using several different testing tools available in Visual Studio.

[1] http://standards.ieee.org/findstds/standard/829-2008.html

CHAPTER 21

Manual Testing

Great quality does not happen by itself. We need to build quality into the process to get great results. The pillars of ALM (traceability, visibility, and automation of processes) should be part of the testing effort to give us predictable quality in our projects.

In this chapter we look at how Microsoft Test Manager (MTM) can be used to manage the testing process. But before we dive into the details of working with MTM, let’s take a look at what we mean by a testing process. In Figure 21-1 we have an example of a common model for testing.

Figure 21-1.  A manual testing process

First we start with planning, this is where we look at the requirements planned to be implemented and decide on how much testing is needed to validate that the requirements are implemented correctly. When we have a plan we can design our testing efforts; this includes of course writing test cases but also setting up test environments and test configurations. With the test assets in place, we can run tests of different kinds; scripted tests and exploratory tests, as well as automated tests. If things go wrong, we file bugs and track how the bugfixes are coming along. When a bugfix is ready to be verified we want it to be simple to get back to the failing test case and re-run the test to verify the fix. Finally after going through this cycle for a couple of times, we want to find our candidates for regression testing. Then we can choose to automate those tests so that we can have quicker test cycles in coming sprints.

This may seem like a waterfall model, and yes, the process is sequential, but we still can work iteratively and incrementally. In an agile project we plan, design, test, and so on in every iteration. Activities such as integration tests and test automation often need more of the release completed to perfom these activities. It is then we adapt and plan those activities for a sprint when the prerequisites are in place.

Now that we have an idea of a testing process let’s look at how MTM gives us tooling to work according to the process in an effective way.

About Microsoft Test Manager

Microsoft Test Manager is the Visual Studio for testers, the one-stop shop for the entire test process. A tester can do all the testing activities within a single application (not entirely true, but pretty close actually).

At a high level Microsoft Test Manager handles

- Test planning

- Test design

- Test execution and test run analysis

- Rich bug reporting with data collection from the machines under test

- Work Item tracking (including bug tracking, of course)

- Test environment management

Sounds like a good match to our proposed test process, doesn’t it?

Figure 21-2 shows how artifacts in TFS and MTM are related in the context of a test plan. We will look at the details of each in the coming sections in this chapter.

Figure 21-2.  TFS and MTM artifacts related to a test plan

Note  Is there a Web UI for the tester? No, the Microsoft Test Manager client is a rich desktop application that needs to be installed where you plan to work with the test assets. Some components such as test cases and test results can be accessed using other clients but you will only get partial functionality outside MTM. There is a third-party solution available as a plug-in to the TFS Web Access called the Web Test Manager, which may be an option: http://www.selagroup.com/alm/products_WTM.html.

Connecting Microsoft Test Manager to TFS

MTM is always connected to the TFS server so the first thing to do when starting MTM is to connect to the TFS project we are working on (see Figure 21-3). The connection to TFS is not only in design-time but also when running tests. This is something to be aware of if we want to perform acceptance testing in a customer environment where typically there is no access to the TFS server.

Figure 21-3.  Connecting Microsoft Test Manager to your TFS project

Note  If your TFS is published over HTTPS or on a different port, just fill in the entire URL. For example https://alm.tfspreview.com connects MTM to a TFS collection on the Team Foundation Service cloud service.

Planning the Tests

In the previous chapter we looked at test planning and what a test plan typically contains. The test specification part of the test plan is something we can directly map to artifacts in TFS and MTM.

What Is a Test Plan?

In MTM a test plan is essentially two things—details about test effort and the set of tests to run as part of the plan (called suites in MTM).

We recommend keeping test plans small; we prefer one test plan per sprint over one for the entire the release. Small plans are more to the point and maps well into the test process. If we look at the status of a test plan, we can grasp what it means. If the plan covers the entire project, it is much harder to understand whether we are progressing as planned. The data in small plans can still be aggregated into reports over the entire project, we just have to use TFS reporting to do so (see Chapter 32 for more information on how to do that).

Creating the Test Plan

First, we need to create a test plan.

For our scenario we will start with nothing and create a test plan for the first sprint as shown in Figure 21-4.

Figure 21-4.  Adding a new test plan

Test Plan Properties

The properties of the test plan contain general information about the test plan, details on how test runs are set up, as well as links to documents and other resources. The links are useful when we want to add context to the plan that does not have a place in MTM, for instance we can reference a test plan document containing all details about the testing for the release we are working on. Figure 21-5 shows the test plan properties.

Figure 21-5.  Test plan properties

Suites

Suites group together the tests we want to run and track in this plan. We can choose from three types of suites:

- Static suite: The content of this suite is manually added test cases.

- Query-based suite: A query-based suite lists all test cases matching a given work item filter.

- Requirements-based suite: This suite shows the test cases associated with a selected TFS requirement.

The Query-Based suite (see Figure 21-6) is great for any situation where you want to make sure you have an up-to date list of tests based on some criteria. Typical usages are suites of tests for a specific application area or all automated tests.

Figure 21-6.  Query-based suite for all automated test cases

The Requirement suite is a little different. Here we use a work item category called that maps to the configured work item type(s) representing a requirement. In our scenario using Scrum this would map to Product Backlog Item and Bug.

Note  If you want to see which work item types are mapped to the Requirement Category in your project you can use the witadmin.exe tool and run the following command:

witadmin exportcategories /p:your_tfs_project /collection:your_tfs_collection_url.

We would typically add all requirements in the sprint to the test plan to associate the acceptance tests with the corresponding requirement. Figure 21-7 shows how we use a work item query matching the Requirement Category and Sprint 1 to find the requirements that we now can add to our plan.

Figure 21-7.  Adding requirements for Sprint 1 to our test plan

Note  Removing a test case from a requirement deletes the link to the requirement and therefore affects other plans using the same test case/requirement association.

The complete structure for the Sprint 1 test plan is shown in Figure 21-8 with placeholders for requirements tests, exploratory tests, and automated tests.

Figure 21-8.  Complete structure for the Sprint 1 test plan

Moving On

We now have a test plan setup. At this point we can choose to continue and do test case design, or start testing by running exploratory tests. In MTM 2010 we needed to have at least one empty test case to run a test, but with MTM 2012 that requirement is removed, so you decide what your needs are. We start by adding test cases first.

Designing Test Cases

At this point we are ready to add some test cases to our plan. Specifically we want to add test cases to test the acceptance criteria for the requirements in our sprint, but we can add any type of test case.

What Is a Test Case?

A test case in MTM represents the test instruction for a tester. It is implemented as a TFS work item, which means we can customize it so that it contains the information the tester needs to complete the test run. The test case can be viewed and changed in any TFS client except for the test steps, which can only be done in MTM. Let’s walk through the essential elements of a test case.

Steps

The steps section is of course the central part of the test case as shown in Figure 21-9. We add steps for the test instructions and provide expected results. The expected result is particularly important to spend some time thinking about because these are the validation points that we use to assert that the test case is testing the right thing. If the expected result is well formulated we can use it to validate the test step in a manual test as well as if we automate it, saving time and making the test runs more repeatable.

Figure 21-9.  Test case with formatted steps

Use formatting to highlight important sections of the test steps. Worth mentioning is that the test step is selectable when the test is being run so if you provide a URL in the test step, the tester can copy the URL and paste it into the browser instead of having to type it.

For recurring steps we can create Shared steps. Shared steps are stored as a separate work item and can be shared between test cases, for example to encapsulate the login steps which might be the first sequence in many test cases.

If we want to test multiple combinations of a test, say for instance to test how the application behaves for users of different roles, we can add parameters to the test case (see Figure 21-10).

Figure 21-10.  Test case with paramters

Each set of parameters shows as test iterations when the test case is run And we also get the nice effect that each data value is copied into the Windows clipboard so we can paste it into the target UI element.

Test Case Summary

The test case summary contains a description field that is useful for documenting the purpose of the test case ( see Figure 21-11). This field is also shown in the Test Runner when later running the test, so use it to write reminder notes for the tester.

Figure 21-11.  Test case summary

Creating Test Cases

In our scenario we want to add test cases to our first requirement “Create an expense report.” The requirement has acceptance criteria defined for it, which is great input to our test case design. As a start we can create one test case for each acceptance criteria and later we can add more test cases for edge cases as we find need for. The complete requirement was shown previously in Figure 20-2. Let’s start by creating a test case for the “An employee can create a project related expense report” acceptance criteria. When adding a test case for a requirement, MTM automatically creates a link to the requirements under the Tested Backlog Items tab. Figure 21-12 shows a completed test case.

Figure 21-12.  Test case for creating a project related expense report

After adding test cases to cover the acceptance criteria, we can take a look at the product backlog item again. A small but effective feature of the Scrum work item design is that we can view the list of test cases at the same time as we see the list of acceptance criteria, as shown in Figure 21-13. This is a great way to check whether we’ve added test cases to cover the requirements.

Figure 21-13.  Create Expense Report requirement with acceptance critera and test cases

With the test cases in place, we now have a test plan ready to start testing (see Figure 21-14).

Figure 21-14.  Sprint 1 test plan with test cases

We can however add some additional structure to the test cases before entering test mode.

Test Configurations

The test configurations allow us to define the test matrix for our tests, for example we need to test our application on Internet Explorer 9 and 10. To do so we can create matching configurations. This is done from the Organize tab in MTM by managing Configuration Variables (see Figure 21-15).

Figure 21-15.  Adding a test configuration variable

With the variables defined, we can add test configurations (see Figure 21-16).

Figure 21-16.  Adding a new test configuration

Finally, we can assign each test to the corresponding configurations as shown in Figure 21-17.

Figure 21-17.  Mapping test cases to test configurations

Assign to Tester

If we have many test cases and many testers, it can be effective for the test manager to assign test cases to the designated tester. One way to divide the work can be to assign test to users by configuration (see Figure 21-18).

Figure 21-18.  Assign test cases to tester

Grouping and Adding Fields

A slightly hidden gem in the MTM UI is that most lists have a pivot feature that allows us to drag columns over the top of the list to group on that field.

We can also add additional columns to the list by right-clicking on the column row.

Figure 21-19 shows examples of grouped columns and column options.

Figure 21-19.  Customizing the work item grid in MTM

Test Suite Status

There is a nice feature in MTM to help us control when tests are available for testing. Each test suite has a status we can set to In planning, In progress, or Completed. Only tests in suites with status In progress are shown in the Test view in MTM.

Moving On

To summarize, here is our shortlist for test design and planning:

- Create the test suite structure, mark suites as In planning.

- Create test cases for requirements and scripted tests.

- Review configurations.

- Assign tests to testers.

- Review the test plan, test suites, and requirements.

- When happy, set the suites to In progress.

With this said, let’s move on to testing!

Running Tests

The testing activity in MTM can be seen as a dashboard for the tester. Here we can do most of the tasks related to testing, including running tests and analyzing test runs, doing exploratory tests and viewing exploratory test sessions, as well as tracking and verifying bugs.

Let’s start by looking at our scripted tests first and then move on to exploratory testing. But before we start it is good to have an understanding of how the Run Tests view works. From the test view we can

- Run the test (with options to override the plan default settings). You can select tests to run in many ways, for instance by suite or by multiselecting the tests to run.

- View the result of the last test run for a test.

- Open test case to read up before starting the test.

- Change the test status. We can set the test to blocked, reset, passed, or failed without starting a test run.

Filtering Test Runs

Previously we mentioned that as a test planner we can assign tests to testers and configurations. When a tester wants to use that information we can create a filter in the Run Tests view so that we only see the relevant set of tests (Figure 21-20).

Figure 21-20.  Filtering test runs based on tester and configuration

Working with the Test Runner

Now we are ready to run our first test. Starting the test in MTM opens up the Test Runner, which is another part of the application. The Test Runner starts in a mode where it takes over the left part of the screen and scales the other area, which is nice if you want to test your application in full-screen mode. You can change docking behavior if you want to position the window in a different way, as shown in Figure 21-21.

Figure 21-21.  Selecting the Test Runner screen position

When starting a new test we get to choose if we want to create an action recording. An action recording is a recording of all user interaction for the test and is a script that we can use later for automatic test playback in MTM or to generate an automated test in Visual Studio.

The Test Runner displays the test steps that we can mark as passed or failed as we go through the test case (see Figure 21-22). If the test step contains parameters we can bind those to the application we are testing by pasting the data value from the clipboard, which speeds up testing multiple iterations. The parameter value is copied into the clipboard by default when you move to the test step containing the parameter. If you want to copy the parameter explicitly you can do so by just clicking on the data link.

Figure 21-22.  Working with a test case in the Test Runner

If the test step has a validation point, it is also displayed in the test step description.

When we are running multiple tests and/or iterations we can easily switch between them using the navigation control in the upper-left part of the Test Runner (see Figure 21-23).

Figure 21-23.  Moving between tests and iterations in the Test Runner

One feature in the Test Runner that can be difficult to spot is the test case summary. If you want to read the summary there is a little expander link just above the list of test steps (see Figure 21-24).

Figure 21-24.  Viewing the test case summary when running the test

Another nice feature is the possibility to quickly switch between the running test and MTM by clicking the little window icon in the top toolbar as shown in Figure 21-25. Switching from the test run back to MTM pauses the current test but only for the length of the MTM session. If you close MTM, the test run will be marked as failed.

Figure 21-25.  Switching between Test Runner and MTM

When in MTM, the test run is shown as In progress and we have the option to resume manual testing to get back to the Test Runner window (see Figure 21-26). We can only have one test pending; starting a new test run with another already running ends the running test.

Figure 21-26.  Resume manual testing

When the test is completed, we get back to the test view. If we want to view the data from the test run later we can just press the View result button and get to the details from that particular run for the test case.

Analyze Test Runs

After the test run is complete we may want to analyze the result or have someone else take a look at the findings. We can always get to the latest result from the Run Test view, but if we want to work with test runs in general we need to switch to the Analyze Test runs view. From the main view we can do basic filtering and grouping of the test result. Note that by default the view is set to show results from automated tests, something that may not be what you expected but primarily this view is used for following up automated test runs. Figure 21-27 shows the Analyze Test Runs view.

Figure 21-27.  Analyze Test Runs view in MTM

A failed test run is shown in state “Needs Investigation,” which we can fix by opening the test run and analyzing the result and then marking the run as completed (see Figure 21-28). Marking the run as completed won’t change the status of the test result for the test cases in the run—just show that we have taken action on the test run result.

Figure 21-28.  Working with Test Runs details

If we scroll down to the Tests section in the report we get a list of the tests that were run in this test run (see Figure 21-29). Here we can drill down and look at the details of a test run and raise a bug afterward. We can also make a decision about the cause of a problem by selecting failure type and resolution.

Figure 21-29.  Test Runs analysis actions in MTM

Opening the result of a particular test shows the details of that test run. One detail to pay a little extra attention to is the Result History list. The result history is a good way to learn more about the test when troubleshooting. If the test works all the time, we probably have introduced an error. If the test sometimes passes and sometimes fails, we could have a regression issue here. Or there may be a problem with the test case, perhaps we need to add more information in the test case so that we can make sure it is run the same way every time.

Note  Test results are stored per test plan. If you want to track test results consistently it usually works best to have small test plans, each with a distinct purpose.

Running Exploratory Tests

We can also run exploratory tests in MTM. This is a new feature in MTM 2012 that allows us to very quickly start testing without having to create a test case up-front. The test experience is similar to the one we have in the standard Test Runner but because we do not have a test case behind the scene, it is naturally more lightweight. There are several ways to start an exploratory test. The Do Exploratory Testing view in MTM is the most common, but we can start an exploratory session from a requirement or the test plan as well.

The Do Exploratory Testing activity (see Figure 21-30) has some features to be aware of. It is quick to start exploring by just pressing the Explore button, but we can also choose to explore specific work items by selecting one or more from the Work Item list. Running an exploratory test on a work item is the same experience as without but we get an association with the work item that we can use later on (for statistics or to create a test case with link to the tested requirement for instance).

Figure 21-30.  Exploratory Test view in MTM

Starting the test opens up the exploratory test session. This is a simple version of the standard test runner that we use to document the test session as we run the test. We can do rich text editing and include screenshots as needed. We can even double-click on the screenshot and open it in an image editor to format or annotate it. Figure 21-31 shows the exploratory test runner.

Figure 21-31.  Exploratory test runner

If we find a problem we can create a bug directly from the test session. The same thing for a test case, if we realize when testing that this session should be kept as a scripted test then we can quickly press the Create test case button to create one. Both of these features copy the result from the exploratory test into the bug or test case for reference. When you create a bug from an exploratory test it is possible to change which steps to include in the bug by clicking on the “change steps” link in the Steps to Reproduce section. This is good if it was a long exploratory test session and you don’t want to get too much noise in the bug report.

Note that the test case does not have a test result the way a scripted test run does, an exploratory test is simply just run. If we find an issue, then we raise a bug from the exploratory test rather than fail the whole test run.

Test Settings

When we run tests we can manually add content to the test results such as attaching files, including screenshots, or writing comments. This is great for traceability and for test run analysis but for bug fixing it is also practical to get detailed information about the system under testing so that a developer can reproduce the problem quickly. For a tester it is often difficult or time-consuming to manage this part of the test run so to solve that problem test platform configures how tests are run, as well as what data gets collected. We control this with test settings in Visual Studio or MTM.

Note  Visual Studio Test Settings are covered in Chapter 16.

The default settings for test runs can be assigned to the test plan on the test plan properties view. We also can override the default by choosing Run with Options when starting a test run. Either way, we get the test framework to locate a test setting we have configured earlier.

To manage test settings we switch to the Lab Center in MTM and select Test Settings. For manual tests it is very straightforward to create a new test setting; first we give the test settings a unique name, then we select the environment where the tests are run (see Chapter 23 for more information about test environments), and finally we add the diagnostic data adapters to use in the test run.

For automated tests we can also configure how the test environment should behave during test, for instance we may need to deploy files to the environment before running a test or to execute pre- and post-test run scripts to initialize and cleanup the environment.

Data Collection

The central part of a test setting in MTM is the Data and Diagnostics section where we can specify which data collectors we want (see Figure 21-32). Most of the data adapters are configurable to help us fine-tune the data collection for best result.

Figure 21-32.  Creating a new Test Settings

The built-in data adapters in MTM 2012 are listed here:

- Action log: Used to collect UI interactions when the test is run.

- ASP.NET Client Proxy for IntelliTrace and Test Impact: Used to collect IntelliTrace and Test Impact data from a web server.

- Event Log: Capture events from the event log, which event to capture, as well as how many events to collect can be configured.

- IntelliTrace: Used to collect run-time and exception information from the system under test that can be used by developers to speed up the time it takes to understand the cause of a problem.

- System Information: Gathers system information from a machine, such as amount of RAM, operating system, and browser type.

- Test Impact: Collects test coverage data used to calculate test impact so we can get help to decide which tests to re-run based on code changes.

- Video Recorder: Records the desktop where the tests are run. Can be configured to only store video recordings on failed test, which can help reduce the amount of data stored in the TFS database coming from test results.

Most of the adapters also lets us configure how the data collection should work, for instance by controlling whether a video recording should be saved for successful test passes as shown in Figure 21-33.

Figure 21-33.  Configure test settings in MTM

Note  If you have data in your application that could help troubleshooting a bug, then you can extend MTM by creating a custom diagnostic data adapter. A custom adapter gets called by the test infrastructure during test execution and can, among other things, pass files to the test engine when a test case completes. More information on how to create a custom diagnostic adapter can be found on MSDN: http://msdn.microsoft.com/en-us/library/dd286737.

When we later want to use the test settings we either assign it to the test plan or choose the test setting when we start a test run by selecting Run Options (see Figure 21-34).

Figure 21-34.  Specifying Test Settings when starting a test

Note  See Chapter 18 for information on IntelliTrace and how that can be used by developers to reduce the time it takes to reproduce a problem.

Typically we will create a test setting per type of test scenario, for instance local testing, detailed diagnostics, and remote testing. The recommendation is to use as cheap test settings as possible. This way we can speed up testing and reduce the amount of diagnostic data that gets collected and when needed we re-run tests using a different setting to gather more information.

Note  The test results stored in TFS can quickly fill up the TFS database so be conservative with what data you save from test runs. The size of the TFS database does not only affect the operational performance, but may also slow down maintenance jobs such as backups. The TFS Power Tools contain a test attachment cleaner tool that can be used to remove unnecessary test artifacts (http://msdn.microsoft.com/en-us/vstudio/bb980963.aspx).

Integration with Builds

So far we have worked with test plans and test cases without a direct relation to the system under test. Okay, we have looked at how we can use product backlog items to document acceptance criteria. We can also use the backlog as a tool for planning which features are implemented when. But when it comes to keeping track of which version of the system we are testing on or to track in which codebase a bug has been integrated in we need more.

One solution to this challenge is to use TFS builds. With TFS builds we can track which code we are testing by assigning a build to a test plan or when starting a test run. Also by having the build associated with the test run we can tag bugs created from the test run automatically. When a developer checks in code and associates the changeset with a work item, the TFS build can, if configured to do so, tag the work item as integrated in that particular build. So as you see we get a lot of nice additional capabilities in place by associating builds to tests. Let’s now take a look at how to set it up.

Assign Build

The Assign Build view in MTM is useful for assigning a build to the test plan as well as to find out what has been changed in a build since the one we are currently using. The latter can be a good tool for us to help figure out whether we should deploy a new build or wait until more features have been completed. In Figure 21-35 you can see there has been one bug fixed from the build in use compared to the latest build.

Figure 21-35.  Assign Build and looking at changes made between builds

Note  If you are not using TFS builds in your project you can still get some or all of the functionality by creating your own fake builds. The fake build needs to be created so that it includes the information to be used by MTM, for instance a drop folder with build binaries or just the build number to be listed in the “Found In/Integrated In” fields in the bug report.

For more information about how to use the TFS API to create a fake build see http://blogs.msdn.com/b/jpricket/archive/2010/02/23/creating-fake-builds-in-tfs-build-2010.aspxt.

If you just want to get the build into TFS check out Neno Loje’s command-line tool that wraps the API mentioned above: http://msmvps.com/blogs/vstsblog/archive/2011/04/26/creating-fake-builds-in-tfs-build-2010-using-the-command-line.aspx

Recommended Tests

Another interesting feature in MTM is Recommended Tests. Recommended tests help us decide which tests might have to be run. The calculation is based on data from previous test runs matched against changed code checked in to TFS. This feature is called Test Impact Analysis and works for any test run against managed code. To resolve the changes we must also use TFS build, this is used for two things; to define the baseline to do test impact analysis against and then we use another build to calculate the difference.

Note  Only test cases are shown in the Recommended Tests view in MTM. For automated tests we need to associate the automated test method with a test case for it to show in Recommended Test view. See Chapter 22 for guidance on how to connect an automated test to a test case.

To get started with Test Impact Analysis and Recommended Test use the following procedure:

1. Create a test setting in MTM that collects test impact data.

2. Create a TFS build.

3. Assign a build to the test plan in MTM. This is used as the baseline for determining recommended tests.

4. Run a manual or automated test. Only successful tests are used to generate impact data, using partially successful could give unpredictable result.

5. Write code, check in, and create new builds.

6. Select a new build and compare it against the previous build to find recommended tests.

Recommended tests should be used to get an indication of tests to run; there are many factors that can affect the outcome of the test impact analysis so we recommend using it together with other techniques to select the tests to run.

Reporting Bugs and Validating Fixes

At this point we have looked at test planning, design, and different ways to run tests. The final thing to deal with when testing is the inevitable case when you discover an error in the application.

The features in MTM for bug management are designed to be simple to use, as well as help us speed up the time it takes to turn a bug around from found to fixed to verified. This may sound like utopia but thanks to the integration in the test runner applications, as well as the data collectors we can actually manage the bug processes in a really nice way using MTM.

Creating a Bug

The first part is to create a bug. Creating a bug can be done from both the Test Runner and the Exploratory Test window (Figure 21-36).

Figure 21-36.  Creating a bug when running a test

When a bug is created, data from the diagnostic data adapters is collected and added to the bug report. The test steps are also copied into the bug form which makes it really simple and quick to report a new bug. In fact it is so easy to do that we can report bug for any problem we encounter, be it during development or test. Figure 21-37 shows an example of a bug report with rich test run data attached.

Figure 21-37.  A bug report created from a test run in MTM

It is sometimes argued whether it is a good practice to report bugs in a sprint where the tested feature is still under development. Our recommendation is that it is better to report the bug and let the team decide when to deal with it rather than distracting a developer with the issue right away.

Note  How does MTM know which work item type is a bug? MTM uses the Bug work item Category to find the default type for a bug and open that form. If your work item type for bugs is called Defect instead of Bug you can update the work item category to reflect this.

Reporting bugs directly from the test run is of course the most common way to do it. But what if we forget or didn’t think it really was a bug and want to add one later? Rather than having to create a bug report from scratch we can instead use the test run result (see Figure 21-38).

Figure 21-38.  Create a bug report from the rest run result

Verifying Bugs

After a bug has been filed it will go through the process of triage and development until it is eventually fixed and ready for test. To make the process of finding the state of a bug easy we can use the Verify Bugs view in MTM (see Figure 21-39).

Figure 21-39.  Verify bugs in MTM

A good way to work with this view is to look at the state and assigned to track the progress of the bug. In the previous example we can see that the bug has been approved and committed by the team to be fixed. No one is working on it yet because the Assigned To field is empty. When the developer fixes the bug, the check-in should eventually become part of a TFS build which in turn can update the Integrated In field on the bug as shown in Figure 21-40.

Figure 21-40.  Verify bugs in MTM with integrated in build set

Note  Why is the Verify button sometimes disabled? We can only use the Verify workflow when the bug is associated with a test case. MTM uses that information to open up the test case for us when we go and verify.

Fast-Forward Playback

When we verify a bug fix we can either run the test case again or we can use a feature in the test runner called fast-forward playback. As the name implies, this feature allows us to replay a test session and it does so by using an existing action recording associated with the test case. As you can see in Figure 21-41, each test step with an associated action recording is shown with an orange line next to it, this information also tells us a little about how the actions were recorded. To use the feature with the best result we recommend you focus on getting a clean recording and make sure to mark each step correlated with the action log. There is no way to edit the action recording later so the only option is to re-run the test and save a new action log.

Figure 21-41.  Fast-forward playback using MTM

The playback feature is of course just as useful (perhaps more) during normal testing as well. How great to be able to run a regression test with the click of a button!

Note  Action recordings are saved per test plan which means that a test case can have a different action recording in different scenarios.

Summary

In this chapter we have covered a lot of ground and you have seen how Microsoft Test Manager can be the tool for almost all activities in the test process. We can do test planning and assignment, work with test case design, and run tests from scripts or as exploratory test sessions. From the test runs we can analyze and file bugs for errors that we find and track the fixing process—all within one single application.

In the next chapter we look at how we can automate testing and save work in the process. Visual Studio has a wide range of tools for automated testing, ranging from UI testing to stress and load testing.

CHAPTER 22

Automated Testing

In the previous chapters we looked at the test process in general and how it can be designed to support agile teams and agile testing. Initially we can (and probably should) define and run our tests manually by using Microsoft Test Manager.

In this chapter we look at how we can evolve from manual tests to automated tests as we learn more about how our system needs to be tested.

Automated tests can also be used to test non-functional requirements. We can continue to build on the same automated tests and create performance and load test sessions to validate production-like use of the system we are developing.

From the ALM perspective we should think about more than just the technicalities of writing automated tests. To get the most of the effort we put into automated testing we need to look at how the automated tests fit into the overall process. How can we track tests against requirements? How can we manage the environments where tests are run? How can we run the tests in an efficient way? These questions and more need to be addressed to get the most value out of the automated tests.

Working with Automated Tests

To achieve the goals of automated testing, we need to plan ahead and think about what we really want to get out of our automation efforts. This section looks at the infrastructure available in Visual Studio to create different types of tests. We also look at how we can set up and manage the test environments so that we can run the tests in an efficient way. Finally, we are also going to look at how we can structure our tests to make it possible to follow up the tests and also get traceability between tests and requirements.

Supported Test Types

Visual Studio 2012 has support for a number of different test types ranging from basic unit tests to automated UI tests up to complete load testing capabilities. What is really nice with working with tests in Visual Studio is its shared tooling for designing and running tests. This is very convenient because now we can start by learning the type of test we want to begin working with, leverage the framework and how we design, run and follow up test runs. As we add additional types of automated tests we do not have to learn new practices, just add the new ones to the existing platform. Figure 22-1 shows how consistent it is to add different test types in Visual Studio.

Figure 22-1.  Test types supported in Visual Studio 2012

Table 22-1 explains the purpose of each type of test.

Table 22-1. Description of Test Types Supported in Visual Studio 2012

Test Types

Purpose

Basic Unit Test

An empty unit test.

Unit Test

A basic unit test with a TestContext and additional test attributes.

Coded UI Test

A Coded UI Test.

Coded UI Test Map

Creates a UI Test Map that can be used to split the UI definitions in a Coded UI test into smaller pieces.

Generic Test

The generic test can wrap an existing function into an MSTest test.

Ordered Test

An ordered test can be used to control a set of tests are executed.

Web Performance Test

Records a web test using Internet Explorer.

Load Test

Launches a wizard to generate a load test configuration.

Creating a Test Environment

Before we get started writing automated tests we should look at where the tests should run once they have been created. A common mistake is to under estimate the challenge of setting up a working environment for the tests, which many times can cause the initiative around automated testing to come to a complete stop. So, before we start writing tests we should analyze the requirements for testing our product by asking the following questions:

- Which kinds of tests should we write?

- Where are tests designed and developed?

- Where should tests run?

- How do we manage updates to the system under test?

- How do we manage updates to the tests?

- How do we run the automated tests?

Going through these topics should help us understand what is needed for our test environment, which we can then design and set up to meet the requirements.

Typical Test Environment

For Kangreen we want to set up the test infrastructure to support the following requirements:

- Developers create automated unit and integration tests.

- Developers run tests on their local environment.

- Testers design manual tests.

- Testers run manual tests in a managed environment.

- Selected scenarios are converted to automated tests for regression and performance testing.

- Environments are created on demand using Visual Studio Lab Management.

- The system under test is deployed by the tester on demand using the TFS Build system.

Figure 22-2 shows how an environment can be setup to support these requirements. A Test Controller registers with a Team Foundation Server and is responsible for publishing test results and diagnostics data to TFS. The Test Controller also coordinates a test session so that the Test Agents run the tests and collect diagnostics data. The test environment is composed of different virtual machines managed by Visual Studio Lab Management. Finally, the TFS build system is used to create and deploy updates to the test environment, as well as to trigger automated testing on the test environment (implementing a so called build-deploy-test workflow).

Figure 22-2.  A typical test environment

Note  Microsoft Test Manager communicates bi-directionally with the Test Agent when an automated test is started from MTM. This is something to be aware of because it will require the MTM client to have direct access to the test machines, which might be against corporate policies.

Installing a Test Controller

The installation of a Test Controller for running automated tests is essentially a matter of connecting the controller with the TFS collection it will be working with. The Test Controller is installed from its own installer and separate from the TFS installation. We can also specify the account the Test Controller will run as. Figure 22-3 shows the Test Controller configuration options.

Figure 22-3.  Configuring the Test Controller

Note  The Test Controller has two modes; it can either be set up to work with automated tests in MTM or to control a Load Test process. If we want both, then we need to setup two controllers, They cannot be installed on the same machine, so we need to machines to enable both MTM integration and load testing.

An important part of the configuration of the Test Controller is how it handles security for the tests. The following Windows groups are created as part of the setup and need to be managed:

- TeamTestControllerUsers: Users in this group can start test runs and view test results.

- TeamTestControllerAdmins: Same as TeamTestControllerUser plus users in this group can also delete and pause test runs, manage test agents and create test environments.

- TeamTestAgentService: Users in this group can connect to a test controller.

Installing a Test Agent

Installing a Test Agent is also a simple process once the overall test environment design has been done (see Figure 22-4). The standalone Test Agent installer is found in the same agent distribution as the Test Controller. After the installation is complete we can configure the following aspects of the agent:

Figure 22-4.  Configuring the Test Agent

- Identity: We can run the Test Agent as Network Service or as a specific user.

- Associated Test Controller: The Test Agent registers with a Test Controller so we need to provide the name of the machine and the port the controller is listening on.

- Run Mode: The third option is to specify how tests are run. The agent can be setup in two modes:

- Service: Test Agent runs as a Windows Service and is used for non-interactive tests.

- Interactive Process: We can also configure the Test Agent to run interactively under a specific account, as well as to have the agent log on to the machine automatically or disable the screen saver.

The complete workflow to integrate automated tests with MTM and TFS can be distilled to the following checklist:

1. Design the test environment and install the necessary components.

2. Create a test plan and assign the environment to the plan.

3. Create test cases in MTM.

4. Create automated tests and associate the tests with test cases. Check in the tests to TFS.

5. Create automated test settings and associate with the test plan.

6. Create automated builds for the system under test, as well as the automated tests.

7. Assign a TFS build to the test plan.

8. Run tests in the environment.

In this chapter and the following chapters on lab management and build automation we will show how to use Visual Studio and Team Foundation Server to complete this process.

Traceability

Ideally we want to be able to know which requirement an automated test is testing. This can be achieved by using test cases and associating the test case with the automated tests as shown in Figure 22-5. By associating the test case with the automation code we have a mechanism to find where the test method is located. This is something that automation tools can take advantage of to automatically find the method to load when the test is to be run.

Figure 22-5.  Test case marked as Automated

Based on this we can then create a query-based suite in MTM to track (and run) all automated tests as shown in Figure 22-6.

Figure 22-6.  Creating a query-based suite in MTM for automated tests

We can also use the TFS Build system to improve traceability and visibility from the automated tests (in fact many features in TFS and MTM, such as Test Impact Analysis and Assign Build, require automatic tests and automatic builds to work). If we integrate our tests with a nightly build, we will get a heartbeat that every day gives us a signal about the quality from the system under test. See Chapters 26 and 27 on how to implement a build-deploy-test process, as well as details on how to automate data collection from test execution, code coverage, and code metrics.

Automating Test Cases Using Coded UI Tests

We begin our journey toward automated testing by looking at how we can create an automated test from a manual test with an associated action recording. The example test case is shown in Figure 22-7.

Figure 22-7.  Test steps in a test case

To create an automated test we need to create a test project in Visual Studio (see Figure 22-8). We can add any test project but because this is be a user interface test, the coded UI test project is the best starting point.

Figure 22-8.  Creating a new coded UI test

With the test project in place we can add a new UI test to the project. When the file is added we get three options as shown in Figure 22-9:

Figure 22-9.  Adding a new coded UI test from an action recording

- Record actions, edit UI map or add assertions: This option takes us to a recording mode where we can run a test and record test scripts using a test recorder. More on how to use this feature later in the chapter.

- Use an existing action recording: Choosing this lets us pick an existing action recording and convert the recording into a coded UI test.

- Cancel: Pressing Cancel exits the wizard and lets us start writing a new UI test directly, using the coded UI test API.

In this scenario we will choose the second option and create a test from an action recording. To locate the recording we use a work item filter (existing query, by id, or by using a filter) and pick the test from the result list (see Figure 22-10). Only test cases with an associated action recording can be added—the rest are shown with a stop sign.

Figure 22-10.  Selecting the action recording to generate a test from

Visual Studio now adds the test to the project and generates code from the action recording. The structure is actually quite nice if the action recording is created with the test in mind. By recording the test steps as distinct steps, the generated code will be very similar to the steps in the test case, which of course makes it easier to understand and maintain. Figure 22-11 shows the code generated from the action recording.

Figure 22-11.  Generated code from an action recording

With the code in place we can make sure it works by compiling and running the test. Coded UI tests run just as any unit test in the Test Explorer so you may want to refer to Chapter 16 about unit testing to have a look at how to use that. Figure 22-12 shows how the UI test was successfully run in the Test Explorer.

Figure 22-12.  Successful run of a coded UI test

The last thing to do is connect the test method with the test case. We typically do this for traceability, but it also lets us run the automated test as part of a test plan (in MTM or in a TFS build).

We connect the test case and the test method by opening the test case, browse all the tests in the current solution from the Associated Automation tab, and select the test method to associate (see Figure 22-13).

Figure 22-13.  Associating an automated test to a test case

Note  As you can see, it is only possible to connect one automated test to each test case. One way to work around this problem is to use another test type, for instance, the Ordered Test, and then group a number of tests into one and the map containing the test to the test case.

This was the quick tour of how to create a coded UI test based on a manual test case. In the next section we will extend the generated test to make it a complete automated test.

Coded UI Tests

We have seen how we can evolve from manual tests with action logs to fully automated UI tests. The generated tests are a good start, but to make them really useful we need to look at the generated code, refactor, and extend it so that it makes up a good test.

UI Test Architecture

The Coded UI test framework consists of a layer of components:

- Test Runner: The Test Runner from MTM can create action recordings from a manual test. The action recording can later be used to play back the test session in MTM or to generate a Coded UI test from the recorded steps.

- Coded UI Test: The implementation of the actual test written in C# or VB.NET. The Coded UI Test is either generated by the Coded UI Test builder code generator or coded directly against the Coded UI Test API.

- Test Recorder: Records actions from a test session, which then can be used by the Test Runner to automate a generated test or by the coded generator to create a Coded UI test.

- Coded UI Test API: The Coded UI Test API can be used to create UI tests which are more tightly connected to the UI elements

- Technology Managers. These are the core components responsible for mapping UI elements to the underlying technology (i.e. web browser or a rich client). We can implement custom plug-ins to support platforms which the native plug-ins does not support.

The complete architecture is shown in Figure 22-14.

Figure 22-14.  Architecture of Coded UI Tests

Supported Platforms

As we can see from the architecture overview, the Coded UI tests require support from the platform being tested. For Visual Studio 2012 the following platforms are supported for Coded UI tests as well as action recordings:

- Internet Explorer 8, 9, 10.

- Windows Forms 2.0 and later

- WPF 3.5 and later

- Windows Win32 (not officially supported but may work)

- MFC (partially)

- SharePoint (partially)

- Dynamics CRM

- Citrix/Terminal Services (partially)

- PowerBuilder (partially)

As you can see from the list many platforms are supported in Visual Studio 2012 but many also have only partial support so we recommend you try for yourself what works with your application.

Elements of a Coded UI Test

A coded UI test consists of four core elements created by the test framework:

- UIMap.Designer.cs: This file contains the .NET code generated from the Coded UI Test builder. The file is auto-generated and will be re-created whenever the test is changed. The core implementation class, UIMap, is marked as partial to allow extensibility.

- UIMap.cs: This file contains an empty partial class for UIMap and this is where we make the extensions to the test when we want to.

- CodedUITest.cs: This is the implementation of the test method for the generated test, the content is generated when file created but will not be changed when we make changes to the test. The name of the class is generated when the test is added to the project but can of course be changed like any other class.

- UIMap.uitest: This is a XML file that defines the test elements and is used to create the auto-generated files. The UIMap.uitest can be edited using the UI map editor.

Using the Coded UI Test Builder

Creating a test from an action recording was easy but what about if we were to start from Visual Studio instead? The best way to get started with coded UI tests is to create some tests using the Coded UI Test Builder.

When we add a new coded UI test to our project we choose the option to “Create actions, edit UI map or add assertions.” We can also start the Coded UI Test Builder from the Unit Test menu in Visual Studio or by right-clicking in the test method we want to extend with more test steps (see Figure 22-15).

Figure 22-15.  The Coded UI Test Builder

From the Test Builder we can create new recordings, add assertions, and generate code for recorded steps. We can start and stop the recording to get only the intended steps recorded.

To generate a coded test with the recorder:

1. Start the Coded UI Test Builder.

2. Press the Record icon.

3. Perform action to the application under test.

4. To view the recorded actions press the Show Recorded Steps icon. We can remove actions we don’t want directly from the list.

5. To finish recording and generate a test press the Generate Code icon and give a name to the test method being generated.

The UI controls found when recording the test are added to the UIMap file together with relevant information from the test run. If we want to change the test case later, we can edit the UIMap using the UI Map editor as described later on in this chapter.

Validation

An automated test would not be complete unless we have a way to assert that the outcome of the test is what we expected. To do this for a Coded UI test is essentially no different from how we make these assertions in any MSTest test, we simply add an Assert statement to the test code. The code to make the assertion can be crafted by hand using the Coded UI test API or be generated from the Coded UI Test builder.

To add a validation point using the recorder:

1. Start the Coded UI Test Builder.

2. Run the application under test to a position where you want to add the validation.

3. Click the crosshair and drag it over the application to locate the control. If the control is difficult to locate, then you can try using the navigation control in the Add Assertion dialog.

4. After locating the control to validate, the Add Assertions dialog is shown (see Figure 22-16) and you can select the property to validate.

Figure 22-16.  Add Assertions dialog

To completely match the manual test case we should add validation points to the automated test to check whether Dave has actually been logged on, as well as assert that the new expense report has been added to the list of pending expense reports. Figure 22-17 shows the result of adding an assertion step to the UI test.

Figure 22-17.  The complete test case with automated validation of test steps

Using the UI Map Editor

Creating the test was pretty easy; the recorders did a good job to help us get the test in place. But quite often the test needs to change—how can we keep up with this? Well, we can of course choose to remove the test and record it again. Sometimes that would be the best option but many times we just need to make small changes. Adding more steps or adding validations is easy to do using the Coded UI Test Builder. Fortunately for us we can also change the existing steps in a designer rather than editing the XML file by hand (remember we cannot edit the generated UIMap code because it will be replaced as soon as we make changes to the test).

To edit the test case, simply double-click on the UIMap.uitest file and the test code is shown in the UIMap Editor (see Figure 22-18).

Figure 22-18.  The UI Map Editor

In the UI Editor we see UI actions to the left and the UI control map to the right. From these lists we can do things such as:

- Rename items.

- Remove items.

- Split lengthy sections of generated script into smaller methods.

- Insert delays in the script.

- Change properties on items.

Data-Driven Automated Tests

A common requirement of automated tests is to be able to run the test a number of times with different parameters. In Chapter 16 we looked at how we can work with data-driven test in details. The same techniques are valid for coded UI tests as well except that the data source for a UI test often points to the data set managed in the associated test case. Using the test case to manage test data for the automated test is nice because then the tester can update the data directly from inside MTM.

Using Unit Tests as Automated Tests

Coded UI tests are probably the ideal type of test for regression testing because they give us the opportunity to automatically test the system just like a real user would. But there are a number of issues that make UI tests difficult as a general purpose solution for automation solution, including

- Requires a UI.

- Requires a desktop session to run.

- UI tests are more complex to design and maintain.

- Difficult to scale.

To work around these challenges we generally recommend considering “under-the-UI” tests as a complement to the UI tests. These tests could be written against the APIs in the system or against the controller layer in an MVC style of application.

A good approach is to use different types of tests for different purposes. Remember the test quadrant from Chapter 20. Some functional tests can be implemented as UI tests, but most would probably do just fine if we implement them as unit tests. Integration and component tests are by definition tests that run under the UI so here unit tests works great too. And for the abilities-tests, such as performance and security tests, we can write unit tests as drivers to these test frameworks.

See Chapter 16 on unit testing for more information about creating and running automated tests based on a unit test framework.

Running Automated Tests

We looked initially at how we can run an automated test within Visual Studio. While this is a great way to run tests in general and during development in particular, it is not the best way for any user or process. For a general tester it would be best to run the tests from inside MTM, for trend analysis it would be best to run the tests from the build process and for build verification tests it may be most suitable to run the tests from a command-line tool.

Of course, Visual Studio ALM gives us the option to choose here as well, so let’s take a look at the options.

Integration with TFS build impacts the automated tests in at least two ways; to get the most value out of the tests we want to run them continuously, which the build system can do of course. The other scenario is when we want to run the automated tests from Microsoft Test Manager.

For MTM to locate the components containing the automated tests, we need to provide a TFS build that produces the test components as part of a build, we can set this for all tests in a test plan (see Figure 22-19) or when starting the test run by choosing the build from Run with Options.

Figure 22-19.  Assigning a build to the test plan

Running Automated Tests from MTM

If we have set up the test environment as discussed initially, we can now simply run the test in MTM again. By default the test will run automated with the test plan settings. Figure 22-20 shows how we can set the build to use, as well as the test settings and test environment for the automated test runs.

Figure 22-20.  Test plan settings for automated testing

If we want to use specific settings for a test run, we can use the Run with Options dialog and kick off the test as shown in Figure 22-21.

Figure 22-21.  Starting an automated test in MTM

After the test has been started, we can watch the progress in MTM from the test run summary view as shown in Figure 22-22. From this view we can also stop the test if we need to.

Figure 22-22.  Watching an automated test run in MTM

We can also look at the progress, for instance to troubleshoot, by clicking on the “Test run log” link (see Figure 22-23).

Figure 22-23.  Looking at the test run log in MTM

Running Automated Tests from TFS Build

Integrating automated tests with TFS Build can be both simple and complicated. Running unit tests as part of a build is just a matter of setting the right properties in a standard build definition. But for tests that can or should not run on the build server we need to start the tests on a different machine. We also need to install the system under test and the test components on a remote machine for them to run. This means we need to deal with environments and complete end-to-end builds. If we can leverage the Visual Studio Lab Management tools, we get most of this out of the box if not, then we need to figure out ways to solve this in our process.

See Chapter 26 for more details on build in general and 27 on how to implement the build-deploy-test workflows.

Running Automated Tests from the Command Line

There are two ways to run automated tests created in Visual Studio, using MSTest.exe to invoke the test methods from the command-line or by using TCM.exe to run automated tests associated with Test Cases in TFS.

Running tests with MSTest

Running tests with MSTest is very straightforward. By default MSTest is installed under

C:\Program Files (x86)\Microsoft Visual Studio 11.0\Common7\IDE\MSTest.exe

And to run any automated test we only need to point it to the assembly containing the tests

Mstest /testcontainer:ER.Web.UITests.dll

We can also add additional switches to filter tests to run, provide test settings, or publish the result to TFS. Figure 22-24 shows an example of a successful test run.

Figure 22-24.  Running a test from the command-line using MSTest

Running Tests from Test Cases

Another option for running tests from the command-line is to use the Test Case Management command-line tool, TCM.exe. This tool is installed under

C:\Program Files (x86)\Microsoft Visual Studio 11.0\Common7\IDE\TCM.exe

There are several options for us to start a test run, for instance:

- Run all tests in a test suite. Locate the Suite Id from the test suite in MTM.
tcm run /create /title:"UI Test Run" /planid:5 /suiteid:11 /configid:2 /

collection:http://tfs.kangreen.com:8080/tfs /teamproject:"Expense Reporting"

You can use tcm.exe to figure out the Ids for plans, suites, and configurations as well:

- List plans
tcm plans /list /collection:http://tfs.kangreen.com:8080/tfs

/teamproject:"Expense Reporting"

- List suites
Tcm suites /list /collection:http://tfs.kangreen.com:8080/tfs /teamproject:"Expense Reporting"

- List configurations
tcm configs /list /collection:http://tfs.kangreen.com:8080/tfs /teamproject:"Expense Reporting"

Note   The plan id, suite id, etc. are also shown in MTM under the corresponding pages, but this is probably not something you noted unless you actively looked for it.

Figure 22-25 shows the output from creating a test run using tcm.exe.

Figure 22-25.  Running a test from the command-line using TCM.exe

After the test has been started we can view the result in MTM just like we have done with manual tests.

Performance Testing the Application

So far we have looked at how to create automated tests mainly for feature validation. Using the same framework and tools we can easily move to performance testing our application as well.

Writing good load tests can be a challenge just like any other test so it will require good design work to get value out of the load tests. Often writing the tests are the easy part, analyzing the results is the real challenge. To understand the outcome of load testing typically requires testers and developers to work closely together to figure out what is causing bottlenecks or other errors to occur.

In this final section we will look at how we can create a test session using Web Performance Test and then run those tests using the Load Test tools in Visual Studio. The Web Performance Tests are a good way to get started when performance testing web application but for general purpose load testing we can use any test implemented using the unit test framework in Visual Studio.

Creating Load Test Scenarios Using Web Performance Tests

First, we need to create a test scenario. For this purpose we can use the Web Performance Test type in Visual Studio. This test type lets us create a test by using Internet Explorer and record a test session. The test script that gets generated can then be customized similar to the Coded UI tests we looked at earlier in the chapter.

To create a Web Performance Test we can start by adding a Web Performance and Load Test project to our solution. Next, we add a new Web Performance Test which opens up an Internet Explorer session with the Web Test Recorder add-in shown. We then use the recorder to capture events as we navigate in the application we are testing as shown in Figure 22-26.

Figure 22-26.  Using the Web Test Recorder to create a Web Performance Test

When we complete the recording we get back into Visual Studio and there we can use the Web Performance Test designer to enhance the recording (figure 22-27).

Figure 22-27.  Web Performance Test designer in Visual Studio

In the designer we can design quite advanced tests without writing any code, for instance we can

- Add recording.

- Add data source.

- Generate code.

- Set request details. This is a nice feature where we can configure think time as response time goals (see Figure 22-28).

Figure 22-28.  Request Details configuration

- Add validation rule. This is also a pretty advanced dialog where we can add assertions to the test, including field content validation and response time validation (see Figure 22-29).

Figure 22-29.  Add validation rule for Web Performance Test

We can also add behavior logic to the test, such as

- Insert request.

- Insert Transaction. A Transaction is often used to mark a sequence of events which we can track during test execution; this is useful for us in order to track sections in a test like the time to login or to do a search.

- Insert Loop.

- Insert Call to Web Test. We can structure complex scenarios by combining simpler tests into a larger flow.

- Add parameters.

- Extract Web Test from steps. This is a convenient way to refactor our tests. Just select a number of steps and extract, they will be added as a new test and a call to it added to the test automatically.

When we want to run the test we can use the same tools as for any other automated test in Visual Studio. In the test run window shown in Figure 22-30 each recorded step is shown as the result, including interesting metrics such as response codes, response times, and bytes sent and received.

Figure 22-30.  Execution result from a Web Performance Test

Setting Up a Load Test

The next step when we have created the core session tests is to create a load test to orchestrate the performance session. Creating a load test can be a complicated thing to set up in most tools, but in Visual Studio it is as easy as adding a new Load Test to our test project.

The wizard (see Figure 22-31) walks us through the process of defining the essential steps for our load test:

Figure 22-31.  The new Load Test Wizard

- Scenario: First, we define the overall behavior of the test, including think times and time between iterations.

- Load pattern: Next, we can specify the type of load pattern to use for the test session. The most common pattern to start with is a step load with a low number of initial users and to add one user at a time. After we have seen the test working we can customize the pattern to more realistic values.

- Test mix pattern: The test mix defines which tests are run by the virtual users.

- Test mix: This is the heart of the test where we add the automated session tests to the load test. If we can several types running we can also specify the ratio of each type of test (see Figure 22-32).

Figure 22-32.  Adding tests to the Load Test scenarios

- Network mix: Used to simulate different network types.

- Browser mix: Used to simulate different browser types.

- Counter set: In this page we can configure which measurements to track during test execution on each of the machines in the test environment.

- Run Settings: Finally, we get to specify the load test duration or number of test iterations to run.

Walking through the wizard then gives us a complete load test to start running as Figure 22-33 shows. Typically we would execute it pretty much as it is to see whether it works and then change it carefully to optimize the test. Just as with the Web Performance Test, we use a Visual Studio designer to configure the load test with additional properties.

Figure 22-33.  The Load Test designer in Visual Studio

Running a Load Test

Finally we can run the load test. When the test is run a progress report is shown (figure 22-34) where we can track the progress and analyze the result live. We can configure the view in a number of ways to follow the test process, for instance by adding counters, zoom in on particular machines, and so on.

Figure 22-34.  Running a load test from Visual Studio

When the test is complete, the result will be published to the load test store (if configured to do so) and we can go back and check earlier tests runs. By default, load tests create a repository on a local SQL Express instance. If running on a local machine, the user must run Visual Studio with administrative privileges to get access to the result store.

Summary

In this chapter we have looked at how we can move from manual testing to automated testing by evolving the test cases. With a structured approach we can save lots of work by reusing the knowledge we can build up around the tests and we can focus on getting the right tests automated.

First we looked at how test cases in MTM can be automated just by creating a Coded UI test off the action log associated with the test case. The UI tests can then be extended with additional test logic as well as validation steps, either by using the test recorder or by using the test APIs directly.

Even though UI testing is the most realistic automated test, it has some issues; in particular UI tests require more infrastructure to run, they are more fragile, and can be more costly to maintain. To get around these issues we can create tests at the layers under the UI by coding against the application model or APIs instead. Tests coded this way are easier to aintain and work great for testing nonfunctional requirements and can be used for stress- and load test scenarios.

We can also create load tests from Visual Studio. It is actually easy to get started—a load test is just another test type in Visual Studio. We can even reuse existing automated tests and run them in a load test scenario if it is suitable. When we need more advanced load tests, we can scale out the solution to a distributed load test rig using the same tools and tests which make it very efficient to run both small scale and large scale load tests.

Next up we are going to look at how and where tests are run. With MTM and Visual Studio we can improve the way we do testing but to be efficient we also need to have good practices our lab.

CHAPTER 23

Visual Studio Lab Management

Most projects today want short cycles and transparent development processes that allow for changing the plan after every cycle. This typically means that in two to four weeks a set of features need to go from developers to testers to stakeholders and perhaps even customers. To support this we need lots of different test environments to validate the development work. There is also a lot of work to keep the environment ready for test, which includes not only making sure the right version of the product is there but includes test data and dependent services as well. If we want to succeed with agile projects we must make sure to manage this complexity well. Unfortunately, it is often far from a simple process to implement in a project though.

Visual Studio Lab Management was introduced with TFS 2010 as a platform to build a lab infrastructure to support these new requirements. It has many features that can be used to speed up the process around managing environments. As a nice consequence it also improves the overall experience around testing and bug fixing with tooling for automated deployment and testing as well as collection of data from machines in the lab environments.

This section covers how to leverage Visual Studio Lab Management to implement a solution for test lab management, which can be used for manual testing, as well as automated processes such as deployment and testing. This will take our software factory to the next level!

Architecture

To set up Visual Studio Lab Management requires knowing more about the features it offers. In the next section we will look at the capabilities and components in the product that will enable us to define the architecture for a Lab Management configuration.

Capabilities

First let’s take a look at the capabilities Visual Studio Lab Management offers. Lab Management is all about making testing easier. We can use Lab Management to create labs to run manual and automated tests. Lab Management also extends the TFS build system with deployment and remote testing capabilities.

Improved Manual Testing

Perhaps the most tangible capability of Visual Studio Lab Management is doing manual testing more effectively. With Lab Management and virtualization is it now possible to create lab environments to allow for testing of complex test systems (database and web servers, integration services) as well as testing multiple configurations (operating systems, platforms, languages).

Environments for Automated Testing

It is also possible to use lab environments to run automated tests. The lab environment is then controlled by the Test Agent installed on machines in the environment. How tests should be executed on the target environment is controlled by settings in MTM and Visual Studio. The Test Agent can be configured to run both interactive and non-interactive tests.

Framework for Build-Deploy-Test Workflows

As mentioned at the beginning of the chapter, one of the key elements of an effective test process is to manage environments effectively. Apart from creating and using the environments, implementing automated build-deploy-test (BDT) workflows that can virtually rebuild a complete environment in minutes is a killer feature of Lab Management. BDT workflows is implemented using the standard build definitions in TFS Build and can be customized and extended to support the deployment needs of your application.

Cloning Environments

Working with complex environment can be challenging work. How many times have we had to struggle to get a test environment with all components set up? Wouldn’t it have been great if it was possible to take a copy of the environment when you have got it working and reuse it for future testing? Lab Management environments can do just that, if we want to create a copy of an environment, we can create a template and then use that to create clones for other testers, other configurations, or whatever our needs may be.

Of course cloning machines can have side-effects, particularly when it comes to starting up several instances of the same type of environment. The solution for this is to use network isolation to create isolated environments.

Network Isolation

To solve the problem with cloned environments, Lab Management allows us to create isolated environments. An isolated environment is simply put a lab environment with managed networking. When an environment is created with network isolation the machines in the environment are placed on a private subnet so that additional running environments with conflicting components can work independent of each other. Figure 23-1 shows an example where an original environment has been cloned. Using network isolation the internal addresses (and all other configuration data) are identical but the external addresses are unique.

Figure 23-1.  Isolated environments

Components

Now with the main capabilities of Lab Management covered, let us take a look at the components used to support those capabilities.

Test Controller

The Test Controller is a service responsible for controlling test execution. It can be used for controlling load tests, which we looked at in the Automated Testing chapter, and also for managing automated test runs, which is what we use it for in lab environments. Each test controller used with Lab Management is bound to a TFS Team Project Collection but can be used by any project in that project collection.

Test Agent

A test agent is a service installed on a test machine and enables three things:

- Test execution. Tests run on the machine are managed by the test agent (including things such as downloading test assemblies and test dependencies)

- Data collection. The test agent collects data as configured for a test run.

- Deployment workflow execution. If a machine is part of a build-deploy-test workflow the test agent will execute the deployment part of the workflow.

With Visual Studio 2012 Lab Management the test agent deploys automatically when a new lab environment is created from Microsoft Test Manager. A test agent is always connected to a test controller.

The test agent can be configured to run tests interactively (typically for running UI tests) or non-interactively as needed. If possible, you should run the test agent as a service to avoid having the test machine running with a logged on user.

System Center Virtual Machine Manager

System Center Virtual Machine Manager (SCVMM) is the component required if we want to create new virtual machines for lab environments directly from MTM.

The SCVMM client also has to be installed on the Team Foundation Server Application Tier to enable the SCVMM Lab Management functionality.

Lab Management 2012 supports the 2008 R2 and 2012 versions of SCVMM.

Architecture

With the preceding in place we can now take a look at a complete architecture for Lab Management. Figure 23-2 shows the components and its use cases.

Figure 23-2.  Lab Management architecture

Note  For more details on how to design your lab see the “Planning Your Lab” on MSDN (http://msdn.microsoft.com/en-us/library/ff756575(v=vs.110).aspx).

Setting Up Visual Studio Lab Management for Kangreen

We will now look at building the Lab Management infrastructure for Kangreen to empower their testing process. Based on the understanding of the capabilities in Lab Management, Kangreen does not want to miss any part of this product and has asked for the setup shown in Figure 23-3.

Figure 23-3.  Kangreen Lab Environment topology

Install and Configure Test Controller

Let’s start by installing the Test Controller. The Test Controller, as we talked about earlier in this chapter, is responsible for integrating a lab environment with TFS. The installation in itself is a straightforward process; the most important thing to consider is where the Test Controller should be installed. If you only need one Test Controller, then you could install it on the same machine as the Build Controller (if you have one).

Once installed, the Test Controller must be configured to register with a Team Project Collection, as shown in Figure 23-4.

Figure 23-4.  Test Controller configuration

Install and Configure Hyper-V and SCVMM

To support snapshots and the creation of new lab environments from templates, the next task is to enable Hyper-V and install the System Center Virtual Machine Manager (SCVMM) in the TFS environment. The high-level installation procedure is as follows:

1. Enable the Hyper-V role on the virtualization server.

2. Install and configure the SCVMM server and the SCVMM administration console on the virtualization server.

Note  We recommend using a separate host group for each team project collection. We also recommend configure a library share for each team project collection since it gives a better control over which VMs and templates are used by which collection.

- 3.  Install the SCVMM client on the TFS application tier. This enables the integration between TFS and SCVMM.

For a detailed description on how to set up Lab Management, look at the product documentation on configuring and administrating Lab Management[1].

Configure TFS for Lab Management

After SCVMM has been set up we can now configure TFS to enable the Lab Management capabilities for a Team Project Collection and its Team Projects. To enable Lab Management do the following:

1. Enable Lab Management for the Team Foundation Server (see Figure 23-5).

Figure 23-5.  TFS Lab Management configuration

a.  Open the TFS Administration Console and go to the Lab Management tab.

b.  Provide the name of the SCVMM server.

c.  Provide credentials on SCVMM used to provision the TFS service account to the SCVMM Administrators role.

Note  The credentials provided must be a member of the SCVMM Administrators role.

d.  Provide information about network isolation.

The TFS service account needs to have permission to register the DNS alias in the DNS zone provided.

- 2.  Enable Lab Management for each Team Project Collection (see Figure 23-6).

Figure 23-6.  TFS Lab Management Team Project Collection configuration

a.  Open the TFS Administration Console, select the Team Project Collection to configure and choose the Lab Management tab.

b.  Provide the library shares and host groups to use for the team project.

c.  Provide a service account.

Note  This account must be a domain account and should have limited privileges because it can be used to run build workflows on the test and build agents.

Environments

A Lab Management environment is a collection of properties that defines a lab environment by

- Name and type of environment.

- Host group for deployed environments.

- The machines in the lab environment and their role (client, app server, database server, etc.). The role can be used to select machines in scenarios such as deployment or test settings.

- Machine properties. Characteristics for the lab machine including tags. The tags are a convenient way to select machines in, again for instance in a deployment scenario.

Designing Environments

With the understanding of capabilies and the installation requirements we can now start thinking on how the environments should be created. Typical points for discussion when looking at types of environments are things like:

- What defines an environment? One machine? A group of machines?

- Are physical machines required for testing or can virtual machines be used?

- Should environments be possible to revert to an earlier state?

- If yes, we need to use SCVMM (or at least virtual) environments.

- Is the database inside the environment?

- If no, then how can we control state if an environment is reverted?

- Should environments be cloned and run concurrently?

- If yes then we should use isolated environments (requires SCVMM environments).

Ideally you want to simulate the different types of environments needed to perform relevant tests, which often means a set of Dev, Test, QA, and Pre-production environments per product version you need to test. Each lab environment contains the machines representing the system under test (which can be a single machine or a complete infrastructure setup including clients, application servers, database servers, firewalls, and so on). Then factor in the need for different operation systems, languages and browser versions. This can quickly build up to a good number of environments and machines to maintain.

This is just what Lab Management is here to help us with! It should not be impossible to manage the need for realistic testing, instead focusing on defining repeatable processes for setting up environments and deploying updates of your product to environments.

Working with Standard Environments

With Visual Studio 2012 Lab Management it is possible to create an environment based on existing infrastructure without installing SCVMM. This is great because we can now start using Lab Management without any additional installation required. The lab machines in a standard environment can be physical or virtual (or both) and the virtual machine does not have to be managed by SCVMM or Hyper-V.

To create a standard environment you only need to know the names of the machine you want to add to the environment and a user account to be used to log in and deploy the test agent service onto the machine.

A standard environment can be connected to directly from MTM and the Test Runner making the integration with lab environments really smooth.

SCVMM Environments

The alternative to standard environments is to use a SCVMM environment. The key features of a SCVMM environment include

- Create new machines from templates. New lab machines can automatically be created by anyone (having permissions) running MTM. It is no longer required to send a request to the operations department to get machines created for a new environment or to do other maintenance operations such as starting, stopping, snapshotting, or restoring machines.

- Support for snapshots. It is possible to work with snapshots from MTM, which enable testers to save state during testing. This is a powerful feature both to unlock the tester if a test case fails (no need to block an environment while waiting for a resolution) but also to give a developer a fixed point in time to reproduce a bug.

- Connection inside MTM. A SCVMM environment can be connected to MTM and the Test Runner just like with the standard environment.

Note  Taking a snapshot on an environment snapshots all machines in the environment at the same time, a great way to manage state for a complete test scenario!

What About Support for VMware?

At this point there is no built-in support for managing virtual environments using VMware the same way as virtual machines in Hyper-V. You can of course create standard environments based on machines managed by VMware but you will not get support for snapshots and direct integration with machines in the environment from inside MTM.

Integration with the build system works with VMware environments just like for any physical environment. If you want to get more integration with VMware, check out the following open-source project: http://vmwarelab.codeplex.com. This project adds additional build workflow activities that allow you to use snapshots as part of a deployment process and you use a custom build process wizard to define the workflow using VMware, as shown in Figure 23-7.

Figure 23-7.  Configure a VMware environment for build-deploy-test

Test Settings for Lab Environments

As we saw in Chapter 21 we can use a Test Settings configuration to control how data is collected from an environment when tests are run. Test Settings for a lab environment works exactly the same and lets us add data collection for the remote machines in our environments, Figure 23-8 shows how this is configured for a test plan.

Figure 23-8.  Test settings for environment

Creating Environments for Kangreen

Kangreen is now ready to start creating environments for the Expense Reporting project. Initially, environments to support the following are requested:

- Manual testing on laptops

- Manual testing on different browsers and operating system versions

- Automated testing

- Automated deployment

Create Standard Environment for a Physical Machine

Let’s start with a simple scenario and create a standard environment for an existing machine. A standard environment can be used to group any physical or virtual machine and enables the build and test feature of Lab Management for the provided machines. We will use the Lab Center and Lab view in MTM to define our environments. Simply click New Environment and choose Standard environment (see Figure 23-9).

Figure 23-9.  Creating a standard environment for a physical machine

Next add existing machines to the environment. We need to provide the name of the machine and an account to be used to log in and install the test agent on the machine as shown in Figure 23-10.

Figure 23-10.  Adding an existing machine to a standard environment

We use the additional tabs to provide more details about the environment as described in the beginning of this section. Clicking Verify checks the provided information and when we click Finish MTM sets up the environment. This may take a few minutes as the test agent is automatically deployed to each machine in the environment by the environment creation process. If the machines in the environment already have the test agent installed it will only be verified and the creation process will go faster.

Create Template in SCVMM

So now we have a standard environment to use for testing on existing machines. Our next challenge is to create environments for testing different configurations. A good approach for managing scenarios, including many machines, is to create templates for each type of machine and then use the templates to create new environments. First let’s look at the steps to create a template in SCVMM that we later can use to create SCVMM environments from inside MTM.

1. Start the SCVMM administration console and go to the virtual machines tab.

2. Create a new virtual machine for the template.

a.  We recommend a naming convention that shows that this is a master image for a VM template, for example, use the form [OS] [Architecture] Master, i.e. “Windows 7 x64 Master.”

- 3.  Install Windows and all the prerequisites for the template.

- 4.  Make a clone of the VM that will be used to create the template.

Note  It is possible to use the created VM as-is to create a template but the recommendation is to clone the VM and create a template from the clone. This extra step allows us to maintain the master and easily create a new template as the requirements for the template changes.

- 5.  Create a template from the cloned VM.

a.  Same recommendation goes for the template—try to include the name of the master VM for correlation, for example [VM] Template, i.e. “Windows 7 x64 Template.”

Note  We are only scratching the surface of what you can do with SCVMM when it comes to creating templates and automating the customization of the machines created from templates.

Create SCVMM Environment

Now when we have created a template in SCVMM the process of creating environments for use by Lab Management is very straightforward, we more or less pick and choose from the library and let SCVMM/Hyper-V generate our new machines. To create a new environment from the Lab Center in MTM based on SCVMM we use the same wizard as for creating a standard environment. Selecting to create a new SCVMM environment changes the behavior of the wizard to use templates from SCVMM when building up the environment (see Figure 23-11).

Figure 23-11.  Create new SCVMM environment

Next on the machines tab we can now choose from the available templates (see Figure 23-12).

Figure 23-12.  Add machines to SCVMM environment

The rest of the settings work the same as for a standard environment.

Cloning Environments

Earlier we talked about how we can use Lab Management to quickly create new environments. With SCVMM templates the time it takes to create a new environment is more or less the time it takes to copy the VM templates and customize them. Unfortunately what often does take time is to get the machines configured with application configuration (installing third-party software, application configurations, and so on). If we just want to copy a working environment, for instance to create labs for different testers or to test different versions of a product, we can further speed up the lab management process but cloning complete environments and deploying the clone as a new environment.

Clone Environment

To clone an existing environment do the following:

1. Go to the Lab Center in MTM.

2. Select the environment to clone and click Store in Library.

3. Give the environment a name, a description, and a location on SCVMM and click Store as shown in Figure 23-13.

4. The environment is now copied to the SCVMM library share.

Figure 23-13.  Store environment in library

Later when you want to deploy a stored environment:

1. Go to the Lab Center in MTM.

2. Select Deploy and pick a stored environment.

3. Give the new cloned environment a new name, a description, and select the location in SCVMM for the new environment (see Figure 23-14).

4. Click Deploy environment to start the deployment process.

5. The environment is created on the provided host.

Figure 23-14.  Deploy environment from library

Note  To run multiple cloned environments we need to handle how the machines in the started environments work. The best way to do this is to use network isolation so that each copy of an environment runs virtually separated from each other.

Using Lab Management for Manual Testing

Now with the model for the test labs in place we can immediately take advantage of this and start using the labs for manual testing. This is great because it is important to show the effect of any change quickly to get confidence and support from the stakeholders.

Manual testers can take advantage of the lab infrastructure in several ways:

- Connect to an environment

- Collect data from environment

- Start and stop environment for test (SCVMM environments)

- Snapshot environment (SCVMM environments)

So why waste time? Let’s go testing!

Selecting an Environment to Use

The first step is to find the environment to use and make sure it’s available for testing. A virtual machine can be stopped and need to be started or an environment can be used by someone else running tests on it. Fortunately MTM will help us get control of this so we can pick our environment and start testing. For the Lab Center in MTM you can manage the environments and use the “In use” marker to tag an environment as in use as shown in Figure 23-15.

Figure 23-15.  Setting the In Use marker

Connecting to an Environment for Testing

After we have made sure the environment is started and reserved for us, it is time to run some tests on it. Select “Run with Options” when starting a test run and select the environment to use (see Figure 23-16).

Figure 23-16.  Select environment for running tests

Next if we want to run tests on a machine in the environment (and not only collect data) we can easily connect to the environment from the Test Runner. This opens up the Environment Viewer window where all machines in the environment are listed and you can quickly connect to any machine in the environment (see Figure 23-17).

Figure 23-17.  Using the Environment Viewer from the Test Runner

Note  The Environment Viewer uses the Remote Desktop protocol to connect to the environment so the port for RDP must be open (default 3389).

By default the owner of the environment connects using a host-based connection; others use guest-based connections. A host-based connection is routed using hyper-v virtual machine connection protocol, which will let you interact with the machine even when it does not have an IP address. This can be very valuable if you need to do low-level tasks, such as install the operating system.

You can also right-click on the machine in the environment and connect using the standard Remote Desktop client. This is typically something you want to do if you want to bring local resources to the test machines (such as printers, disks, or USB devices) or to use the clipboard to copy and paste stuff between machines.

It is also possible to set the default environment for manual testing on test plan as Figure 23-18 shows.

Figure 23-18.  Setting the Test Environment for manual runs on the Test Plan

Manual testing can be performed both outside and inside an environment. Testing from the outside is often the choice for applications where the client can be run on the local machine and simply connects to sites and services in a test environment. If the client should be run inside the environment, then you many want to install and run MTM on the machine where the test is run, typically for the possibility to create an action recording.

Snapshot Test in Lab

For a SCVMM lab environment we can take a snapshot of the complete environment. This is a great way to set a label on a test environment as a given point in time and allows us to do things like

- Continue working on other test cases in the environment and still have the context left when you want to go back to a paused test case.

- Pause testing and let someone else use the environment without messing up your test data.

- Associate the state of test with a bug report. A classic problem in bug reporting is that when a developer gets to look at a bug the environment in which it was found has been changed and it is no longer possible to reproduce the problem. With a snapshot the developer can revert to the snapshot when time comes to repro the problem.

When working with a SCVMM environment we can take a snapshot from the Environment Viewer (see Figure 23-19) or the test runner.

Figure 23-19.  Create an environment snapshot from the Environment Viewer

From the snapshot we can quickly move back to a previous state to continue testing or to analyze a cause of a problem. If a snapshot is taken from a test run, the link to the snapshot is automatically added to a reported bug that in turn can easily be accessed by a developer when starting a debug session as shown in Figure 23-20.

Figure 23-20.  Connecting to a snapshot from Visual Studio

Automated Testing and Lab Management

In the previous chapter on automated testing we looked at how automated tests can be associated with a test case. The connection between automation and test case allowed us to track how and where test cases have been automated. Figure 23-21 shows a test case with an associated automated test.

Figure 23-21.  Test case with associated automation

To run an automated test in a lab environment from MTM we must select the following when starting the test:

- A build. The build is used to find the automation component specified on the test case.

- A test settings definition. Test settings define properties for the test run, such as deployment items and data collectors.

- The target environment. This is where the tests should run.

Just as for a manual test run it is possible to define all this on the test plan as well, as shown in Figure 23-22.

Figure 23-22.  Setting the test environment for automated runs on the Test Plan

When executing automated tests in MTM we should pick out the environment the same way as for manual testing to make sure others know the environment is in use. This can be automated in build-deploy-test workflows by customizing the default lab workflow; we will look at how to do this in Chapter 27 when we look at automating deployment to lab environments.

Running Automated Tests from MTM

Now let’s look at running the automated tests created for the Expense Reporting application directly from MTM. It turns out that running an automated test is very similar to running a manual test, we just have a different view for monitoring the test progress (see Figure 23-23).

Figure 23-23.  Automated test execution

If we want to view the execution steps for the test run we can click to view the test run log (see Figure 23-24), this is a good place to start looking when troubleshooting failing tests.

Figure 23-24.  Automated test run log

Running Automated Tests as Part of a Build

A key scenario to support is of course running automated tests in a lab environment as part of a build-deploy-test workflow. Chapter 27, where we go through the automated deployment process in detail, shows how to set this up.

Running Automated Tests from the Command-line

A third option for starting automated tests is to run them from the command-line. This may sound like an edge-case but it actually quite nice to be able to start a test run without having to open up MTM. Tcm.exe is the tool to do this and we can use the same settings as in MTM when starting a test run:

tcm run /create

/title:title

/planid:id

/collection:teamprojectcollectionurl

/teamproject:project

(/suiteid:id /configid:configid | /querytext:query)

[/settingsname:name]

[/owner:owner]

[/builddir:directory]

[/testenvironment:name]

[/login:username,[password]]

[/include]

Which with our manual example would translate to:

tcm.exe run /create /title:"Automated BVT" /planid:2 /collection: http://tfs:8080/tfs/teamproject:"Expense Reporting" /suiteid:2 /configid:2

The required arguments are

- planid: is the id for the test plan to run tests from, the plan id is displayed under Organize, Test Plan Manager in MTM.

- collection: is the name of TFS project collection.

- teamproject: is the name of team project.

- suiteid: is the id of the test suite to run tests in, the suite id is displayed when selecting a suite in the Plan or Test views in MTM.

- configid: is the id of the test configuration to run tests on, we find this id on the Organize, Test Configuration Manager tab in MTM.

An example of a running test is shown in Figure 23-25.

Figure 23-25.  Automated test running

Summary

In this chapter we have looked at how Visual Studio Lab Management can be used to enable a more efficient test process. Lab Management makes it possible to dynamically create test environments for manual and automated testing. The agents in the infrastructure also simplify data collection during testing which helps developers troubleshoot reported bugs. With the use of SCVMM and virtualization a tester using Lab Management can create snapshots of the system under test, making it possible to easily save its state to be used later for debugging or to run tests repeatedly off the same baseline.

In Chapter 27 we will further explore the capabilities of Lab Management when we look at implementing a complete automated build-deploy-test workflow.

[1] http://msdn.microsoft.com/en-us/library/dd936084(v=vs.110).aspx

CHAPTER 24

Metrics and ALM Assessment for Software Testing

Software testing is an important area. Testing should be a constant part of any development and not only a phase at the end of the project. Microsoft’s online ALM assessment can help us to greatly improve our testing practices. There are also good metrics we can use during our projects to make sure we have high-quality testing in place.

Metrics

There are a number of metrics to can use as KPIs for software testing.

- Number of bugs per state. How many bugs do we have that are active, resolved, or closed? Are the number of active bugs increasing and number of resolved and closed bugs constant? Then we need to look into how we perform our testing.

- Number of bugs sent back from testers for more information. A large number might indicate that communication between developers and testers must improve.

- Code coverage. This shows us how much of the code has been covered by our automated unit tests. We get the value as a percentage of the whole code.

- Tests run results. How are our tests performing? Do we have many failed tests? If so, what can we do to improve this?

- Percent Requirements covered by test cases. Do we write test cases for all our requirements? If not, what is the reason?

- Percent Requirements covered by testing. Do we actually run the tests we have test cases for? If this figure is low and the figure for Percent Requirements covered by test cases is high we might have an issue we need to deal with.

Standard Reports

The metrics we get in our report concerning testing can be really helpful in our projects. The reports described here are only available in the two MSF process templates and not in the Scrum template. Let’s take a brief look at the reports:

- Bug status (Agile, CMMI)

- Reactivations (Agile, CMMI)

- Bug trend (Agile, CMMI)

- Test Case Readiness (all)

- Test Plan Progress (all)

Bug Status Report

The first covered is the Bug status report. This report gives us information about the cumulative bug count based on bug state, priority, who it is assigned to, and of course, the bug severity. We see the number of bugs and the number of resolved bugs (see Figure 24-1).

Figure 24-1.  Bug status report

Reactivations Report

The Reactivations report (see Figure 24-2) is used to see how many bugs have been resolved or closed too early. If the bug needs to be opened again it is called a reactivation. A high number indicates that the developers can improve their bug fixing process and not close or resolve the bugs unless they really are ready to be closed.

Figure 24-2.  Reactivations report

Bug Trend Report

Next is the Bug trend report (see Figure 24-3). This report helps us track the rate at which our team is finding, resolving, and closing bugs.

Figure 24-3.  Bug trend report

Test Case Readiness Report

A test case can have two states; Design or Ready. When the test case is created the state is set to Design and after the team has reviewed and approved the test case, it is set to Ready. The Test Case Readiness report (see Figure 24-4) provides an area graph that shows how many test cases are in the Design or Ready state over a time period that we specify. By reviewing this data, we can easily determine how quickly our team is designing test cases and making them ready for testing. The value of this report is perhaps questionable. What does it really say to us? When would we need this information? Decide for yourself if this is something you need in your projects.

Figure 24-4.  Test case readiness report

Test Plan Progress Report

After the team creates test plans and starts to run tests by using Microsoft Test Manager, we can use the Test Plan Progress report (see Figure 24-5) to track the team’s progress in testing the product. We will get information about the number of test cases that passed, how many failed, and the number of inconclusive, blocked, or never run tests among other things. This information can be good input to a project manager in conjunction with the code coverage metrics described in Chapter 14. We can also see this report from inside Test Manager.

Figure 24-5.  Test plan progress report

Note  The reports in this section are mapped to the particular process template they support. If you find a report you like, it is possible to customize it to work with another template. In Chapter 32 we describe how to customize an existing report to do this and other things.

Custom Reporting

The reporting capabilities in TFS give us access to most of the information we manage in our ALM process. In the previous section we have seen how standard reports give us metrics for our project at a general level. By customizing, extending, and creating new reports we can really get the intelligence to know what works well in our projects and what does not.

Note  In Chapter 32 we will look at the details of reporting in TFS, including how to create custom reports based on the data models described here.

Data Warehouse Model

Test Result Tables

To query about data for test results, test case association, and test outcome we can use the test result tables’ bugs (see Figure 24-6).

Figure 24-6.  Test results data model

Test Run Coverage Tables

The test run coverage tables (see Figure 24-7) lets us report on code coverage for a test run.

Figure 24-7.  Test run coverage data model

Also see the Work Item Test Results tables in Chapter 9, as well as the Build Details tables in Chapter 29 for more warehouse data useful for analyzing the testing process.

Assessment

To help us evaluate an organization’s maturity in different ALM areas Microsoft has developed their ALM assessments (see Chapter 4). Based on the score of the assessment we rreceived, a maturity level for a specific area that we can use for evaluating which direction to take our ALM efforts.

So based on the score, we can help the organization reach the maturity level they need for these areas. Table 24-1 list questions that can be used as a basis for an ALM assessment in the software testing area.

Table 24-1. ALM Assessment Questions

Area

Sample Question

Discussion

Test Resource Management

Is there a dedicated test lead in place?

Are there appropriate tools available to perform automated testing?

TFS will help with this.

Is there a management system in place to track work items, defects, and change requests?

Here we have another good opportunity to show the benefits of TFS.

Test Planning

Is a test plan or strategy in place before starting testing?

This is something for the project manager or product owner to consider. We can always benefit from having and executing this plan.

Is the test team represented at the design review stage?

The test team should always be involved in our opinions.

Do test plans consider integration testing with other systems and third-party products?

Test Management

Is test execution tracked against the test plan?

Is the test plan followed?

Are the appropriate reporting processes in place?

TFS will definitely help us getting the metrics we need.

Is the end-user or customer-acceptance criteria well defined and evaluated?

This should be included in the definition of done.

Has testing the following non-functional requirements been taken into consideration where appropriate: performance, scalability, security, accessibility, regression, localization, and load/stress/soak?

This should be covered by the definition of done.

Has code coverage been considered where appropriate?

Discuss with the customer what level of code coverage is necessary. TFS will help us implement the automated tests and give us statistics of code coverage.

Test Types

Is User Acceptance Testing (UAT) used?

TFS includes good tools for handling UAT. The Test Manager is very helpful here.

Are automated UI tests performed?

Easy to implement using TFS.

Are automated integration tests used?

Easy to implement using TFS.

Any data generation tools?

Easy to implement using TFS.

Any stress test tools?

Easy to implement using TFS.

Any Performance Analysis tools?

Easy to implement using TFS.

Database Testing

Do you have automated testing of your databases?

Don’t forget to include the database development in your testing.

Is there suitable test data to ensure application tests are valid?

Do you have a repeatable data set for testing?

We can use TFS and VS to accomplish this. This is a good developer and testing practice.

Summary

In this chapter have seen how we can use TFS to retrieve information for KPI assessment and also how we can see the test status using standard reports from TFS.

We have also shown how many of the assessment questions from the Microsoft online assessment can help us plan for successful implementation of testing practices.

The next section of this book covers how we can use TFS to release our application.

PART 6

Releasing

Over the years much has changed in the way we develop software. We have started to build more often than in the past, sometimes every night. Instead of a build development phase and a separate build phase, we strive to form one continuous flow. This process is referred to as a continuous delivery process as it can be implemented to help us deliver features to our customers in an efficient and deterministic way. Analyzing how we release our software gives us insight into what steps in the process have dependencies to other steps, which steps are handled manually, which steps are error-prone, and so on. This helps us to improve our process so that we can deliver higher value to stakeholders and customers.

In Part VI we look at the concept of Continuous Delivery and what it means. In this part we also look at build automation using the TFS Build system. With automated builds in place we can create new releases of our software whenever we want and in a predictable way. We also can use automated builds to drive the continuous integration and delivery process by integrating deployment and testing with the build process. Finally, we can use automated builds to support the ALM process by measuring code quality, providing metrics, and giving feedback in a consistent way.

In Chapter 27 we focus on application deployment and running automated tests on dedicated test servers. We show the overall build-deploy-test process and focus on how we can extend the default process template by adding the deployment and test on test server activities with as little effort as possible.

Software systems, development processes, and project resources have tended to be distributed more and more over the recent years. This means that they become more and more complex and specialized. Most often, software products are in an ongoing development cycle with testing and releases integrated, making it all even more complex to manage. In this part we see how we can use Visual Studio 2012 and Team Foundation Server 2012 to improve our release management process.

CHAPTER 25

Continuous Delivery

In the previous chapters we’ve looked at development and testing practices and seen how the tools in Visual Studio help us to be more productive in each practice. In the chapters to come we will take our set of practices and implement a process where it all comes together to form one continuous flow. We will refer to this process as a continuous delivery process as it can be implemented to help us deliver features to our customers in an efficient and deterministic way. Many steps (perhaps all) in this process can and should be automated if we want to fulfill the ALM vision but automation is not important in itself. Instead, the work we do to improve the delivery process is what makes us successful. Analyzing how we release our software will give us insight into what steps in the process has dependencies to other steps, which steps are handled manually, which steps are error-prone, and so on. This knowledge is then used to help us focus on the most important parts first.

To enable continuous delivery we will look at some important techniques we can use as tools to support this process.

Continuous Integration

“Continuous integration is a software development practice where members of a team integrate their work frequently; usually each person integrates at least daily—leading to multiple integrations per day.”

—Martin Fowler

The term continuous integration (CI) was introduced by Martin Fowler and is now the defacto standard in agile projects. Having worked in a project with a CI process in place it is really hard to imagine how a project could actually work without it. Of course it can, but an agile project requires new ways of working and just like Scrum is said to be all about common sense, so is also CI. But there are several problems with agile development from a deployment perspective, such as

- Testing. We’ve seen in earlier chapters the need to do testing earlier and more often in an agile project because software is built incrementally in short iterations.

- Cross functional teams. Ideally the team should be self-organized, meaning more people need to be able to deploy software.

- Shippable product in every iteration. Manual routines used to work, but now it is not okay to spend one week on installation tasks if the sprint is two weeks.

Continuous integration can help to resolve these issues. In fact, Scrum has a solution for this—use the retrospective to find ways to improve. How you can get going with improvement is what we will look at next.

Why Should We Implement Continuous Integration?

Even if the all the above makes sense it can still be hard to justify the work to implement it. So instead of just having a good gut feeling that this is a good practice worth the time required to set it up, we have listed our favorite reasons here. Continuous integration can

- Reduce risks

- Reduce manual routines

- Create shippable software

- Improve confidence in the product

- Identify deficiencies early

- Reduce time

- Improve project visibility

Still even with the good arguments for why CI makes sense we occasionally hear concerns such as the following:

- Maintenance overhead. We will need to maintain the build environment and the CI solution. Hard to argue against this one. But show us any factory that will work without maintenance; why should a software factory be any different?

- Effort to introduce. Sure it will take some time to get the process started. For a new project not so much, for an existing solution we may need to add CI capabilities incrementally for a good return on investment.

- Quality of current solution. Some may argue that the current process is too poor to automate. If we ever hear that argument we must make sure to get CI in place.

- Increased cost. New hardware for the build infrastructure will need to be purchased. But think about the savings we will get by raising quality and identifying problems much earlier in the process.

- Duplicates work. With CI we need to automate what we already do manually. Well, yes initially we do but the goal should be to share as much as possible of the work that developers and testers do anyway. For instance the build system can use the same solution files the developers use and the developers can use the deployment script to update their machines when a new version needs to be installed locally. In fact how could we possibly manage to do manually what the CI process can do for us many times every day?

Finally to get continuous integration working the team needs to agree on the rules around the process. If the rules are not followed there is a potential risk the quality of the result will degrade and people will lose confidence in the process. We recommend using at least the following rules as a starting point

- Check in often. The CI process needs changes to work, the smaller changes we can commit and the more specific the changes are the quicker it will be for us to react on things that go wrong.

- Do not check in broken code. Checking in often is great but don’t overdo it. Don’t check in code until it works and never check in broken code. If you need to switch context, use the shelve or suspend feature in TFS to put things aside for a while (see Chapter 15 for details on these features).

- Fix broken build immediately. If you happen to break something, it is your responsibility to fix it.

- Write unit tests. The system needs to know what works and what doesn’t. Unit tests and other inspection tools should be used to make sure the code does more than just compile.

- All tests and inspections must pass. With inspections in place you must pay attention to the result. Use feedback mechanisms to make people aware when something is broken.

- Run private builds. If you can do a test build before checking in you can avoid committing things that don’t work. TFS can build from a shelveset using a feature called Gated Checkin.

- Avoid getting broken code. Finally, if the build is broken, don’t get the latest code. Why go through the problem to work on code that doesn’t work? Instead use the version control system and specifically get the latest version that worked.

Components in Continuous Integration

So now we know what continuous integration is all about. Or do we? What actually does a CI process contain? Compiling code to a set of deployable files? Running unit tests? We think build automation with integrated running of unit tests is a great start but the CI process can be more than that. Figure 25-1 shows a process we consider a complete CI solution and should be what we strive to achieve.

Figure 25-1.  Components in the continuous integration process

Let’s drill down into each of these components in more detail.

Build Automation

Build automation is the core step in the CI process. The automated build system is typically used to drive the CI process and not only do compilation and execute unit tests. An automated build should be possible to trigger by anyone (having permissions to do so) at any time and it should be possible to set up a schedule for the builds (such as nightly builds). The build process should also publish the build results at a central location so people can go and get builds easily, as well as look at the build status.

Automating the build process is of course not always a trivial task but the benefits it gives are many, including

- Saves time

- Reduces errors

- Predictable results

- Helps us learn about problems early

- Reduces dependency on individuals

Any new project should implement an automated build skeleton because having the foundation in place from the start makes it so much easier to just do the right thing.

When implementing automated builds we also need to think about what kind of build we need in which scenario. It is common for development teams to not just have one build setup for a branch, but instead more likely all of the following:

- Continuous integration builds. Lightweight to give feedback quickly, often incremental with limited testing.

- Private builds. Test the changes to be committed before actually checking them in.

- Nightly builds. Complete builds with more exhaustive testing.

- Release builds. Complete builds with release notes.

Database Integration

Another activity that often takes a lot of time in the deployment process is database integration. Traditionally updating databases as part of a change is a manual process where either the developer writes database change scripts or a DBA performs a comparison between two known database versions and then runs the upgrade manually.

There are, however, ways to do this as part of the continuous integration process. For instance if we can run a tool that performs a comparison and generates a change script we should be able to tweak the build process to do that automatically. If we are afraid that if the process fails we will have an environment that doesn’t work we can protect that by automatically running a backup before the upgrade and should something fail we have the build process restore the database to its previous state.

Deployment

To use the build result for manual or automated testing we need to deploy the build onto a test lab. The deployment process can be as simple as just copying over a set of files to the target environment or it may require local installations on multiple machines. But there are tools and techniques to solve any of these challenges; we simply have to spend time learning about the steps involved in getting a release installed.

Testing

Now when the software has been deployed, we can run tests on it. The core compile phase of the build process typically runs the core unit tests but for other automated tests such as regression tests, build verification tests, or smoke tests, we want to run those tests on a realistic environment with proper test data available. Ideally we want to have a process where we can run automated build verification tests after every deployment to verify that the software works well enough for our testers to spend time testing it. After that we can add more automated regression tests as we find value for, the process of running them as part of the continuous integration process will be the same.

Inspection

Having all these steps integrated is great but how do we know whether something goes wrong? Compiler errors and failing tests can easily be trapped and visualized. But what about code quality? Can we for instance fail a build if the code coverage for unit tests is below the level we agreed on? Or if the code has become more difficult to maintain over time can we get notified of that? The continuous integration process is often thought of as the heartbeat of the project and this is something we can take advantage of for inspection. Some inspections should lead to immediate action, such as compiler errors should fail the build. Others can be reported to a central location so that we can look at the trends later on, like for instance code metrics and overall build status.

Feedback

The last step in the process is to have a good way to notify the team about what works and what needs to be looked at. A good feedback system will give us ways to know about an issue immediately so that we can fix it before it grows to a big problem. We can try to be as creative as possible here to make the build result visible to the team in a good way like on a build monitor or by sending out email alerts.

In fact, wouldn’t it be great to have this kind of information available as part of the project status on the team’s home page? Figure 25-2 shows how builds can be pinned to the home page in the TFS Web Access.

Figure 25-2.  Continuous feedback from the build process using a TFS Web Access favorite

Figure 25-3 shows how we add a build definition to the team favorites from the build explorer view in the TFS Web Access.

Figure 25-3.  Adding a build definition as a TFS Web Access team favorite

Now we know what we can do but how does this fit into the overall delivery process? To take advantage of continuous integration we should also think about continuous delivery.

Continuous Delivery

The problem with continuous integration is not that is not a good solution. It’s just that it can be a solution to a non-existing problem. Deployment as part of the CI flow is not just about automating the build, test, and release processes. We need to think about delivery to really add value to the deployment process.

Continuous integration is great and it gives us a framework for efficiently producing software in a controlled fashion. But to get the most out of it we need to look at how it fits into the overall process of delivering software. In an agile project we want to deliver working software in every iteration. Unfortunately this is easier said than done; it often turns out that even if we implement CI and get the build process to produce a new installation package in a few minutes it takes several days to get a new piece of software tested and released into production. So, how can we make this work better?

The “null” Release Cycle

Let’s start by asking the following simple question:

“How long does it take to release one changed line of code into production?”

The answer is most likely much longer then we would want to. So what stops us from improving? First we must know more about how we release our product. Even in organizations that follow good engineering practices the release process is many times neglected. A common reason why this happens is simply because releasing software needs collaboration across different disciplines in the process. To improve the situation we need to sit down as a team and document the steps required to go from a code change to the software released into production. Figure 25-4 shows a typical delivery process and in practice work happens sequentially just like in the picture.

Figure 25-4.  A typical delivery process

When we have come this far we now know a lot more about the delivery process, which means we can start optimizing the process.

1. Look at what steps in the process take the most time and see what can be done to improve.

2. Look at the steps in the process that most often go wrong and understand what is causing it.

3. Look at the sequence of the steps and think about how they need to be run in sequence.

Having looked at the process and asked the questions, we now have a better process as shown in Figure 25-5.

Figure 25-5.  An optimized delivery process

In this model we have changed the process so that most steps are automated by implementing automated tests as well as automated build and deployment. Releasing to production automatically is not for the faint-hearted so this would be done manually but using the same automated scripts as the automated deployment to test and staging environments. We do however believe it can be possible to automate even release to production, especially if we have had this working from the first iteration of the project. By doing so we would build up confidence for the process and having seen it work throughout the development cycle should make us trust the process even in this critical stage. We have also parallelized the acceptance test and preparation of the production environment. By doing this in parallel we can push the release to production as soon as the acceptance tests are green instead of the traditional stage to production first after the acceptance tests have passed.

Release Management

Continuous delivery gives us a great practice to produce updates in a controlled and effective manner. But without an intentional release management discipline we can lose much of its value. What we need to add to the picture is how the release planning ties into the deployment process and ensure we know what features we want to deploy where and when.

In a Scrum project we have a good situation when it comes to release management because the first thing we do is create a product backlog and continuously groom the backlog as part of the project cycle. In other words, the backlog is our release plan. If we don’t work with Scrum we need to use other means to create the release plan so that we have knowledge of which features we are going to deliver when.

With a release plan in place we can now design our delivery process to support the difference phases in the project. If we have need for concurrent development we can implement a branch strategy to support this. With multiple branches we can add continuous integration builds to keep the code clean and our environments up to date and so on. Based on the release plan and the continuous integration process we can even automate the release notes for every release we do.

Summary

Proven practices are good; best-practices in a streamlined process can make your development work really stand out against the competition. In this chapter we’ve looked at continuous delivery and seen how practices such as automated build, deployment, and testing give us a framework for delivering new software quickly. With inspection and feedback actively built in to the process we get complete control and can always know what part of our software works and what does not.

The following chapters give us the details to set up a continuous delivery process. We will start by automating the development practices using the TFS build system and implement a continuous integration process. With CI in place we can take the work further and use automated deployment and testing to implement a complete delivery process. Finally, we are going to look at how we can use TFS to support the release process and give us full control over the delivery flow. All this together will really make our delivery stand out!

CHAPTER 26

Build Automation

In the previous chapter we looked at how continuous integration and continuous delivery can help us build software at a faster pace. Projects working iteratively and incrementally must have solid routines for managing the test environments to assure the quality of the product; and what better way to support these routines than automated builds?

In this chapter we will look at how we can implement build automation using the TFS Build system. With automated builds in place we can create new releases of our software whenever we want in a predictable way. We can also use the automated builds to drive the continuous integration and delivery processes by integrating deployment and testing with the build process. Finally, we can use the automated builds to support the ALM process by measuring code quality, providing metrics, and giving feedback in a consistent way.

Automated builds are a core component that many other practices depend on, so be sure to establish the practices around automated builds as early as possible in the project. If we have the core process for automated builds in place from day one, we can add other capabilities as demands increase over time.

At Kangreen we need to implement an automated build solution that satisfies the following requirements:

1. A nightly build should create an installation package for deployment to team environments.

2. A continuous integration build should validate all check-ins to TFS.

3. A release build should be available to create a release package for deployment to test, staging and production environments.

4. Release builds should produce components stamped with the release version number.

5. Release builds should be stored in a separate release management database.

We will use these requirements throughout the chapter to build a complete automated build solution that does much more than the out-of-the-box template. In Chapter 27 we will look at how we can implement a solution for packaging and deployment of the release build.

TFS Build Architecture

Just like the rest of TFS the build system is designed to support both small startups and large enterprises, which means the system must be scalable to handle enterprise-scale requirements. At the same time the system must not be complicated to set up and maintain in order to work for a small company with limited resources.

A TFS Build system contains the following components

- Team Foundation Server: The TFS holds the source code, but also the build scripts and other build artifacts.

- Build Controller: The Build Controller processes build requests and assigns them to the most suitable build agent based on availability and constraints.

- Build Agent: The build environment contains one or more Build Agents. The Build Agent is the service that does the heavy lifting in the build process, such as compiling code, running unit tests, and code analysis tools. The Build Agents can be installed on multiple machines connected to the same build controller to scale out the build environment as needed.

- Drop Folder: The build output is by default copied over to a file share, or drop folder, for consumption by others.

- Symbol Server: If configured, the build process will output symbols to a symbol share for reference. Having the symbol information available for each build makes debugging easier because we now have a shared place to locate the symbol information, including reference to source code in TFS.

- Lab Environment: A lab environment can be setup to integrate the build process with automated deployment and testing. Labs can be created using Visual Studio Lab Management or can be managed by other tools. The built-in processes have support for Visual Studio Lab Management environments out of the box.

The high-level process of TFS build can be summarized as

- When a new build is requested TFS queues it on the assigned Build Controller.

- The Build Controller looks at the build request and assigns it to a suitable Build Agent.

- The Build Agent runs the build workflow. Build results are copied to a drop folder and symbols to a symbol server share.

- Finally, if using Visual Studio Lab Management the controller can deploy the build to a lab environment and run automated tests in the lab.

Figure 26-1 shows the logical setup of a TFS build environment.

Figure 26-1.  TFS build architecture

See Chapter 30 for more details on implementation planning and what to consider when designing a TFS topology.

Installing the Build Service

The first component to install is the build service. The build service is the collective term for a host process servicing a build controller or build agents. The service can be installed on multiple machines to scale out a TFS build solution.

The installation is done from the TFS installation so what we need to do after the TFS installation is open the Team Foundation Server Administration Console and select the Build Configuration.

Next, choose Configure Installed Features to configure the build service and provide the following information:

- The TFS project collection this build service is servicing. A TPC can have multiple build services, but a build service can only service one TPC. This is something to be aware of when partitioning team projects into team project collections. Unfortunately it is only supported to have one build service per machine which is the real limitation, otherwise we could share build machines by setting up multiple services.

- Ports and protocols for the build agent communication.

- Identity running the service.

- Identity to connect to TFS as (by default the service identity).

- Should the service be run interactively? This is required if the build process will contain activities that need to interact with a Windows desktop.

Figure 26-2 shows an example of how the Kangreen build system was set up.

Figure 26-2.  Build service configuration

Installing the Build Controller

After the build service is installed we can create a build controller from the Team Foundation Server Admin Console by clicking on New Controller.

The configuration is very straightforward (as shown in Figure 26-3). We can name the controller to make it easier to select the correct one later when we define the build processes. The controller can be disabled and restricted to run less concurrent builds than the total number of enabled agents.

Figure 26-3.  Build controller configuration

Installing the Build Agent

Finally, we add build agents to our setup. Again this is done from the Team Foundation Server Admin Console this time by clicking on New Agent.

Figure 26-4 shows the configuration dialog for a build agent. We give it a name and description and connect the build agent to a build controller.

Figure 26-4.  Test agent configuration

We also can add tags to the agent. A tag can be seen as attributes we stick on the agent to describe its capabilities. We can then later use the tags in a build definition to define the requirements the build definition has. The build controller looks at the tags in a build request and then finds a compatible agent for the build.

Creating an Automated Build

The initial task of any build automation effort should be to set up a nightly build. Because this typically involves most parts of the build processes it is a great start—with the nightly build in place we can trim it down to create continuous integration builds or extend to fit the needs of our release builds.

We use the Visual Studio Team Explorer (see Figure 26-5) to define our automated builds. In TFS an automated build is created as a build definition that can be seen as the sum of a build process and the configuration that applies to this specific build type.

Figure 26-5.  Build management in Team Explorer

From the Team Explorer—Build window we get an overview of the available builds definitions and the list can be filtered if the team project contains many builds. We can drag a build definition to the My Favorites section if we want to track its status. The favorite pane shows a nice diagram of the build history and we get good feedback of the build quality this way. We can always navigate from a build/build definition to the build log to view the build details or to the build explorer to look at the build queue.

The build window has action menus that allow us to create new build definitions or manage the build services. When we choose to create a new build definition we will go through a wizard to map our build requirements to a build process.

The first step is to name the build definition and give it a description (see Figure 26-6).

Figure 26-6.  Creating a new Build Definition—General settings

Tip  Because the build definitions are presented as a flat list it is a good idea to implement a naming convention to help locate the correct builds. We prefer to use <product> <branch> <trigger> as the convention, for example: Main Nightly Build would be the nightly build for the main branch in a team project with only one product. Main CI would be the continuous integration build for the same branch.

We can also control the queue processing for this build definition, for instance by setting it to Disabled if we temporarily want to prevent new builds to be queued.

Next we select when the build should be run. As shown in Figure 26-7 it is easy to go from a manual build to a scheduled build to a CI build. Think about which builds will run when using scheduled builds so not all builds are queued up at the same time. The end effect will be the same but build times may look strange because the total build time will include the time waiting for an available agent to process the build.

Figure 26-7.  Creating a new build definition—Trigger settings

The third step is to map the workspace for the build (see Figure 26-8). Creating a workspace for the build definition is the same as setting up a workspace for a developer (Chapter 15). Use Cloaking to avoid downloading files to the build server that will not be used by the build process (for example, documentation). The symbol $(SourceDir) maps to the build folder defined when the build agent was configured; this is where we will have to go if we need to troubleshoot build problems locally on the build agent.

Figure 26-8.  Creating a new build definition—Workspace settings

The next step (see Figure 26-9) lets us specify which build controller (and therefore which build agents) we want to run our build on.

Figure 26-9.  Creating a new build definition—Build Defaults settings

We also get to select the staging option for the build—none or a file share. Most build definitions use a file share for storage and just as for naming the build definitions we recommend deciding on a folder structure to help locating build output easily.

Tip  Using the format \\fileserver\drop_share\ < product > \ < branch > gives a good matching structure similar to the build definition name format we suggested earlier. For example, the drop folder could contain the Main Nighty Build and Main CI under the Expense Report\Main folder and the Release Build under Expense Report\Release.

Then we come to the core part of the build definition—the process. The process is where we define what the automated build actually should do.

We first select the build process template that forms the build process (see Figure 26-10). Next we configure the process by providing the required and optional arguments. In the default template all we need to do is provide the projects to build, the rest can be left as default.

Figure 26-10.  Creating a new build definition—Process settings

In a later section in this chapter we look more at the details of the default template.

Finally, we can configure the retention policy for our build (see Figure 26-11). TFS has a built-in job that takes care of cleaning the drop folders for our build definitions to avoid filling up the disks.

Figure 26-11.  Creating a new build definition—Retention Policy settings

We change the policy to keep as many builds as we want per build outcome, but think about the amount of data that might be kept if you are too generous with the policies (Keeping 10 latest for each outcome is potentially 80 builds per build definition. With only 10 build definitions that’s 800 builds on the file share!).

It is also possible to change what gets deleted when the policy is applied (see Figure 26-12). By default, all except test results are removed. The test result is required to be kept if you want to be able to analyze test result and code coverage historically. It is also needed to keep the test results to get the recommended tests feature in MTM working.

Figure 26-12.  Creating a new build definition—Build Delete Options

That’s it; we now have a completely automated nightly build in place! Next, we are going to look at how we can run and analyze the test run.

Running a Build

We can start a new build manually from the Team Explorer – Build window or from the Build Explorer. At the time we queue a new build we can override general values from the build process as shown in Figure 26-13, including

- What to source to build—latest or from a shelveset (see “Running a Private Build” later in this chapter).

- Override the default Build Controller.

- Set the priority in the build queue.

- Change the drop folder.

Figure 26-13.  Queueing a new build

We also can change the process parameters that have been defined as visible when queuing the build (see Figure 26-14). We will look at process parameters later when we create custom build processes.

Figure 26-14.  Viewing parameters when queueing a build

Using the Build Explorer

After the build has been queued we can monitor the status from the Build Explorer (see Figure 26-15). The Build Explorer can be opened from a running build in the Team Explorer or from the build definition.

Figure 26-15.  Build Explorer

We use the Build Explorer to manage much of the build process, such as to monitor build progress, analyze build logs, and start new builds.

We can open the build report to look at a running build or to analyze a completed build (see Figure 26-16). The build report can be used to manage the build in many ways. We can, for instance, navigate to the drop folder for the build, look at the diagnostics logs, or run one of the many commands from the action menu.

Figure 26-16.  Build Explorer activity log

Using the Web Access to Manage Builds

Visual Studio and the Team Explorer will be the primary interface for build management but we want to make you aware of the build capabilities in the Web Access. We can use the Web Access to run, view, and manage our builds. Creating the build definitions and managing the build environment needs to be done in Visual Studio but now with the Web Access we have a tool that anyone in the team can use to look at the build results. Testers can queue builds to update the test environment whenever they are ready to take on a new build. Release managers can look at the build summary to learn about the build quality (see Figures 26-17 and 26-18).

Figure 26-17.  Build Explorer in TFS Web Access

Figure 26-18.  Build activity log in TFS Web Access

Implementing Continuous Integration

So far we have looked at setting up a standard build definition for our nightly build process. In Chapter 25 we looked at continuous deployment and how continuous integration is a critical practice for any team that wants to build software incrementally and iteratively.

Implementing continuous integration can require some work to establish when it comes to the practical tasks within the team. Fortunately when it comes to build automation it is a very small step to go from a nightly build to a CI process.

To create a CI build definition we would do only a few things different from setting up a standard build, in fact we can use for instance a nightly build and make the following changes to setup CI in our project:

- Clone a standard build, for instance the nightly build.

- Change the name of the build definition. It may be a good idea to disable the build definition while working with the build configuration to avoid triggering.

- Change the trigger type to continuous integration, rolling builds, or gated check-in.

- CI triggers on every check-in.

- Rolling builds are CI that trigger a new build only when the previous has completed, which may cause several changesets to be included in the build. We can also configure the trigger to build no more often than a given interval, for example every 30 minutes. Rolling builds are generally a solution if we have a high load on the build server (large teams or long-running builds).

- Gated check-in builds trigger before the code is committed to TFS. See the following section for an example on gated check-in. Similar to rolling builds, the gated check-in trigger can be set up to merge and build a number of submissions instead of one at a time.

- Change the build defaults staging location to not copy output files because the CI build should not be used for staging.

- Optionally change the process clean workspace to None. This causes the process to do an incremental get, downloading only the changed files since the last build. The compilers do an incremental build on the components affected by the changed source files. Potentially this can speed up the CI build considerably but a clean build is always preferred, so make sure it has significant effect on the build times before switching to this. Review which steps of the build process that add value to a CI build and disable features which are okay to run nightly in order to make the CI build fast. Unit tests should typically always be run, but slower tests could be sufficient to run only nightly.

Note  You can add ***NO_CI*** to a check-in comment to prevent a CI build from triggering. This should generally be avoided because it discourages the whole idea with CI. But occasionally it may be tempting to use, for instance if you have non-code content (document. images) in source control and want to avoid triggering new builds. Consider setting up the workspace so that non-code folders are cloaked, this prevents the CI build from triggering.

Cloning a Build

Even though it’s not very complicated to create a new build definition, creating one can be tedious and if the process is complicated there is always work to get the properties right. For instance, when we create a new branch we may need to setup a CI build, a nightly build, and a deployment build to the team test server. It would be nice to have a tool that could automate this process.

So far Visual Studio does not have this function built-in, but the TFS Power Tools (http://visualstudiogallery.msdn.microsoft.com/27832337-62ae-4b54-9b00-98bb4fb7041a) includes a Visual Studio add-in that implements the clone functionality. Figure 26-19a shows how to run this command.

Figure 26-19a  Clone build definition from the TFS Power Tools

See http://msmvps.com/blogs/molausson/archive/2010/10/21/clone-a-build-definition.aspx for a walk-through of how to use the TFS Power Tools to clone a build definition.

There is also a nice Clone to Branch feature in the Community TFS Build Manager Tool (http://visualstudiogallery.msdn.microsoft.com/cfdb84b4-285e-4eeb-9fa9-dad9bfe2cd10) that can be used to clone a build definition as shown in Figure 26-19b.

Figure 26-19b.  Clone Build Definition from the Community TFS Build Manager

Gated Check-in Builds

The gated check-in build trigger is used to enforce that code submitted to TFS passes the server build before it is committed. When a developer checks in code and it matches a workspace in a gated check-in build definition, a notification dialog is shown (see Figure 26-20). By default the pending changes are preserved locally but this can be overridden by changing the options. We recommend using this behavior; the opposite will often result in more merge conflicts because it will undo the pending changes while the gated check-in runs on the shelveset.

Figure 26-20.  Gated check-in notification dialog

After the build has been queued, we also get a notification in the Team Explorer (see Figure 26-21), where we can navigate to the Build Explorer and look at the build progress.

Figure 26-21.  Gated check-in notification in Team Explorer

After the build has completed we need to update the pending changes with the merged result from the gated check-in build. From the build log we have a command to reconcile the workspace that does just this (see Figure 26-22).

Figure 26-22.  Reconcile workspace after a successful gated check-in build

It can take a little while for everyone to get familiar with gated check-in, especially the need to reconcile the workspace. After you are there you have the benefits of CI but without the problem that a failed CI build is a problem for everyone because the code has been committed. With gated check-in the problem is only the developer failing the build, no one else gets to see the broken code.

Running a Private Build

We can take advantage of the capabilities that gated check-in offers to run a private server build. This is a really powerful way for a developer do a complete server-side validation build before committing code to TFS. We can use this feature for example if we want to make sure a large refactoring, such as a framework upgrade, and deploy the build to a test machine to run tests on it. A gated check-in build would have committed this if the build succeeded, so a private build would have been more appropriate in this case.

Figure 26-23 shows how to queue a private build. Select “Latest sources with shelveset” and then either select an existing shelveset or create a new one from the pending changes. Checking “Check in changes after successful build” results in more or less the same as a gated check-in where the changes are committed to TFS if the build succeeds.

Figure 26-23.  Running a private build

Configuring Builds

Setting up the initial build is pretty straightforward. The challenge comes when we want to do more than just compile .NET projects. In this section we are going to look at what we can configure in the default build processes, essentially this means enabling or disabling features in these workflows.

Often simply configuring the build process is not sufficient and we need to extend the build further. We will look at both customizing the templates with existing activities, as well as implementing custom activities for our special needs.

By default, TFS comes with three build process templates—these are added source control in each team project under the $/BuildProcessTemplates folder. The default templates are the following:

- DefaultTemplate.11.1.xaml: This is the default template for building Visual Studio 2012 projects.

- LabDefaultTemplate.11.xaml: This is a build template for Visual Studio Lab Management that takes the result from a TFS build, deploys it to a lab environment, and runs automated tests in the lab. We will look at this workflow in Chapter 27.

- UpgradeTemplate.xaml: This template is for compatibility with the MSBuild-based TFS build workflows used pre-TFS 2010.

In this chapter we focus on the default template because it is, as it says, the default template for TFS 2012 builds.

The default template contains these high-level activities:

- Download the build definition.

- Generate a build number.

- Select a build agent for the build.

- Initialize the server workspace.

- Get the source code for the build.

- Create a label for the build.

- Compile the code.

- Run tests.

- Associate work items with information from the build.

- Calculate impacted tests.

- Index sources.

- Drop the build.

- Send build notification events.

We can configure the behavior of these activities and more from the Process tab in the build definition.

Required Parameters

The only required parameters in the default template are the items to build. The configurations to build can be specified but is optional, if we don’t specify this Debug, Any CPU is the default. We can select any msbuild based project (*.sln, *.*proj) and they will be built in the order listed. We can specify zero or more configurations to build, if none is specified it will default to Debug, Any CPU. All projects will be built for each configuration. Figure 26-24 shows the required build process parameters.

Figure 26-24.  Required build process parameters

Basic Parameters

The basic parameters let us control the most common settings such as test settings, build number format, and logging verbosity (Figure 26-25).

Figure 26-25.  Basic build process parameters

Configuring Testing

The Automated Test settings let us configure many aspects of the test execution happening as part of a build. In particular we can select the run settings and the criteria for selecting the tests to run. Selecting tests to run is commonly done by using a file pattern (**\*.Tests.dll will match any file ending with .Tests.dll recursively from the test run directory. See chapter 16 for details on test context and dependencies).

Figure 26-26 shows how we can control the framework for test execution and control if we fail the build based on failing tests.

Figure 26-26.  Configuring test run settings

Configuring Code Coverage for a Build Definition

In Chapter 16 we looked at unit tests and how the test framework can analyze the test run for code coverage. To get a trend over the code coverage we need a way to measure code coverage in a predictable way and what better than to use for instance the nightly build to collect and report that metric.

Enabling code coverage on your build definition is as simple as selecting the “Enable Code Coverage” item in the options combo box when you specify your test assemblies to be run (see Figure 26-27). This assumes we are using the new Visual Studio Test Runner, if not then we can use .testsettings as described in Chapter 16 about unit testing.

Figure 26-27.  Enable code coverage in a build definition

Figure 26-28 shows the code coverage output in the build result.

Figure 26-28.  Code coverage in the build report

Customize Build Number

By default, a build is assigned with a build number of $(BuildDefinitionName)_$(Date:yyyyMMdd)$(Rev:.r), for instance Main Nightly Build_20120710.1. This can be changed to another pattern as appropriate using the Format Editor in see Figure 26-29.

Figure 26-29.  Customizing the BuildNumber format

If the Label Sources property is set to true, then the source code for the build is assigned to a label with the build number as the name (see Figure 26-30).

Figure 26-30.  Viewing labels created by Team Build

Note  The build number is not set on the resulting build components, which we might expect. If we want to version stamp the build output, we recommend including the build number as a way to relate a built file to the build that produced it.

Advanced Parameters

The advanced parameters, as the name implies, are settings we configure less frequently (see Figure 26-31). Many of the settings enable or disable features (such as test impact analysis and testing). There is also the possibility to control the underlying msbuild compiler, by passing in custom arguments.

Figure 26-31.  Advanded build process parameters

Get version is useful if we want to build based on a specific version, for instance if we want to re-create a release build labeled R1.0 we can pass in LR1.0 as the value to the parameter.

Passing Parameters to MSBuild

The workhorse MSBuild is used to build the projects in TFS builds. If we want to customize the behavior of MSBuild, we can pass arguments from the build process parameters as shown in Figure 26-32. We looked at the architectural layer diagram in Chapter 13. The diagram serves several purposes, including checking that no code dependencies have been created that violate the rules in the layer model. Because the validation can take some time to run it makes sense to include it in an automatic build.

Figure 26-32.  Configuring layer validation as part of build

Customizing Builds

In the previous section we looked at how to use an existing process template. In this section we will cover how to create or extend a build process template with custom logic.

The build process is defined using the Microsoft Workflow Foundation XAML language. This is basically a large XML file but we will edit it using the workflow designer most of the time.

Tip  TFS Build system is a pretty extensive solution and you may need to do more than we have covered in this chapter. The ALM Rangers Build Customization Guide (http://vsarbuildguide.codeplex.com/) contains numerous examples of build customization that can be used to quickly implement extensions to your build process.

Built-in Build Activities

The goal should be to use existing activities before creating our own. The main reason for this is less work for us—no code to write or maintain. But to do this we need to know what we have in our toolbox. First, we will look at the built-in activities for TFS build. The built-in activities are of course there to support the default build process but they are great building blocks we can leverage when creating our own custom workflows. Table 26-1 lists some of the native activities. For the complete list of built-in build activities, see http://msdn.microsoft.com/en-us/library/gg265783(v=vs.110). A great way to learn about the native activities is to open up the default template and explore how they are used there.

Table 26-1. Example of Native TFS Build Activities

Activity

Description

AgentScope

Controls where parts of the build process executes.

ConvertWorkspaceItem

Converts a TFS server path to a local path.

CopyDirectory

Copies files.

FindMatchingFiles

Retrieves a list of files based on a file pattern.

InvokeProcess

Runs an external process, useful as a generic way to start programs from the build process.

MSBuild

Runs a MSBuild script.

WriteBuildMessage

Writes a message to the build log.

TFS Build Extensions

Another set of TFS build activities is available in the community driven project TFS Build Extensions (http://tfsbuildextensions.codeplex.com) on Codeplex. The library contains almost 100 activities and new releases are published almost quarterly. The source code for the activities is also available in case you would like to learn how the activites are implemented.

Table 26-2 lists some of the activities in the TFS Build Extensions package. For a list of build activities in the TFS Build Extension project see http://tfsbuildextensions.codeplex.com/documentation. Many of the activities are documented with examples.

Table 26-2. Example of TFS Build Activities in the TFS Build Extensions

Activity

Description

Email

Sends an email as part of the build process.

File

Basic file management.

LabManagement

Custom activities for TFS Lab Management including listing environments, shutdown environment and environment locking.

TeamFoundation

Various activities to integrate with TFS.

TFSVersion

Activity for handling component versioning.

VB6

Runs a VB6 compilation.

Zip

Compresses file to a .zip file.

XML

Processes an Xml file.

Installing the TFS Build Extensions

Installing a custom TFS activity so it can be used in the Workflow Designer requires the activities to be added to the Workflow Toolbox. From a tab in the toolbox we select Choose Items and then browse to the assembly to add to the toolbox (see Figure 26-33).

Figure 26-33.  Adding TFS Build Extensions to the Workflow Toolbox

For a complete up-to-date walkthrough of setting up the TFS Build Extensions, go to the project documentation at http://tfsbuildextensions.codeplex.com/wikipage?title=How%20to%20integrate%20the%20extensions%20into%20a%20build%20template&referringTitle=Documentation.

Creating a Build Process Template for Assembly Versioning

Now we are ready to turn what we have learned so far into a practical scenario. Kangreen wants a build process for creating a release build that stamps all components with a version number. To solve this we can use the default template and extend it. After looking through the library of activities, we have found that the following should do the work:

- FindMatchingFiles: Finds a set of files matching a given pattern. Built-in activity.

- TFSVersion: Activity from the TFS Build Extensions to update assembly info files with a given version pattern.

- WriteBuildMessage: Writes an informational message to the build log. Built-in activity.

The overall process for implementing this custom process template is as follows:

1. Create a new build process template.

2. Create variables to hold internal state during build.

3. Create arguments to let the user configure the process from the build definition.

4. Add the custom activities to the workflow.

5. Add the custom activities and template to TFS.

6. Make TFS locate the custom activities.

7. Create a new build definition using the new template.

8. Run a build from the new build definition.

Create a New Build Template

First we need to create the base template to do work in. We can of course start with a blank workflow document but in practice we will start by copying one of the existing ones. In this scenario we will start by making a copy of the DefaultTemplate.11.1.xaml file.

Developing a custom build workflow is similar to writing an application so we need to plan how the execution flow should look, which parameters should be used to hold state in the workflow, which arguments we should be able to set externally to control the workflow, and so on.

The workflow itself is expressed using workflow constructions such as

- Activities: The core actions in the workflow consists of control flow activities and

- Variables: Used for storing data inside the workflow.

- Arguments: Used for passing data in and out of the workflow.

Tip  The article “A Developer’s Introduction to Windows Workflow Foundation (WF)” in .NET 4 on MSDN (http://msdn.microsoft.com/en-us/library/ee342461.aspx) is a great start if you want to learn more about developing Windows Workflow applications.

Create Process Variables

We can create workflow variables to hold state in the workflow, typically to pass data between the different activities. We give the parameter a name, specify the type (can be any CLR type the workspace has a reference to), set the scope for the variable (defines the reach for the variable) and we can also provide a default value.

Figure 26-34.Working with variables in a build workflow

For the scenario we need to create the following variables:

- VersionFiles (IEnumerable < string>): Used to hold the list of matching AssemblyInfo files.

- VersionNumber (string): Used to hold the computed version number.

Create Process Arguments

A process argument is similar to a variable but is used to pass data in and out of the workflow so instead of scope we set the direction (see Figure 26-35).

Figure 26-35.  Working with arguments in a build workflow

We can customize how the argument behaves as a process parameter by using the Process Parameter Metadata Editor (see Figure 26-36), which we can open from the Metadata argument in the workflow designer. Using the Metadata Editor we can control the display name, if it’s a required parameter, when the parameter should be shown, and name the category is displayed in. By adding a #value before the category name we can control the order of the arguments to create a nice list in the build definition process view.

Figure 26-36.  Adding a process parameter

For the scenario we need to create the following arguments:

- MajorVersion (string), In parameter.

- MinorVersion (string), In parameter.

We add a category with id #500 Versioning for the new group of arguments we need for the assembly versioning.

Add Activities to New Build Template

With a template to work in and variables and arguments to work with we are ready to customize the workflow. We use the Workflow Toolbox (see Figure 26-37) to drag activities to the workflow designer.

Figure 26-37.  Using the Workflow Toolbox to add activities

One challenge with custom build workflows is to find where to add the new logic. In our scenario we are going to add logic to set the version number of .NET assemblies. The built-in mechanism in .NET for handling this is to use an AssemblyInfo-file containing version attributes (see Figure 26-38). When the code is compiled the attributes are inserted into the resulting component.

Figure 26-38.  An AssemblyInfo file

We can automatically change these values by customizing the build workflow so that we update the AssemblyInfo file after the files have been downloaded to the build server and before the code is compiled. A challenge here is that every project potentially contains an AssemblyInfo file so the process must first find all files. Figure 26-39 shows a complete workflow with the versioning logic inserted as a sequence after the GetWorkspace activity (which downloads files from TFS to the workspace).

Figure 26-39.  A customized workflow with logic for assembly versioning

Next we configure each activity using the property dialog in Visual Studio (see Figure 26-40).

Figure 26-40.  Setting properties for the TFSVersion build activity

To complete the scenario we set the following properties for the activities:

FindMatchingFiles

Finds all the AssemblyInfo.cs files recursively in the sources directory on the build server.

- DisplayName. FindMatchingFiles.

- MatchPattern. String.Format(“{0}\**\AssemblyInfo.cs”, SourcesDirectory).

- Result. VersionFiles (this is the variable created earlier).

TfsVersion

Sets the version information in the AssemblyInfo files found by the FindMatchingFiles activity.

- DisplayName. Run TfsVersion.

- Files. VersionFiles (variable set by the FindMatchingFiles activity).

- Major. MajorVersion (argument defined earlier and provided from the build definition).

- Minor. MinorVersion (argument defined earlier and provided from the build definition).

- SetAssemblyFileVersion. Checked (sets this attribute in the AssemblyInfo file).

- StartDate. 2012-01-01 (used as the offset for generating the build number part of the version number).

- Version. VersionNumber (variable created earlier to hold the version number generated by this activity).

WriteBuildMessage

Writes a status message to the build log.

- DisplayName. WriteBuildMessage.

- Importance. BuildMessageImportance.High. The message is set to High to be shown with the default logging level (normal) in a build definition. See the section on “Logging In the Build Process” later in this chapter for more details on how logging works in TFS build.

- Message. String.Format(“Version set to {0}.”, VersionNumber).

Check in the Custom Build Process to TFS

That’s it—we now have the custom process for assembly versioning in place! Next we need to add the customization to TFS. In our scenario this means adding the custom build workflow as well as the TFS Build Extension assemblies for the TfsVersion activity. Figure 26-41 shows the files we need to add in our case.

Figure 26-41.  Adding custom build workflow and activities to TFS

Making TFS Locate the Custom Activities

The next step is to make TFS aware of the custom activities. There is a built-in mechanism that copies assemblies to the build agent so they are available to customized build templates. This is a very handy solution because we only need to check in the activities in TFS and configure the build controller to locate the assemblies. The build controller then downloads the activates to the agents, which means we no longer have to go to each build agent and update the dependencies ourselves anymore.

To configure TFS to locate the activities we open the build controller properties (see Figure 26-42) from the TFS Administration Console or from the Team Explorer in Visual Studio.

Figure 26-42.  Configuring a build controller to locate custom assemblies

Create a New Build Definition

With the above in place we can now create a new build definition for the custom workflow. The new thing now is that we need to select the new build process template for our build definition. We locate the process from the Process tab in the build definition, the first time we must add it as a new process template using the dialog shown in Figure 26-43.

Figure 26-43.  Selecting a custom build process template

We configure the build as described in the section on build configuration earlier in this chapter. Figure 26-44 shows how nicely we can add the major and minor versions to use.

Figure 26-44.  Configuring the versions number in the custom build process template

Test the Custom Build Workflow

Finally we are ready to queue a new build to test the custom workflow. Figure 26-45 shows a successful build with the version set to 1.0.186.4.

Figure 26-45.  A successful build with a new version number generated

To prove that the resulting assemblies have the version number set we can go to the drop folder and look at the file properties (see Figure 26-46). This completes the scenario—we have now successfully implemented a custom build workflow that uses built-in and custom activities to implement assembly versioning!

Figure 26-46.  Proof of the assembly versioning process working

Logging In the Build Process

An important part of analyzing builds is the build log. The build log typically contains trace information, error messages, and other diagnostic output from the build process.

We can control the logging level in the build process both on the build definition side and the activity side using the BuildMessageImportance enumeration. By default only messages with BuildMessageImportance. High are shown in the log but if we want a more detailed log we can control this, for instance when queuing a new build as shown in see Figure 26-47.

Figure 26-47.  Setting the logging level when queueing a new build

Extending Builds

So far we have looked at configuring and customizing build processes. This probably solves most of our needs, but sometimes we need to extend the automated build with our own custom process steps.

We have two ways to create custom activities: implementing a XAML Activity or a Code Activity.

Creating a Solution for Developing Build Extensions

We could edit the XAML files directly using XML or the Workflow editor but for practical reasons we recommend creating a .NET solution for managing the extensions. The solution we recommend is set up like this:

- CustomActivity: A .NET Class Library used to implement custom activates and extensions. This project references the TFS assemblies needed to implement the custom activates (typically Microsoft.TeamFoundation.Build.Client and Microsoft.TeamFoundation.Build.Workflow as start).

- Template: A Workflow Activity Library used to manage the custom build workflows. This project references the CustomActivity library, which makes the custom activities appear in the Workflow toolbox so we can add them to the build workflows.

Implementing an XAML Activity to Execute a Remote Command

One way to create a custom activity is to compose other workflow activities into a reusable XAML fragment. An XAML activity is a quick way to wrap complex pieces of workflow logic or activities that require significant configuration. We can use the same constructs as for a build workflow to declare variables and in/out arguments.

As an example of an XML activity we will look at wrapping a call to the PsExec tool (http://technet.microsoft.com/en-us/sysinternals/bb897553.aspx) to remove execution of an arbitrary command as part of the build process. A typical use-case of PsExec in a build process is to run an install script on a remote machine. Figure 26-48 shows how we can create a basic sequence that essentially wraps the InvokeProcess command. We have also exposed the arguments required to use the custom activity.

Figure 26-48.  An XAML Activity wrapping PsExec to execute a remote command

The benefit of wrapping the call to PsExec lies in the details of the call. Figure 26-49 shows how the many arguments to psexec.exe are hidden from the user when the call is wrapped in the custom activity.

Figure 26-49.  Wrapping PsExec using the InvokeProcess activity

Using the XAML activity is no different from any other activity; just drag the activity from the toolbox (see Figure 26-50) to a build workflow sequence.

Figure 26-50.  Use of the PsExec custom activity in a build workflow

Next if we look at the properties (see Figure 26-51), we have a much simpler interface to PsExec.

Figure 26-51.  Setting the arguments for the PsExec activity

Implementing a Code Activity

The second approach to implement a custom activity is to use.NET and write a code activity. Writing a code activity is a bit more involved because it requires programming and not only configuration. The overall process is

- Add a new Code Activity to the project.

- Add References to required assemblies.

- Declare parameters to interface with the activity.

- Implement Execute method override.

- Use the workflow Context to interact with the build system, for instance to do logging.

Kangreen has a requirement on the build process that all release builds are to be stored in a configuration database that other tools can integrate with. The scenario is suitable to implement as a code activity because it requires both interaction with the TFS build system and a SQL Server database.

Figure 26-52 shows the core implementation of a code activity.

Figure 26-52.  Core implementation of a code activity

The parts of the code snippet to understand to write a custom activity are

- Line 12: The BuildActivity attribute is used to declare where in the build system the activity is intended to be used (Controller, Agent or All).

- Line 15-16: Arguments can be In or Out arguments and be optional or required.

- Line 20: We can get access to the build system through the workflow context, in this case we ask for an IBuildDetail which provides lots of details about the running build the activity is part of.

- Line 22: Here we use the IBuildDetail reference to get the build details used to save the build to the build store.

- Line 24: We can use the context to write messages to the build log. In this case we are using the TrackBuildMessage extension method to write a trace message to the log with a logging level of Normal (which will not be shown by default because the default setting of a build definition will only show messages of High importance).

Figure 26-53 shows the code that saves the build information to a SQL Server build store.

Figure 26-53.  Adding build information to a SQL database

This is standard .NET database code with the exception that errors are written to the build log using the context class.

To use the activity we add it to a custom build workflow and set its properties. As shown in see Figure 26-54 in this case it is appropriate to add the logic after the Run On Agent sequence. We have also added a condition so that the build is only stored in the build store if the build was successful.

Figure 26-54.  Using a code activity in a build workflow

The workflow has an argument for the connection string so that it can be set in the build definition. A complete build definition using the activity is shown in see Figure 26-55.

Figure 26-55.  Configuring the custom activity in a build definition

Finally see Figure 26-56 shows the result of a successful build.

Figure 26-56.  Log from a successful build adding the build to the build store

Build on Team Foundation Service

If we use the hosted Team Foundation Service there is also a hosted build service available for us as part of the service. The hosted build service is very similar to the on-premises version with some exceptions:

- Currently we will get one build controller and one build agent per hosted team project collection. This means only one build will run at a time on the hosted build service.

- Build agents are shared by all users of the hosted TFS service. The build machine will be re-initialized between every build so there is no risk that the build files could be seen by others.

- We cannot connect to the hosted build to configure or install software on it. The build agent has some software pre-installed, but the rest need to be downloaded as part of the build, for instance using NuGet packages. Currently, Visual Studio 2010 and 2012 are installed plus the Windows SDKs but this is most likely something that will change over time.

- Drop folders are different. Typically we will drop the build into TFS because the generic hosted build service may not have access to a file share to drop the files to.

Note  The hosted TFS and hosted build services are still in preview when this chapter was written so make sure to check the current details of the service if you find it interesting.

Feedback

From an ALM perspective it is important that we can get feedback from builds when we need it. Depending on the role it could mean immediately to react on a build failure or it could mean to weekly look at a trend over code quality indicators collected by the build process. In Chapter 29 we will look at metrics from the build process, but for now we will focus in the status notifications.

Build Notifications

Visual Studio 2012 comes with a handy tool-tray application that we can configure to monitor builds we are interested in. When a build completes, an event is raised by TFS, which is something the build notification application will subscribe to.

To use the Build Notification we need to configure it initially. Start the application from Start→Microsoft Visual Studio 2012→Team Foundation Server Tools→Build Notifications. The application loads in the Windows Tool Tray area and when opened looks like the illustration in see Figure 26-57.

Figure 26-57.  Looking at build status using the Build Notification tool tray application

To get the notifications we are interested in we can select the build definitions we care about (see Figure 26-58) and also control who triggers the build and which events we are interested in.

Figure 26-58.  Configuring the build notifications

When a build completes matching the configuration, an alert is shown (see Figure 26-59).

Figure 26-59.  Build notification alert

We can also use TFS Web Access as shown in see Figure 26-60 to add mail notifications from build events. These events let us subscribe to the following events:

Figure 26-60.  Creating a build alert from the TFS web client

- Build status events.

- Build controller status change. This can be subscribed to if we want to be notified when someone adds new build activates to the environment because that is something which causes the build controller to be restarted.

- Build Quality change. Useful if we want to initiate a workflow based on someone setting the quality of a build, for instance to deploy a build to a stage environment when the build quality is set to build verification passed.

Team Explorer and Web Access Favorites

Both the Team Explorer and Web Access client can show the build status, as well the trend information (see Figure 26-61). We add a build definition to the favorites list simply by selecting “Add to Favorites” from the context menu. If we hover over a bar in the chart, it will display the build time which helps us quickly understand how long a build is expected to take.

Figure 26-61.  Build information in the Team Explorer

For Web Access we can also add the build definition to the Team Favorites, which makes it visible on the Team’s home page (see Figure 26-62).

Figure 26-62.  Build information in Web Access

Summary

This chapter covered the fundamentals of setting up an automated build process for a realistic project. The requirement was not only to get the build automated but also included unit testing, assembly versioning, and saving the build result to an external build repository. You have also seen how the build system can be used to support the ALM process with metric collection and feedback tools.

The complexity of the automated build process will of course depend on the type of software you are producing but you should now be familiar of the core tasks a build master must know:

- Setting up a build environment.

- Create automated builds with different trigger types.

- Customize the build process workflow.

- Extend the build process with custom activities.

Next up we will complete the story around continuous delivery by looking at how we can enhance the automated builds to also include automated deployment and automated testing.

CHAPTER 27

Deployment

We have come a long way toward the implementation of a continuous delivery process. We looked at unit testing in Chapter 16 and we went through other types of automated testing in Chapter 22. We covered build automation and running tests on the build server in Chapter 26. So now it is time to complete our work and enable a completely automated delivery process.

In this chapter we focus on application deployment and running automated tests on dedicated test servers. Figure 27-1 shows the overall build-deploy-test process and we will extend the default template by adding the deployment and test on test server activities with as little effort as possible.

Figure 27-1.  The build-deploy-test process

Designing a Deployment Solution

There is a big problem with deployment: all solutions are unique. Probably very few applications built can use the exact same solution to install them.

But of course if we look at the requirements there are many similarities and for the most part quite simple steps that build up the deployment solution. Typically we need to assemble a set of files into a release package. We need to deal with configuration data such as connection strings and web service addresses. Then we may need to setup a target environment, for instance by creating and configuring a web service or a web site. Finally we often have a database that needs to be installed or upgraded as part of the deployment.

There are many different options for implementing a deployment solution, for instance we can use the following tools to package and deploy our application:

- XCopy or RoboCopy: The simplest solution is often the right choice. For an in-house application simply copying the application files to the target environment can be an easy and suitable solution.

Go to http://technet.microsoft.com/en-us/library/cc733145 for more information about RoboCopy.

- ClickOnce: A good option for a Windows application that we want to make available on an intranet, for instance. With ClickOnce we can have the build process create a ClickOnce package and then copy that package to a file share from which the users run the application. ClickOnce installs and updates the application on the client machine.

You can read more about ClickOnce deployment at

http://msdn.microsoft.com/en-us/library/t71a733d(v=vs.110).aspx.

- Windows Installer: There is no built-in support for creating a Windows installer package in Visual Studio 2012 so we have to use a third-party offering to create an installer. Using third-party solutions such as Wix or InstallShield LE for creating a Windows Installer MSI installation package is of course the most flexible solution for packaging an application. The MSI file can then be copied over to the target environment and installed using the Visual Studio Lab Management agents or using tools that supports remote execution such as PsExec, WMI or PowerShell.

Information about using Wix to create an installation package is available at

http://wixtoolset.org/.

- WebDeploy: For web applications, we can use the WebDeploy framework introduced with Visual Studio 2010. The WebDeploy framework lets us either deploy to a target environment directly or it can create an installation package that we can distribute to the target environment and run there (for instance when we want to deploy on a production server running on a different network). WebDeploy contains many deployment features, including file copy, web site configuration, and database management. There is also a feature for managing environment specific web configurations called Web Config Transformations, which uses Xslt patterns to modify web config file to suit the target environment.

More information about WebDeploy can be found at the product site

http://www.iis.net/download/WebDeploy.

- PowerShell: The PowerShell environment offers advanced scripting capabilities that can be used for everything from simple file copy operations to complete remote configuration of web sites.

Read more about PowerShell at

http://technet.microsoft.com/en-us/library/bb978526.aspx.

- Database: To create or update databases we can use the tools for database deployment found in Visual Studio 2010 or SQL Server 2012 Data Tools. These tools can both deploy new databases or automatically generate upgrade scripts based on differences between two databases.

Deployment to Windows Azure

The previous discussion around deployment options applies mainly to on-premises solutions. With the cloud becoming more and more accepted as a hosting platform, we need to be able to implement deployment workflows in Windows Azure as well.

We will not cover deployment to Windows Azure, but if you are interested in a solution the following resources should get you started:

- Continuous Delivery for Cloud Applications in Windows Azure:

http://www.windowsazure.com/en-us/develop/net/common-tasks/continuous-delivery

- Continuous Delivery to Windows Azure by Using Team Foundation Service Preview:

http://www.windowsazure.com/en-us/develop/net/common-tasks/publishing-with-tfs

- TFS Build Extension activities for Windows Azure:

http://tfsbuildextensions.codeplex.com/wikipage?title=Getting%20started%20with%20the%20Windows%20Azure%20activities&referringTitle=Documentation

Running Automated Tests as Part of the Build

We have now looked at different strategies for deploying our application. With deployment comes the opportunity to update our development, test, and even production environments whenever we want. To make this automation predictable it is a good idea to add automated tests to verify that the new deployment works as expected.

There are several options for running automated tests as part of a build, the most common are:

- Running tests on the build server.

- Running UI tests on the build server.

- Running tests on a target server.

- Running UI tests on a target server.

Let’s now look at how to set up each of these options.

Running Tests on the Build Server

The first option is to run all tests on the build server. We have seen how easy it is to integrate unit tests in the build process (Chapter 26) and running other automated test types is necessarily not a big problem. We will just add the automated tests to the build just like the unit tests. We do however need to deal with any dependencies the tests need may have (integrations with other system, database and so on). We covered ways to handle dependencies in chapter 16 above what is covered in this chapter.

In general, we recommend avoiding running tests that introduce additional software on the server because it can affect the performance as well as stability of the build server. In particular, running interactive tests on the build server requires the build server to run in a special mode as described next.

Figure 27-2 shows the logical setup if we want to enable running tests on the build server.

Figure 27-2.  Automated testing on the build server

Running UI Tests on the Build Server

If we want to run tests against the user interface we typically need to have an interactive session running that can open application windows as part of the test. If we choose to run UI tests on the build server this means we need to be running the build service interactively. This is generally not recommended because it affects the way the service is run. But if we decide to set up the build service for interactive testing, we do so from the Build Service properties in the TFS Administration Console (see Figure 27-3). A good practice if you want to run interactive tests on the build server is to set up a dedicated build service for this and use build agent tags to direct builds with UI test to that particular environment.

Figure 27-3.  Configuring the build service to run in interactive mode

Figure 27-4 shows the TFS Build Service running in interactive mode. As the display text says, the service will run until someone exits the service. If someone logs off the interactive session or restarts the machine, it will unfortunately result in the build service not running.

Figure 27-4.  TFS Build Service running in interactive mode

Running Tests on a Test Environment

Running automated tests on a test environment requires us to create a test environment on which to run the tests. In Chapter 23 we covered how to use Visual Studio Lab Management to create test environments, but of course any physical or virtual machine can be used as a test environment.

Next we need a way to configure the build to run tests on a remote machine. With a test environment that is managed by TFS and Lab Management many of the pieces for running tests in a remote environment are already in place. Later in this chapter we will implement a complete build-deploy-test workflow implemented on the Lab Management services. If we are not using Lab Management, we can run MS Test remote and have the result published to TFS. See Chapter 22 for details on running MS Test from a command-line.

Figure 27-5 shows how we can design a lab for automated testing, both at the build level using unit tests and at the functional level using other types of automated tests.

Figure 27-5.  Automated testing on a build server and test environment

Running UI tests on a Test Server

If we want to run user interface tests on the test server we have the option to configure the Visual Studio Test Agent to run as an interactive process (see Figure 27-6). We can set a number of settings for the interactive process:

Figure 27-6.  Configuring the Test Agent to run interactively

- Specify the user that will run the interactive process: For details on the requirements for the test agent account see http://msdn.microsoft.com/en-us/library/hh546459(v=vs.110).aspx.

- Have the test agent log on as the interactive user automatically: This is a good option if we want the labs to run unmanaged but it may have implications on corporate policies because the workstation will run unlocked.

- Disable screen savers to run: Most UI test will fail if the desktop session is locked, so by default we should disable screen savers.

Fail a Build on Test Failure

By default, failing tests will not fail a build, only mark it as a partially successful build. There are reasons for completing the build regardless of test outcome for instance if we really need to get a new build out but tests are failing and we do not have time to fix them. But if we allow this to happen it is very easy to get into a situation where it is accepted to have failing tests and we soon lose the quality of the automated testing. We recommend failing builds on test to be the default and rather customize the build workflow to allow the behavior to be changed when starting a new build. Figure 27-7 shows how we configure the build definition to fail the build on test failure.

Figure 27-7.  Configure a build definition to fail the build on test failure

Running a build with failed tests results in a failed build, as Figure 27-8 shows.

Figure 27-8.  Build failed because of a failed test

Implementing the Build-Deploy-Test Workflow

Now we have looked at options for our deployment and we have gone through the ways to run automated tests. We are now ready to combine our skills of creating automated tests, creating test environments and setting up automated build processes to create a complete end-to-end scenario for the complete build-deploy-test workflow.

We will focus on using as much of the existing infrastructure in Visual Studio 2012 as possible, so we will base our implementation of a BDT workflow on TFS Build and Visual Studio Lab Management. A good BDT process allows us to quickly manage a large number of environments because the whole deployment process is now automated and with the validation from automated tests we can feel confident that the machines get updated in a controlled way. In other words we are ready to go into continuous delivery of or software!

Implementing a BDT Workflow

Even though it can be a complex task to implement a build-deploy-test workflow, a lot of work has been done for us with Visual Studio Lab Management. A BDT workflow in Lab Management is mostly implemented using shared infrastructure such as the build and test services. This means we can focus on what needs to be done and not so much how to do it. Figure 27-9 illustrates the built-in workflow in Lab Management.

Figure 27-9.  A build-deploy-test workflow in Visual Studio 2012 Lab Manager

To make it easy to get started with a continuous deployment workflow, Team Foundation Build comes with a build template tailored for Lab Management that implements a build-deploy-test process.

The Lab Management build template LabDefaultTemplate.11.xaml contains the following core components:

- Capability to revert a lab environment to a known state.

- Select or run a compile build to get a version of the application to deploy and test. This is a normal TFS build that would include running unit tests if appropriate.

- Run deployment scripts to deploy the application. Test agents are used to run the scripts locally on each machine in the environment.

- Take a snapshot of the environment so we can revert to a clean state whenever we want without having to do a new deployment.

- Run tests on the machines in the lab environment.

- Create a build report.

In the sections that follow we will set up a BDT process for Expense Reporting to be used for deployment to the team’s test environment. It is important that the test environment is smoke tested using a set of the automated tests before letting testers get access to it. The test environment is based on SCVMM so snapshots can be used where appropriate. Physical or virtual machines not managed by SCVMM can still be used to create a lab environment and use the LabDefaultTemplate, except for snapshot functionality, which will be disabled.

Designing the BDT Workflow

When discussing the deployment process for the Kangreen Expense Reporting application we got the following high-level description of the desired process from the Configuration Manager:

“First we build a new version of the application to be used for deployment. Then we revert the environment to a known state to have a common baseline for the continuous delivery workflow. Next we deploy the solution using a command script. This can be the same script used to manually deploy to local developer machines or into the production environment. We then take a snapshot before the tests are run to have a known state to be used before any test suite is executed. Having this snapshot allows us to get back to a good state whenever we want to during testing without having to do another deployment. Finally we run a BVT test suite to validate the deployment.”

Based on these requirements we have created the topology in Figure 27-10.

Figure 27-10.  The build-deploy-test topology for Kangreen Expense Reporting

In summary, the figure shows the following:

1. The TFS build server hosts the build controller and agent, as well as the test controller.

2. The test lab is created using Visual Studio 2012 Lab Management and consists of a client and a server machine. The server machine hosts the web application, the services, and the database.

3. The client has a test agent installed to run the test, as well as to collect diagnostic data.

4. The app server has a test agent installed for diagnostic data collection from the test runs.

5. The build controller orchestrates the overall build workflows.

6. The build agent runs the compile build.

7. The build agent runs the BVT workflow.

8. The test controller orchestrates the test runs.

9. Snapshots can be created over the complete lab environment to create a consistent view of the test lab (including database state since the database is part of the environment).

Let’s now go through the set up of the BDT workflow for Kangreen Expense Reporting.

Implementing the BDT Process

We will use as much of the existing Visual Studio 2012 features as possible to implement the BDT process. To implement the process we need to

1. Create an environment for testing. This is described in Chapter 23.

2. Create automatic tests to run as part of the build. Chapters 16 and 22 cover automated testing.

3. Create a build definition to create a new version of the application. We covered build automation in Chapter 26.

4. Create a build definition for the deployment and testing part of the workflow.

We will consider steps 1-3 complete and focus on step 4 to create the BDT workflow.

Create a BDT Build Definition

To create a build-deploy-test build definition we follow the procedure for a default build with the following exceptions:

- Workspace: May not be necessary as the BDT workflow will only deploy and test. If the BDT workflow has dependencies that the BDT build uses (for instance deployment scripts), then we may need to setup the workflow as well.

- Build Defaults: No drop folder is needed as the BDT workflow normally will not produce any output.

- Process: Select the LabDefaultTemplate.11.xml as the build process template.

Figure 27-11 shows our BDT workflow being setup.

Figure 27-11.  Creating a build-deploy-test build definition

A special requirement to run Visual Studio Lab Management BVT workflows that queue a compile build as part of the workflow is that there needs to be at least two agents available. If only we have one build agent then the deployment build will be blocked by itself waiting for the build workflow to begin. Figure 27-12 shows a Build Controller with two build agents.

Figure 27-12.  Configuring the build service to have at least two agents

The only required argument in the LabDefaultTemplate is that we open the configuration wizard and configure the build as described in the following section.

Selecting the Target Environment

The first step in the Lab Workflow wizard is to select the lab environment. We need to select an existing Lab Management environment and we can optionally configure the workflow to revert the environment to a specific state before the deployment is done given that the environment is using SCVMM (see Figure 27-13). See Chapter 23 for details on how to work with Visual Studio Lab Management to create test and lab environments.

Figure 27-13.  Selecting a Lab Management target environment

Selecting Application Build to Deploy

Next we select the build to deploy. We have the following options for picking a build that the workflow should use:

- Use a Team Foundation build: Here we can either queue a new build as part of the BDT workflow or we can pick an existing build.

- Use a build from a specified location: We can also choose to deploy a build from a selected location; this is useful if other tools are used to produce builds to deploy.

Figure 27-14 shows the build to deploy configuration dialog.

Figure 27-14.  Setting up the build to deploy in a lab workflow

Defining the Deployment Steps

With target environment and build to deploy selected, we can now add the deployment steps (see Figure 27-15). The Lab Workflow is very generic; it allows us to run commands or scripts on each of the machines in the environment. We can address the machines either by role or by name—whatever is most suitable.

Figure 27-15.  Configuring deployment for a lab workflow

The scripts are run locally on the target machine so it needs to be available as specified on the target machine. If necessary, we can also specify a working directory.

When we add deployment commands we may need to pass in arguments from the build process. Currently we can use the following properties:

- $(BuildLocation): This is the location of the build. If you specified to use the build from a shared location, then this variable represents that path. For the other options, this is the full path for the build based on the configuration that you selected to build and the build drop location in the build definition. If you build your application as part of your workflow, you can use this to access the latest files that were created by that build.

- $(InternalComputerName_ <VM Name>): This is used to obtain the computer name for a virtual machine that is part of a virtual environment. You might know the virtual machine name, but not the computer name. If you have a deployment script to set up a web server that requires the computer name, you can pass this as an argument to the script. For example, if the virtual machine name for the web server was VM1 and the computer name was MyWebServer, you would type $(InternalComputerName_VM1) as the argument for your script and this would pass the value MyWebServer to your script.

- $(ComputerName_ <VM Name>): This is the fully qualified domain name of the virtual machine. This can be used to access the computer even from outside the virtual environment. You might want to pass this as an argument to set up a web server. For example, if the virtual machine name for the Web server was VM1, you would type $(ComputerName_VM1) as the argument for your script to pass the fully qualified domain name of the virtual machine.

In the example in Figure 27-16 we locate the deployment script on the drop folder of the build. We also pass in the build location (drop folder) as an argument to the deployment script so the script can find the application to deploy.

The following simple deployment script is sufficient to deploy the built application to the app server.

REM ---

REM --- Map arguments passed from build

REM ---

SET DropPath = %1

SET DeploymentPath = c:\ExpenseReporting

REM ---

REM --- Deploy application

REM ---

RD %DeploymentPath% /S /Q

XCOPY %DropPath%\_PublishedWebsites\MvcMusicStore %DeploymentPath% /S /Y /I

Adding Tests

Finally we add the tests we want to run as part of the BVT workflow (see Figure 27-16). The tests to run need to be defined in a test plan in Microsoft Test Manager and we can then filter out tests to run by test suite and configuration, for instance to specify that for this BVT workflow we will only run Windows 8 tests. Chapter 21 contains information on how to create and manage a test plan in MTM.

Figure 27-16.  Configuring testing for a lab workflow

Running the BDT Workflow

Running the BDT workflow is just like running any other build in TFS, we can use the Team Explorer – Builds to queue a new BDT build manually or we can set the build definition to trigger based on other criteria. Figure 27-17 shows a build report for a completed lab workflow build.

Figure 27-17.  Build Report for a completed build-deploy-test workflow

Note how nicely we get information about which build was deployed to which environment. We also get a good view over the test result.

This completes the setup of a build-deploy-test workflow. As we can see it really ties together the work we already have done, which means little extra work is needed to get a continuous delivery process in place. This is just what TFS and ALM is all about; adding separate practices is good but integrating them can really create an impressive process!

Customizing the Lab Build Template

Even though there is enough flexibility in the default template to solve much of the deployment and testing work there are always scenarios where you want to do something different (or even better). Some common customizations are

- Database backup and restore.

- Inserting test data.

- Start/Stop the lab environment.

- Finding an environment for a configuration based on tags.

In Chapter 26 we looked at how to customize and extend a TFS build workflow; customizing a Lab Workflow just the same, we are only working on a different part of the process.

For more details lab workflow customization, look at the ALM Rangers Visual Studio Lab Management Guide (http://vsarbuildguide.codeplex.com) and the TFS Build Extensions (http://tfsbuildextensions.codeplex.com/) open source project.

Lab Activities

Just as for normal builds there is a set of built-in lab activities to support the LabDefaultTemplate, which we can take advantage of when we want to customize or extend a lab workflow.

Table 27-1 shows some of the built-in lab workflow activities, for a complete list see http://msdn.microsoft.com/en-us/library/ff934562(v=vs.110).aspx.

Table 27-1. Built-in Lab Workflow Activities

Activity

Description

CreateLabEnvironment

Creates a new environment from a stored template and deploys it on the specified host group.

ExecuteRemoteTestRun

Creates a test run on an environment and waits for it to finish.

GetBuildLocationAndBuildNumber

Returns the build location and build number for a build, deploy, and test scenario.

ReleaseEnvironmentFromDeployment

Releases the environment from deploying a build.

ReserveEnvironmentForDeployment

Marks the environment as deploying a build.

RestoreLabEnvironment

Restores the environment to the specified snapshot.

RunDeploymentScript

Runs the deployment script.

RunWorkflow

Queues a new workflow on a build controller of a build server.

SnapshotLabEnvironment

Takes a snapshot of an environment.

StartLabEnvironment

Starts the environment.

StopLabEnvironment

Stops the environment.

WaitForEnvironmentReady

Waits for the enviroment to get in a Ready state.

Lab Activities in the TFS Build Extensions

In Chapter 26 we introduced the TFS Build Extensions, a community driven project for TFS build extensibility (http://tfsbuildextensions.codeplex.com). The project contains several build activities for lab management workflows as well; some of them are listed in Table 27-2. Chapter 26 describes what is needed to integrate the TFS Build Extension activities into a custom build workflow.

Table 27-2. Lab Build Activities in the TFS Build Extensions

Activity

Description

GetEnvironment

Retrieves a Lab Management environment by its name.

ListEnvironments

Retrieves a list of all of the environments that matches the specified set of Tags.

LockEnvironment

Locks a specified environment, indicating that the environment is now “owned” by the requesting build.

PauseEnvironment

Set a Team Foundation Server Lab Management Lab Environment in a Paused state.

ShutdownEnvironment

Shuts down a Team Foundation Server Lab Management Lab Environment.

UnlockEnvironment

Unlocks the environment specified.

In the next section, we will look at how we can use the TFS Build Extension.

Adding Support to Start and Shut Down the Environment

As an example we will look at how to create a customized lab workflow to implement the logic for starting the environment (if not running), running the deployment as usual, and then shutting it down (if we started it). We will use activities from the TFS Build Extension project as building blocks.

We customize the lab workflow by making a copy of the LabDefaultTemplate11.1.xaml workflow file.

Start Lab If Not Running

First, we need to add logic to start up a lab environment if it is not running. Again, a pretty simple customization once we have designed what the logic looks like. We need to do the following to the default lab workflow:

- Define a variable DidStartLab (bool) to hold a flag as we started the environment so we know if we should shut it down later.

- Define a variable LabState (LabEnvironmentStatus) to hold the current lab environment status. (LabEnvironmentState).

- Add a workflow sequence after the GetEnvironment activity in the default template.

- Add a GetEnvironmentStatus activity to get the current environment state. Use the LabEnvironmentUri variable to identify the environment.

- Add an If condition to test if the lab state is stopped.

- In the true lane add a StartLabEnvironment activity. Use the LabEnvironmentUri variable to identify the environment.

- Set the DidStartLab flag to True using an Assign activity.

- Write a message to the build log using a WriteBuildMessage with BuildMessageImportance set to High.

Figure 27-18 shows the complete workflow for starting a lab environment if it was in a stopped state.

Figure 27-18.  Workflow logic to start a stopped lab environment

Shutdown Lab If Started by the Workflow

The other end of this exercise is to shut down the lab environment if we started it in the previous step. For this we can use the TFS Build Extension activity ShutdownEnvironment because the built-in lab activities only allow us to stop an environment. In our case, a graceful shutdown is more appropriate.

We can add the logic to shut down the lab after the Run Tests on Environment sequence in the default template. The custom workflow will do the following:

- Check whether we started the lab environment by testing on the DidStartLab flag set in the Start Lab sequence earlier.

- In the true lane, add a ShutdownEnvironment activity. The activity takes the name of the environment to shut down, which we can get from the LabEnvironment.Name variable.

Figure 27-19 shows the custom workflow for shutting down a lab environment.

Figure 27-19.  Workflow logic to shut down a running lab environment

Custom Lab Workflow Parameters

When we created the BDT workflow earlier in this chapter we saw how there are a few predefined macros we can use to pass information to the deployment scripts. But if we would like to create a custom workflow argument and be able to pass that into the deployment script, the default template will not allow us because the known macros are validated when we add the deployment command.

We can solve this problem in a couple of ways

- Replace the Lab Workflow Wizard dialog. This gives us the power to do whatever we want to configure the process but it is quite some work to implement the code in the wizard.

- Extend the lab workflow with custom logic to handle a custom argument.

Because the second option seems to require less work let us take a look at what this could look like.

The customized deployment template extends the default lab template by allowing us to provide custom build process parameters using $[] instead of $() to reference the arguments in the Lab Workflow wizard. The custom deployment arguments are replaced before the execution of the deployment command. Figure 27-20 shows how we can use the custom parameter in the deployment script arguments.

Figure 27-20.  Using custom parameters in the Lab Workflow Wizard

The custom deployment parameters are added to the build definition as name-value pairs as shown in Figure 27-21.

Figure 27-21.  Passing custom deployment parameters to a lab workflow

To implement this customization, we can add the workflow logic before the Lab Agent scope, where the deployment scripts are run (see Figure 27-22).

Figure 27-22.  Workflow logic replaces custom lab workflow macros with parameter values

In the Initialize Custom Deployment Arguments section each custom argument is split on the “=” sign. The argument name is then replaced with its value in the command string defined in the lab process wizard (see Figure 27-23).

Figure 27-23.  Workflow logic to shut down a running lab environment

Tip  The ALM Rangers Visual Studio Lab Management Guide (http://vsarbuildguide.codeplex.com/) contains several examples of lab workflow customization, including a custom template for managing custom deployment arguments.

Summary

In this chapter we have seen how Visual Studio 2012 Lab Management can be used to create a continuous delivery process. What is really nice is that we can achieve this without exceptional work; it is more a work of adding different good practices into a great end-to-end workflow.

And if we choose to build our own deployment solution we have looked at different solutions to packaging, deployment, and testing—all of which can be integrated with the TFS build system.

CHAPTER 28

Release Management

Software systems, the development processes. and project resources have a tendency to be distributed more and more in recent years. This also means that they have become more and more complex and specialized. Most often software products are in an ongoing development cycle with testing and releases integrated, making it even more complex to manage. We can also see that the platform our applications and system run on is becoming more and more complex with the addition of cloud-based systems. There are a lot of pieces that are floating around and must fit together as seamlessly as possible; otherwise we might not get the value or success we expect from our projects. This is why release management becomes more and more important.

Release Management

Release Management (RM) as a discipline is relatively new, but it is growing fast. RM is used to manage software releases. Many companies need to have one or more co-workers dedicated to oversee the integration and flow of development, testing, deployment, and support of the systems the organization has. In the past this has often been the job of project managers, but they rarely have the time or opportunity to focus on their projects and also look at company strategy when it comes to releases. Hence, the introduction of release managers.

Release managers must have a general knowledge of the software development process and all aspects of it; they also need to understand operating systems and platforms applications. What’s more, they should have some appreciation of the business side so that the releases can fulfill business needs.

Since agile development processes like Scrum have been more and more popular the number of releases of an application or system has increased. This further stresses the importance of release management and its friends in IT operations. One of the best benefits of the agile introduction (apart from better success rate in projects) is that development teams and operations teams have been forced to collaborate more closely in order to support release events.

Agile Release Planning

Let’s take a look at the agile release planning from Chapter 8 again. The Product owner (Fiona) used the initial backlog to look for themes in the user stories. In our example she came up with a few:

- Expense report management

- Search functionality

- User management

- Customer management

- Project management

- Smartphone availability

Fiona quickly saw that three themes were going to be part of the first sprint. The Expense report management, User management, and Customer management were all part of the first sprint according to the initial sprint planning.

Considering that there were many chores in the first sprint, she knew that all three would not fit in. She aimed at getting the Expense report management theme done.

Fiona also knew the initial theoretical velocity (10 story points), which she used as an input for how much work she could expect in each sprint. With 44 story points in total, the project would take 4.4 sprints to complete. She rounded this up to 5 sprints (keep in mind that this is an extremely simplified example).

Using this information as an input, she could start to plan when to release what feature and came up with a release plan:

- Expense Report Management in Sprint 1 and 2

- User Management, Project management in Sprint 3

- Smartphone availability in Sprint 4

So even if the Product Owner is responsible for the release planning of a project with only one Scrum team, imagine the complexity if there was five, six, or more other projects ongoing with many teams working in parallel. This is why release management is so important. Figure 28-1 shows an example of a release plan using the TFS Web Access product backlog with forecasting enabled.

Figure 28-1.  Release Planning in TFS Web Access

Release Management in Visual Studio 2012

Part of release management is the release planning. Besides planning, there are many other activities that the release manager needs to do that can be solved by tools in Visual Studio 2012.

As a way to get started with effective release management we have put together this short list of what we think are important practices (with cross-references to the chapters in this book where we cover the details):

1. Implement an efficient version control system (Chapter 15).

2. Implement a good branching strategy. This is something we will dive into later in this chapter.

3. Implement automated builds (Chapter 26).

4. Write unit tests (Chapter 16).

5. Automate application deployment (Chapter 27).

6. Automate database deployment (Chapter 27).

7. Establish good procedures for managing test environments (Chapter 23).

8. Automate build verification tests (Chapter 22).

9. Automate inspections (Chapters 17, 26).

10. Implement practices for feedback (Chapters 17, 26).

11. Release notes.

12. Requirement versioning.

13. Test case versioning.

14. Build versioning.

Release Notes

Release notes are documents that primarily describe the changes in a product between two releases. A release notes document should contain information about new features and corrections, as well as instructions on how to apply the update. These documents are distributed with the software, very often when the product is still in the development or test phases. If the product is already released to customers the release notes accompanies the product manual especially when a bug fix or an enhancement is being released to the product.

The release notes details the changes or enhancements made to the software and are communication documents that are shared with customers within as well as external to the organization (if the software is publicly available). The release notes are also intended for the people managing the configuration of different software installations and to those who deploy patches to installed software.

Previously release notes were written by technical writers, but nowadays more and more other members of the development team such as developers or testers write them. One of the reasons for this is that more and more teams use Scrum, which also mean that roles become more diffuse and the knowledge and skills are spread among all team members. Another reason is that writing release notes requires knowledge of the complete development process, including tests performed and if you are not part of the development team (including testers of course) these notes can be hard to write.

Let’s now see how Visual Studio can help us to get information we can use in our release documentation.

Release Notes in Visual Studio 2012

If we want to use Visual Studio and TFS as the source of release notes information we should implement the solution on work items and associations to. There is already implicit support for this in the standard templates, but we typically need to apply our model over the basic functionality. What we can use as a start is shown in Figure 28-2.

Figure 28-2.  A model for managing release notes with TFS

The model should be read as follows:

- A release plan is modeled using Iterations (sprints) following the ideas around release planning described earlier in this chapter.

- Features and corrections (PBIs and Bugs work items) are assigned to the plan through the iterations.

- Implementation (documents, code, etc.) is associated with work items as links and changesets.

- Release note information is documented in the work items.

- Changesets contains the changed files in the repository.

Based on this convention we can query the TFS database and create reports like the Excel report in Figure 28-3. The query filters on the team project (Expense Reporting) and the current iteration (sprint 1). For the release notes we need to choose which PBIs to included in the report, typically we include only done PBIs (excluding not started and not completed PBIs) We cover custom reporting in TFS in Chapter 32 is you want to know more about that.

Figure 28-3.  Release notes for a sprint using Excel and the TFS data warehouse

Change Details in the Build Report

One of the features of TFS build is that the build report can be used to see change information similar to the model described in Figure 28-2. The default behavior of a TFS build is to associate build information with changed work (work item and changesets) and include the delta changes between the current and the previous build in the build report. Figure 28-4 shows an example of a TFS build report.

Figure 28-4.  Build report for a release build

This is not a replacement for a proper release notes report but it contains useful information that testers and change managers can take advantage of without running a separate report.

- Test run information: This section contains information about the tests run as part of the build. We can expand the section to look at the details and even download the test results to Visual Studio for further analysis (and trouble-shooting).

- Code coverage from test runs: Similar to the test run information this section shows code coverage gathered from the test runs. The code coverage data can also be downloaded to Visual Studio if we want to analyze the result at a finer detail.

- Associated Changesets: The associated changesets section lists the changesets committed to TFS since the last good build. This gives us a nice summary of what has been changed and can be used by change managers to understand what needs to be deployed if this build is to be released. We can click on the changeset links to get to the changeset details.

- Associated Work Items: The final section in the default report is the list of associated work items. The concept is the same as for the changesets and gives us a good log over which features have been resolved since the last good build. We can navigate to the work item if we want to look at details. Testers can use this section to understand whether the build contains enough new features to be deployed to a test environment.

TFS Build Extensions Build Notes

Because the data for release information is stored in TFS we can develop custom reports to show the information. Or we can create custom tools that give us data for deeper analysis. One nice example of this is the Build Notes feature in the TFS Build Manager, which is a part of the community driven project TFS Build Extensions (http://tfsbuildextensions.codeplex.com/).

The Build Notes tool works similar to the default build report but creates a Word document as a result. Figure 28-5 shows how we can generate a build notes report based on a build in TFS (using the TFS Build Manager).

Figure 28-5.  Build Manager to generate build notes

We can select what information we want to include in the build notes report (see Figure 28-6). For instance, if we only care about work items and changeset information, then select that information.

Figure 28-6.  Available options for creating build notes

The tool then analyzes the build and generates a Word report such as the one shown in Figure 28-7.

Figure 28-7.  A complete build notes report

Versioning

An important aspect of release management is how to handle versioning of the artifacts we work with in our development and maintenance projects. In this section we look at how the major assets in a development project can be handled to support multiple versions, specifically requirements, source code, test cases, and builds.

Versioning Requirements

The first category we will look at is requirements. In TFS we primarily have two ways of managing requirements; as a document in SharePoint or as Work Items.

Versioning Documents in SharePoint

SharePoint is a great tool for managing documents and document workflows. As a part of this feature set SharePoint also supports document versioning, which is something we can use in our strategy for document versioning.

To enable versioning on documents in SharePoint we can turn the feature on when we create a document library as shown in Figure 28-8.

Figure 28-8.  Enabling document versioning in SharePoint

Enabling versioning at this point sets the default settings for document versioning. If we want to change the versioning settings, enable versioning on a single document or on a document library at a later point, we can edit the settings as Figure 28-9 shows (navigate to Library Settings→Versioning settings→Document Version History for the settings for a document library).

Figure 28-9.  Configuring document versioning in SharePoint

If we then select the history for a versioned item in SharePoint we can see the version history (see Figure 28-10) and open a historical version to look at the earlier state.

Figure 28-10.  Viewing Ddocument Version History in SharePoint

There is no built-in support for generating the differences between two versions so this is something we need to solve on top of the versioning support in SharePoint. A good way to deal with this problem is to use the Track Changes feature in Microsoft Word (see Figure 28-11).

Figure 28-11.  Using change tracking in Microsoft Word

A good versioning strategy can then be to use versioning in SharePoint with change tracking in Word. For each major release the tracked changes are resolved and we start with a clean document for the new release. If changes to the documents are too difficult to track over releases, then we can of course create copies of the documents but be aware of the manual work if we want to move changes from one document to another.

Versioning Work Items

Another way to manage requirements in TFS is to use work items. It is very easy to get started with work items for requirement management (as described in Chapters 7 and 8), but it is not as easy to manage versions of work items. Sure we can view the history and figure out the changes between two updates, but that is not going to help much if we need to maintain multiple versions of requirements. Also, it is not possible to lock a work item, which is often desired for requirements.

One way to solve this problem is to copy work items when a breaking change is made. We can use the Create Copy of Work Item tool to quickly create a new work item from an existing as shown in Figure 28-12.

Figure 28-12.  Copying a work item

When we copy a work item we can choose the target team project and work item type. This can of course lead to problems if the templates are incompatible but in this scenario it’s not a problem. The result of a copy is shown in Figure 28-13.

Figure 28-13.  Result of copying a work item

Notice that not all data is copied, for instance only the Related and Tested By links are re-created and not links to for instance tasks. The idea here is that the re-created links are the static ones and the rest would be things we build up new in the cloned work item.

Note  It is also important to notice that the linked work items are not copied; we only get the links copied. So if we really wanted a copy of the related information, we need to copy each linked item and create a new link to the clone. If you need to do this more than occasionally, we recommend looking at the TFS API and automate the cloning.

Versioning Code

The next area to cover is how we manage multiple versions of code. This is technically a simpler problem because TFS has support for creating multiple branches of our code base, which we can use to support the different version. We covered the fundamentals of branch and merge in Chapter 15, so if you want to know more about the practical use of the tools, jump back to that chapter.

Implementing a Branching Strategy

In the following section we will look at two ideas for managing concurrent work; a traditional branch pattern and a single branch approach.

The Typical Branch Pattern

Figure 28-14 shows a typical pattern for branching and merging. The figure contains the following different branch types:

Figure 28-14.  A typical branch and merge pattern

- Main: The parent version of the code base representing a stable version used to produce the next release version.

- Development: This is a child branch from Main used for development work.

- Release: This is also a child branch from Main, this time for stabilizing and supporting a released product version. In the release branch we can make hot-fixes without risking mixing in new or changed features. We typically create one release branch for each release.

- Patch: A child branch from a Release branch. A Patch branch is used to create minor versions or service packs if we want to support multiple minor versions of our product.

When changes are made we can use the version control tools to merge changes from one branch to another. The principle we recommend is to only allow merging from more stable code to less stable code, which means we never merge from Main to a Release branch, if we want to port a feature from development into a released version we will implement the changes in each branch by hand. The main reason is to minimize the risk of accidentally merging incomplete work. Of course if we implement the new feature in the Release branch, we can merge to Main and then to Development using the merge tools as usual.

Single Branch Pattern

There is a problem with branching that becomes more of a problem when we want to implement a continuous delivery process. Merging code will never be possible to automate 100% because once in a while conflicts will occur. When this happens the process stops until the issue is resolved, this prevents a good flow in the delivery process. One way to address this is to avoid branching and do all development in one single branch. But how can we solve the problems that branching is there for us to fix? Well, instead of isolating work in branches we isolate it using feature toggles, or conditions, in the code base. We can then use the switch to enable a feature when it is ready but it may have been released to customers a long time ago. This approach has other interesting features: for instance we can develop a new functionality and release to customer. Then we can use the feature toggle to enable the new functionality to a pilot test group without having to setup dedicated environments for this.

Martin Fowler has written an article about feature toggles available here: http://martinfowler.com/bliki/FeatureToggle.html.

If you should use the typical branch model or go for the single branch approach depends to a great extent on the situation you have in your project. In general, the typical branch model is a safe bet. We pay a price for the branch maintenance, but it gives us a controlled way to handle changes in multiple versions of the code base.

TFS Branching and Merging Guide

The TFS Branching and Merging Guide is another Visual Studio ALM Ranger project. This project contains deep, practical guidance around code management using TFS with scenarios ranging from basic needs to multiversion maintenance configurations.

You can download the latest version of the guide from http://vsarbranchingguide.codeplex.com.

Versioning Test Cases

Next up is versioning of Test Cases. We focus on two scenarios: copy test cases from an existing plan to re-use a structure from one test plan in another. The other scenario is to clone test cases to create new versions of the test cases.

Copy Test Cases from an Existing Test Plan

To re-use assets in MTM we can use the copy test suites from another test plan function to import test plan structures we have created earlier. This is very convenient, especially when working with a concept where we evolve test suites over iterations as the product is implemented. Figure 28-15 shows how we can initiate a copy of test cases from an existing test plan

Figure 28-15.  Import test cases from an existing test plan

Next we select the test plan and suite to copy (see Figure 28-16).

Figure 28-16.  Import test cases from an existing test plan—selecing suite to copy

Finally, the test suite is copied into the current test plan (see Figure 28-17).

Figure 28-17.  Import test cases from an existing test plan—complete

Be aware that it is the references to the test cases that get copied, not the test cases themselves. So changing a test case in one test plan will affect all other test plans!

If we want to version the test cases, we need to clone them, which is what we will look at next.

Clone Test Cases

Earlier in this chapter we looked at versioning work items and how the copy work item command can be used to create a cloned work item. Unfortunately, linked items are not copied so versioning of work item structures needs to be solved by hand or using another tool.

For test cases this has been big enough a problem to have its own tool to solve the problem. We can use the tcm.exe command-line tool to clone a test suite instead of just copying.

To clone the test cases we need to have a source test suite to locate the test cases to clone. Then we need a target test suite in another test plan to insert the cloned items into. All test cases in the source suite get copied. We can also optionally set one field during the cloning, something we may use to distinguish the cloned item from the source if necessary.

Figure 28-18 shows an example of how to locate a test suite Id needed for the copy.

Figure 28-18.  Finding the test suite id in MTM

With the details at hand we can now run the clone command:

tcm suites /clone /suiteid:15 /destinationsuiteid:18 /collection:http://tfs:8080/tfs/defaultcollection/teamproject:”Expense Reporting”

The clone operation can take some time on a large set of test cases; if we want to check the progress we can run the following command:

tcm suites /clone /status:1 /collection:http://tfs:8080/tfs/defaultcollection /teamproject:”Expense Reporting”

Clone operation 1 has succeeded. Summary information:

Started on: 2012-07-13 00:41:19

Started by: Administrator

Test cases cloned: 6

Shared steps cloned: 0

Completed on: 2012-07-13 00:41:22

The result is shown in Figure 28-19. Note the new work item Ids as an indication of a successful cloning.

Figure 28-19.  Result of cloning Test Cases

Versioning Builds

A small but important piece to keep in mind when implementing a versioning scenario is the builds. As we create new branches of our code we also need ways to easily build out a version of that branch.

The build name should match the branch name to make it easy to relate a built version to the branch it came from.

We recommend creating a build definition for each active branch. If a branch is not frequently changed it may be sufficient to keep the release build but the important thing is of course to have routines to that when needed it is easy to build a new version.

In Chapter 26 we looked at maintaining build definitions and how we can use the TFS Power Tools to clone a build definition. Whether we use a tool or not to clone a build definition we should to make the following adjustments for the new build:

- Change the name and description to match the branch.

- Change the workspace mapping to the new branch.

- Change the drop folder to reflect the new branch.

- Change the process settings to refer to items (projects typically) in the new branch.

Summary

Release management is an important part of any software organization, yet often this is something that is done with little attention. We typically spend much more time on requirement gathering, backlog grooming, and application design then on activities such as optimizing the release process, looking at how we work in concurrent versions, or how we promote changes between branches. Unfortunately we cannot avoid these tasks so it is a high risk that we get an error-prone or time consuming process around the release management activities.

In this chapter we have looked at important practices for effective release management and seen that many of the ALM activities we have looked at throughout this book are tied together in the release management process. For example, automating the release notes creation can not only reduce the time it takes to prepare a release but also enforce a unified way of working with requirements, work items, code, and builds. We also looked at versioning and how this is something that needs to be applied to the whole development process, not in practices in the respective practices. With a consistent way to manage versions, we can improve quality and speed up the delivery process.

CHAPTER 29

Metrics and ALM Assessment

This is the final chapter in our assessment and metrics series. We will cover some metrics and reports for release management, including the build process.

Metrics

Much of the metrics we can get from TFS are presented as reports. Using these figures as KPIs we can easily compare our progress when it comes to build quality.

A quick look at Information Technology Infrastructure Library (ITIL) (http://www.itilnews.com/ITIL_v3_Suggested_Release_and_Deployment_KPIs.html) will give us some other KPIs we can use. If we want to use them we might need to create our own reports to automate the retrieval of this information. ITIL mentions among others these KPIs:

- Number of software defects in production, which is the number of bugs or software defects of applications (versions) that are in production

- Percentage of successful software upgrades (excludes full installations)

- Number of untested releases (i.e., not tested and signed-off)

- Number of urgent releases

- Average costs of release, where costs most likely are based on man-hours spent

Note  The Information Technology Infrastructure Library (ITIL), is a set of practices for IT service management (ITSM) that focuses on aligning IT services with the needs of business. ITIL describes procedures, tasks, and checklists that are not organization-specific, used by an organization for establishing a minimum level of competency. It allows the organization to establish a baseline from which it can plan, implement, and measure. It is used to demonstrate compliance and to measure improvement.

Standard Reports

There are three reports in TFS 2012 that we can use to get information about our builds:

- Build quality indicators

- Build success over time

- Build summary report

The Build quality indicators (see Figure 29-1) report shows a summary of some important values for our builds. Using this data we can see whether we are close to releasing the build. Some of the information this report shows:

Figure 29-1.  Quality Indicators report

- Active bugs. How many active bugs that existed at the time of the build.

- Code churn. The number of lines of code that has been added, removed, and changed in the check-ins before the build.

- Code coverage. How many percent of the code has been covered by tests.

- Inconclusive tests. The number of tests that did not succeed or were paused. If the build did not succeed, the tests are either not counted or counted as inconclusive.

- Failed tests. How many tests failed during the build.

- Passed tests. How many tests that were passed during the build.

The Build Success Over Time report (see Figure 29-2) shows us the status of the last build for each build category (a combination of build definition, platform, and configuration) run for each day. We can use this report to help us keep track of the quality of the code that we check in. Furthermore we can, for any day on which a build ran, view the Build Summary for that specific day.

Figure 29-2.  Build Success Over Time report

The Build Summary Report (see Figure 29-3) shows us information about test results, test coverage, code churn, as well as quality notes for each build.

Figure 29-3.  Build Summary report

Note  The reports in this section are mapped to the particular process templates they support. If you find a report you like, it is possible to customize it to work with another template. In Chapter 32 we describe how to customize an existing report to do this along with other things.

Custom Reporting

The reporting capabilities in TFS give us access to most of the information we manage in our ALM process. In the previous section we have seen how standard reports give us metrics for our project at a general level. By customizing, extending, and creating new reports we can find out what works well in our projects and what does not.

Note  In Chapter 32 we will look at the details of reporting in TFS, including how to create custom reports based on the data models described here.

Data Warehouse Model

The data warehouse in TFS contains several interesting views for analyzing the build information, for instance to create reports on build quality or change information for a particular build.

Build Project Tables

The tables contain build project information that can be used to query data about files and projects in a build. The fact table also contains quality metrics such as compiler errors and warnings (see Figure 29-4).

Figure 29-4.  Build project data model

Build Details Tables

The build details tables give us data to query about builds, build status, and build quality (see Figure 29-5).

Figure 29-5.  Build Details data model

Build Changeset Tables

The build changeset tables give us information about which changesets were associated with a particular build (see Figure 29-6).

Figure 29-6.  Build Changeset data model

Build Coverage Tables

If we want to query the warehouse about test coverage from tests run as part of a build, we use the build coverage warehouse tables (see Figure 29-7).

Figure 29-7.  Build Coverage data model

Assessment

The following table (Table 29-1) lists questions that can be used as a basis for an ALM Assessment in the release management area (see Table 29-1). A lot of the assessment questions cover the build process, but many others cover operations and how we deploy applications into production environments.

Table 29-1. ALM Assessment Questions

Area

Sample Question

Discussion

Build Management

Is a build process well defined?

Most of these topics below are covered by TFS version control system.

Is an automated build verification process in place?

Is there a library of all successful builds?

We can use TFS to help us put the build outcome into specified folders.

Is there a regular build schedule?

Is the build process automated?

Yes in TFS. If the customer lacks this we can set it up for them.

Can developers replicate the build process locally?

Is a build easily reproducible?

Yes in TFS. If the customer lacks this we can set it up for them.

Can you map source control changes to specific builds?

Yes in TFS. Source control changes are associated with each build as an associated changeset and can be viewed on the build summary page.

Is there an effective build failure/success notification process?

Yes in TFS. If the customer lacks this we can set it up for them.

Is there a build status/progress web page in place?

Yes in TFS.

Are drop locations structured/organized effectively to support DEV/testing/Deployment efforts?

We can tailor it as we want in TFS.

Designed for Operations

Is an overall architecture in place for the infrastructure environment?

Do the operations team and development team communicate well at present?

Communication is key to success. Do not forget to involve the infrastructure team.

Does a dedicated team own the infrastructure architecture?

Do the developers understand the implications of application deployment to the live environment?

Database Deployment

Do you deploy database changes to a staging database prior to production?

Often forgotten. Help the customer if this is not in place.

Summary

This chapter showed how we can use TFS to retrieve information for KPI assessment and also how we can see the build status using standard reports from TFS.

We have also shown how many of the assessment questions from the Microsoft online assessment can help us plan for successful implementation of release management.

The next part of this book covers the architecture of Team Foundation Server and how we can plan for a TFS installation.

PART 7

Team Foundation Server

In the final part of this book we look a little more at TFS itself. To appreciate what options we have when it comes to deploying TFS in our organization we need to have an understanding of the architecture of TFS. Given that, we can carry on with planning the actual deployment. Do we upgrade from an earlier version? Do we complete a clean install? Do we need to distribute TFS among several servers to handle a large workload? These topics are covered here.

Chapter 32 looks at how reporting works in TFS. This is essential—especially if we need to perform customizations for our organizational needs.

The final two chapters are very interesting indeed. We discuss how to use TFS together with Eclipse (just to mention one development tool that is not .NET). Using Team Explorer Everywhere we can access version control, work items, and many other aspects of TFS from our Eclipse environment, even from Mac OS X. This is useful if we work in an organization with development on many platforms.

We finish the book by covering TFS in the cloud. Microsoft has a cloud-based version of TFS running on Windows Azure. At the time of writing this is still in prerelease and by invitation only, but it looks really good. This could be a perfect solution when we do not want to spend time, money, and effort on maintaining our own TFS environment on our own premises.

CHAPTER 30

Architecture and Implementation Planning

Before we install TFS into our infrastructure there are some things to consider. We need to plan for the deployment so that we can get optimal result of TFS. Some things we need to consider are

- TFS Architecture; the logical application, data, and client tiers

- Physical server location that will host those tiers

- Team Foundation Build and the build computers that will run in our environment

- Team Foundation Server Proxy for our remote locations

In this chapter we take a look at some of these aspects so that you can plan for the best solution for your organization. Let’s start with TFS architecture.

TFS Architecture

We can divide the TFS services into three logical tiers:

- The Logical Application Tier

- The Logical Data Tier

- The Team Foundation Client Tier

These tiers can all run on one machine or they can be divided on several. It all depends on what the needs are in the organization.

Team Foundation Server 2012 also includes a set of web services and databases which we can install and configure separately on the server or servers that host the logical application, data, and client tiers for TFS. This means that we do not have to keep one specific layer on one server, we can distribute even the logical tiers. This gives us the flexibility to adapt TFS to our specific infrastructure needs.

Note  The reason we call two of these tiers “logical” is purely because the services on them can be logically grouped together. The data tier has all databases/data services; the application tier has all TFS application services, and so on. This does not mean that the application tier must run on a single machine. We can divide the components of each tier on several physical servers to handle a large workload.

Let’s look at what services are included in the application tier.

Logical Application Tier

The application tier is the core of TFS. Here we find all services TFS offers.

First we find all web services for team project collections. We all recognize this so no further explanation is necessary:

- Version control

- Build service

- Lab management

- Work Item tracking

- Team Foundation Framework services.

This set of services is running on the Application Tier of TFS and allows us to access the different aspects of Team Foundation Server, including administration, security, and events. They enable us to extend Team Foundation functionality by developing TFS extensions and adapters to integrate third-party tools with TFS 2012.

We also find the Application Web services:

- Core Framework services (see above)

- Supporting Framework services, such as SharePoint server extensions and reporting services

The Virtual Machine Manager Administration tools let us manage our virtual machines if we have enabled this feature set.

There are also administrative tools (see Figure 30-1) for TFS that belong to the logical application tier. These tools help us perform most of the administrative tasks that are required to manage our TFS infrastructure. Figure 30-1 shows the administrative console for TFS. This console is installed by default on any server on which you install one or more of the following components:

Figure 30-1.  TFS Administrative console

- An application tier for Team Foundation Server

- Team Foundation Server extensions for SharePoint Products

- Team Foundation Build

- Visual Studio Lab Management

The Test Controller is a service responsible for controlling test execution. It can be used for controlling load tests, which we looked at in the Automated Testing chapter, but also for managing automated test runs, which is what we use it for in lab environments. Each test controller used with Lab Management is bound to a TFS Team Project Collection but can be used by any project in that project collection.

These web services and applications depend on some features of the platform they are running on. First of all they need to have the Visual Studio Team Foundation Background Job Agent (TFSJobAgent) running. This service provides a scheduling mechanism for Web services and jobs for Team Foundation Server. This is a Windows service and is also used to run the tasks created by various wizards. As two examples we can mention the New Team Project wizard and Create A Team Project Collection wizard. The TFSJobAgent runs on any server that is running a Web service or Web application in the logical application tier for Team Foundation Server.

The TFS web services also need to run on ASP.NET in Internet Information Services (IIS). The minimum operating requirement is Windows Server 2003.

Optional Components

There are also some optional application tier components we can install:

- Extensions for SharePoint products and technologies. We can integrate Team Foundation Server with one or more SharePoint Web applications if we want. From TFS 2010 we are not required to use SharePoint products. Using SharePoint the project managers and product owners can create SharePoint sites, referred to as a team project portal, for a team project or a site collection for a team project collection. If you want to integrate one or more SharePoint Web applications with Team Foundation Server, you must first install and configure the Team Foundation Server Extensions for SharePoint Products on the server that is running the Web application. You can then configure mappings from Team Foundation Server to the SharePoint Web applications that you have configured with extensions (http://msdn.microsoft.com/en-us/library/bb552177.aspx).

- SQL Server reporting services. Reporting services are a server-based report generation software system. They can be used to prepare and deliver a variety of interactive and printed reports and are used for TFS reporting if selected during installation.

Logical Data Tier

The logical data tier for Team Foundation includes three main databases that store data for all projects in TFS and one additional database for each project collection.

These databases include data, stored procedures, and other associated logic for TFS. They could all be stored on one physical server or they can be distributed across many servers if necessary. Let’s take a quick look at them so we get an understanding of what data they store.

TFS_Configuration (Configuration database). This database stores the catalog of resources and the configuration information for Team Foundation Server. TFS_Configuration contains the links, pointers and information about the operational stores for Team Foundation Server. TFS_Configuration will be installed on the instance of SQL Server that is used when Team Foundation Server is first installed and configured. If we want we can move this database to another server so that we can distribute the load across several servers. So we are not locked to the first SQL Server we used during installation.

- TFS_Warehouse (Warehouse Database). Here is where we find data from work item tracking, source control, builds, testing tools, and so on. Team Foundation server uses this data for its built-in reporting functionality. TFS_Warehouse is installed on the instance of SQL Server that is used when Team Foundation Server is first installed and configured. If we want, we can move this database to another server so that we can distribute the load across several servers.

- TFS_Analysis (Analysis database). This multidimensional database stores the aggregated data from team project collections. TFS_Analysis is installed on the instance of SQL Server that is used when Team Foundation Server is first installed and configured. If we want, we can move this database to another server so that we can distribute the load across several servers.

- Team Project Collection databases. Each team project collection has its own database where data is stored for all team projects in the collection. These databases can be installed on any SQL Server instance that is compatible with TFS.

To use the full version of SQL Server with TFS 2012 we need to run at least version 2008 R2 or higher. We also need to use Windows Server 2008 or higher as the operating system.

There are some optional databases and services that we can install as well:

- SharePoint web application databases

- Reporting extensions

Team Foundation Client Tier

The client tier might speak for itself. This is the tier where we install the Visual Studio GUI of course, but it might also be the Team Explorer Everywhere, a standalone Team Explorer or our Microsoft Office tools like Excel and Project. In this tier we find:

- Visual Studio 2012 Team Explorer (Build, version control, Work Item Tracking)

- Version control commands (Office integration, Office Project integration)

These tools use the Team Foundation Object Model and run on any supported platform, so please check the latest compatibility list at the Microsoft web site.

There are also some optional components we might want to install on client tier computers:

- Test controller. The test controller can be distributed on several computers to spread the load.

- Test Professional (includes Microsoft Test Manager). Some testers might only need to use the test tools and then the Test Professional will be enough.

- Team Foundation web client. We can of course use the web access page from a computer not running any other part of TFS or Visual Studio.

- Visual Studio 2010/2008/2005. We are not limited to Visual Studio 2012, we can also use Visual Studio 2010, 2008, or 2005 if we want but then some of the latest features from the 2012 version will not be available. The older Visual Studio versions need a patch to work with TFS 2012.

Optional Components on Separate Servers

TFS includes some optional components that we can install on separate servers to distribute the load on TFS. We install and configure them separately, and we are not forced to install them.

- Virtual Machine Manager on VMM server

- Build services on a separate build server (see more in the following sections)

- Proxy server (see more in the following sections)

- VMM Library server (stores VMs and templates)

- VMM host server (hosts deployed virtual environments)

- Virtual machine (machine roles, build agent, lab agent, test agent)

Team Foundation Build

As we saw in Chapter 25, the Build process is important. Build automation is the core step in the Continuous Integration (CI) process.

So to accomplish a good build process we need to consider where we want to implement our build server(s). Depending on the number of users, the chosen automated build process, and so on, we might experience such a load on our build server that we need to use more than one server. We do not want to find ourselves in a situation where the build process will be a bottleneck, slowing down our continuous delivery process. Neither do we want the build process to take processing power from our TFS server, making work item tracking, version control, and so on, slow.

Installing Team Foundation Build on a separate server is always a good idea, especially if you have a more than 10−20 developers working at the same time. This limit is hard to fix however, because it depends on the hardware of the server. But if you have more than one or two scrum teams we think you should consider having a separate build server.

Note  Each project collection requires at least one build controller, and it is not possible to install multiple build (or test) controllers on the same machine. This can quickly result in a lot of servers if the organization creates a lot of project collections.

Team Foundation Server Proxy

Many development organizations are located in different places. They might be in the same country or on another continent. Because they belong to the same development organization they often share the same code and the same version control system (read TFS). In many cases this works fine but sometimes we find that for instance version control is slow. This is where the TFS Proxy comes into place.

The purpose of the TFS Proxy is to manage a cache of downloaded version control files. If we implement a TFS Proxy at the remote location(s), each source control version of a file will be downloaded one time per location instead of one time for every developer. This improves the version control performance when developers are working on the same TFS Project or even more than one project. Keep in mind that the Proxy only improves download speed; it will not make the check-in process any faster.

To use the Team Foundation Server from the Visual Studio client, we have to set the Team Foundation Server Proxy name and port in Tools, Source Control, Visual Studio Team Foundation Server (see Figure 30-2).

Figure 30-2.  Setting the TFS Proxy in Visual Studio

Planning the Implementation

There are numerous things to consider when planning your TFS installation. This book is too short to consider them all. Instead we suggest you take a look at the ALM Rangers TFS planning guide at http://vsarplanningguide.codeplex.com/. They offer excellent documentation and tools for planning your installation. They have detailed walk-thru documents asking various questions about your installation so that you can arrive at a solution that will work very well for you and your organization.

Note  We will not discuss TFS running on Windows Azure in the cloud in this chapter. For more information about this please refer to Chapter 34.

Physical Server Location to Host Our TFS Tiers

As we can understand from the TFS architecture discussed previously we have many ways to install our TFS environment. Depending on the requirements of your organization and the way you are located, you need to plan your physical server location(s) as well. One starting point is to look at the number of users that you need to support. Use this for planning and then add some capacity for future growth. Table 30-1 shows a simplified view of how many users you might expect on different server configurations. The ALM Rangers referenced previously provides the updated systems requirements and you can also check for any changes at http://msdn.microsoft.com/en-us/library/dd578592(v=vs.110.

Table 30-1. Basic Number of Users for Different Server Example Configurations

With Table 30-1 as background you can start your server planning. Our suggestion is to get as good hardware as possible and not go for the lower-end servers. Having a separate database server is a very good idea so configuration 3 in the Table 30-1 is a good start. If you already have a SQL server installation in your organization, you can use that and not install a new one. Just check that all requirements for SQL Server are fulfilled before starting TFS installation and that there are available resources to host the TFS databases.

Furthermore, you need to consider whether you are going to use a separate build server. If you have a small team of developers you might be okay with a single server setup, but we suggest you start with a separate build server as well.

The next thing to consider is where to place the servers. Is your datacenter reachable with good speed from all your locations? In that case place them in the datacenter. If you have remote locations, please consider adding a TFS Proxy server at the remote location(s), especially if bandwidth is limited. See what follows for more information on the TFS Proxy server.

Note  You can always scale out or up at a later time. So even if you go for a single server installation at first you can distribute an increasing load by adding servers and moving parts of TFS (like the database, the build server and so on) to the new servers when needed later on.

Ports and Network Protocols

To plan for the best implementation you must also be aware of all network protocols that TFS and its components use. Check out http://msdn.microsoft.com/en-us/library/ms252473(v=vs.110).aspx for the latest details. Figure 30-3 shows the ports and protocols for an extremely simple setup just to give a glimpse of the complexity you might experience. We will not cover all the required ports and protocols in this book, but to plan the physical location of your TFS servers you need to take this into consideration from the start. Make sure you involve the infrastructure team when planning; they know all there is to know (hopefully) about the network.

Figure 30-3.  The ports and network protocols for a simple TFS setup

Now we will take a look at two setups you can use as a base for discussions with your development and infrastructure team, but first let’s talk system requirements.

System Requirements

When planning our TFS infrastructure we need to make sure our hardware and software meet all requirements as well. At http://msdn.microsoft.com/en-us/library/dd578592(v=vs.110) you can find the latest information about this topic. With TFS 2012 the requirements have changed a bit. You must use a 64-bit server operating system to install Team Foundation Server from now on. Microsoft supports 32-bit client operating systems, but they no longer support 32-bit server operating systems.

You can install Team Foundation Server on a server that is running one of the following operating systems:

- 64-bit versions of Windows Server 2008 with SP2

- 64-bit versions of Windows Server 2008 R2 with SP1

- 64-bit versions of Windows Server 2012

Note  TFS does not support the Server Core installation option for either Windows Server 2008, Windows Server 2008 R2, or Windows Server 2012.

If installing Team Foundation Server or SQL Server on Windows Server 2008 R2, you must have .NET Framework 3.5 or above installed. On Windows Server 2008 R2, you can install .NET Framework 3.5 by using the Add Features Wizard from Server Manager.

Because TFS installs SharePoint Foundation 2010 with the standard configuration, you can’t use it on Windows Server 2012. If you want to install Team Foundation Server on Windows Server 2012, use the advanced configuration and skip the installation of SharePoint Foundation 2010.

A cool thing (at least for demo purposes) is that you can install Team Foundation Server on a client computer as well. There are some limitations however. TFS on client operating systems does not support integration with SharePoint Products or reporting. If you want to use either of these features, you must install Team Foundation Server on a server operating system. These client operating systems are supported:

- 64-bit or 32-bit versions of Windows 7 with SP1 (Windows 7 Home Premium with SP1, Windows 7 Professional with SP1, Windows 7 Enterprise with SP1, Windows 7 Ultimate with SP1)

- 64-bit or 32-bit versions of Windows 8 Release Preview

Service Accounts

As discussed earlier in this chapter a TFS deployment can be setup on a single machine or on a set of machines each hosting the different services. Depending on the requirments for the deployment the different services can be configured to run under accounts with the appropriate permissions. It’s good to understand which accounts are used by what service to administrate the environment in a proper way. A standard TFS setup would use the the following accounts:

- TFS Service Account. The service account used for the TFS web services.

- TFS Reporting Account. Account used to generate reports for TFS reporting.

- TFS Build Account. Service account used for running builds and communicatate build information to TFS.

But again, the specific requirement we have when designing the TFS environment must guide what accounts we use for each service. See http://msdn.microsoft.com/en-us/library/ms253149(v=vs.110).aspx for an up to date guide regarding accounts used by TFS services.

Simple Implementation

The easiest TFS implementation is a single server installation (see Figure 30-4). When using this setup all TFS components are installed on a single server.

Figure 30-4.  A simple single TFS server installation

Of course you can use this setup if you have a very small development organization. This works fine for a long time if you do not add too many users. Consider however that the build process and automated tests for instance take their toll on the resources of the server, and might slow down version control and work item tracking for the users. If you want better performance you should consider going for a medium installation instead. Remember that you are not stuck with this installation. You can scale up and out as you need so if you hesitate, go for the single server installation.

Medium Implementation

If you know from the start that you will have relatively high load on the TFS, you should consider going for a more complex setup. In Figure 30-5 we can see that we have added a database server hosting the TFS Data tier. We have also added a separate build server, which further spreads the load.

Figure 30-5.  A medium TFS server installation that can be a good staring point for a new installaltion

We consider this setup a good starting point for most organizations. Some benefits are

- The build process does not slow down TFS

- Database processing doesn’t take resources from TFS

- Database backup does not slow down TFS

- A larger number of users are supported

- A larger number of team projects can be created

If you find that this implementation is closing in on the end of the road you can consider going for a more complex TFS environment or maybe scale-up your servers.

Complex Implementation

The complex implementation (see Figure 30-6) builds on the medium. But some new servers have been added to even further distribute the load on TFS. We have included the System Center Virtual Machine Manager that manages the virtual machines we use for our tests using Lab Management (see Chapter 23).

Figure 30-6.  A more complex TFS server installation

Previous Version of TFS Already Deployed

So what if you already have an earlier version of TFS in your organization? We will discuss the installation process in the next chapter, Chapter 31, where we will also look at the upgrading options and the requirements you need to fulfill.

Summary

In this chapter we have taken a look at the architecture of TFS 2012. The reason for this is that we want to show you the opportunities you have when it comes to scaling TFS. Basically, TFS is a three-tier application (logical tiers), but parts of each tier can be distributed on different servers in order to handle large workloads.

When planning a TFS installation you need to consider the estimated workload so that you can deploy TFS in the best way for your organization.

The next chapter will cover TFS installation and the options we have there. We will also discuss upgrade options from previous versions.

CHAPTER 31

TFS Installation, Upgrade, and Administration

In the previous chapter we discussed the architecture of TFS. We also covered some basic implementation planning. In this chapter we will take this further and look at the different TFS installation options we have. We will also go through some common upgrade scenarios and look at what needs to be done to get a pre-TFS 2012 instance upgraded to 2012.

Installation Options

When running the setup of TFS the installation wizard wants to install the core components on your machine. You get the option of where to install these components on the first setup page (see Figure 31-1). Make sure to choose a suitable location because it is always impractical to change the location later.

Figure 31-1.  Starting the setup of TFS core components

You start the setup by clicking Install Now. After the core components installation is finished you are welcomed to the TFS configuration center as seen in Figure 31-2.

Figure 31-2.  The TFS Configuration Center from where we install TFS

We can see that we have options to configure our Team Foundation Application Server and also our Build service. Let us focus on the TFS application server for now.

We can configure TFS in four different ways:

- Basic

- Advanced

- Application-Tier Only

- Upgrade

Let’s now take a look at these installations in a little more detail.

Basic

The Basic installation of TFS is the option to use if we want to have a single server TFS installation up and running quickly. The TFS basic installation even runs on a client operating system if we want to setup a minimal development environment.

If we want, we can add more than the core TFS features (such as work item tracking, version control, and builds) and get reporting in our projects as well as team portals. We need to install SQL Server before we install the TFS if we want to have reports. If we can manage without the reporting features, we can let the TFS installation install SQL Server Express for us.

Team Foundation Server can install SharePoint 2010 for us if we want, but we can use an existing installation of SharePoint Products with the basic configuration wizard as well. This is new for TFS 2012 and is great because we get the simplicity of the basic installation but can use a version of SharePoint Products other than the one that ships with Team Foundation Server. If SharePoint is not installed on the same server as the TFS application tier, then we need to install the SharePoint TFS Extensions on the SharePoint server. The SharePoint TFS extensions installs the TFS specific SharePoint features such as WebParts.

Note  If installing a SQL Server instance yourself, please install SQL Server 2008 R2 or SQL Server 2012. You must also install the following SQL Server features for a complete TFS setup: Database Engine, Full-Text Search, Analysis Services, and Reporting Services. After you have installed SQL Server, configure Analysis Server for automatic failover.

The standard installation is very straightforward and will need very little user interaction. Here are the steps you need to follow:

- In the Team Foundation Server Configuration tool, choose Basic, Start Wizard. The Basic Configuration wizard appears.

- Read the Welcome screen, and then choose Next.

- Perform one of the following actions as seen in Figure 31-3:

Figure 31-3.  Choose to install SQL Server Express or use an existing SQL Server

- Choose Install SQL Server Express to host the configuration database on an instance of SQL Server Express, and then choose Next.

- Choose Use an existing SQL Server Instance to host the configuration database on an existing instance of SQL Server, and choose Next. Then, in SQL Server Instance, type the name of the server that is running SQL Server or the named instance that will host the configuration database, and choose Next. Choose Test to test the connectivity to SQL Server.

- Review the information on the next screen, and then choose Next. The wizard validates your configuration.

- When validation is complete choose Configure (see Figure 31-4). The wizard applies the configuration settings.

Figure 31-4.  Starting TFS configuration

- Choose Next.

- Choose Close.

- Choose Close and then you are done. The Team Foundation Server Administration Console appears. We will cover this in more detail later in the chapter.

Advanced

If we need to configure a more complex setup of TFS, we can use the Advanced configuration. This option allows us to install and configure TFS on more than one server and it also lets us have more control over the options that are available to us.

Some of the options we have are to use an existing SQL Server or SharePoint server that we want to use to host team projects or TFS data. We can also place TFS and its configuration databases on different servers.

We can also choose to install SQL Server and SharePoint on separate new servers if we do not have any existing one we can use. These installations can be hosted on different servers as well. If you use multiple servers, you can distribute the load between Team Foundation Server and the configuration database, or you can ensure that prerequisite server software for features such as reporting or the portal site is running on capable hardware.

Note  Don’t mistake the multiple server installation with the scale-out features that Team Foundation Server offers. The scale-out features include the ability to create a Team Foundation Server farm and add a team project collection to an instance of SQL Server that was not part of the original deployment of Team Foundation Server. However, these scale-out features are not part of the advanced setup scenario above.

If you configure TFS on multiple servers, you must consider your user accounts as well. You need an Active Directory domain in your network because you cannot use local accounts. Make sure that the account that you use to install Team Foundation Server is a member of the Administrators security group on the servers where you will install Team Foundation Server. Be sure to involve your infrastructure team in this process so that you will not run into problems later on.

If you plan to configure reporting, you must also be a member of the Administrators security group on the server that is running SQL Server Reporting Services. If you plan to configure SharePoint Products, you must be a member of the Farm Administrators group on the SharePoint Central Administration site.

Also make sure you have noted the service accounts that you will use to install Team Foundation Server, SQL Server, and SharePoint Products.

The steps you need to follow to configure your TFS installation using the advanced options can be found at http://msdn.microsoft.com/en-us/library/dd631919(v=vs.110). We do not cover all these steps here because there are too many. Please refer to this web page for these instructions.

Application-Tier

If you already have a TFS server installation in your organization you have some options for improving performance or repair a TFS installation in the application-tier only configuration option. There can be many reasons for using this option:

- You might want to build redundancy into your TFS installation by adding a new TFS server and create a server farm. This requires Network Load Balancing installed and configured on the participating servers. This also requires that the application tier and the configuration database are installed on separate servers.

- You might want to improve speed of your TFS installation by adding TFS servers to a farm. Scaleout deployments should also use a load balancing solution to distribute the load evenly over the farm.

- You might have a TFS application tier that has failed for some reason and needs to be replaced.

- Maybe the server hardware is getting obsolete and you need to move the TFS Application Tier to another server.

Note  You must have installed TFS 2012 using a domain account for the service account (TFSSERVICE); otherwise you must change to a domain account.

Installing an additional Application Tier to an existing TFS installation

- Install TFS 2012 as described earlier.

- From the Team Foundation Server administration console choose Application-Tier Only Configuration.

- In the SQL Server Instance dialog select the SQL Server hosting the TFS configuration database for your setup.

- Choose a database from the list to which this Application Tier will be connected.

- Choose the execution account for the TFS services.

- Choose the authentication method.

- Finally step through the final steps in the wizard and the new application tier is setup for you.

Upgrading from Earlier TFS Versions

If you are already using TFS in your organization there is a good chance that you can upgrade to TFS 2012. You can perform an upgrade from the following earlier TFS versions:

- Developer Preview or BETA release of Team Foundation Server 2012

- Team Foundation Server 2010 with or without Service Pack 1

- Team Foundation Server 2008 with Service Pack 1

There are two upgrade paths that you can choose. You an upgrade your TFS installation and keep using the existing hardware or you can upgrade and move TFS to new hardware. We recommend moving to new hardware if possible because it will let you keep the old installation to fall back to in case something goes wrong.

Keep Existing Hardware

To upgrade and keep your existing hardware, follow these steps:

- Verify that your operating systems and your hardware are supported by TFS 2012.

- Check that SQL Server is version 2008 R2 or higher. If not, you need to upgrade SQL Server first.

- Backup your TFS data. The installation deletes your databases.

- Select which account TFS will use. The default is the Network Service account. You probably want to keep using the old account but you can choose the Network Service account if you want.

- Using control panel you need to uninstall the previous version of TFS. This might seem dangerous but uninstalling TFS will not remove the TFS database. We can always re-create an application tier by restoring and reconfiguring the TFS database.

- If you are running SharePoint on a computer that is not the TFS computer, you need to uninstall the TFS extensions for SharePoint as well. If SharePoint is running on the TFS server, the extensions will be automatically removed during uninstall of the previous TFS version.

- Run the Team Foundation Server install from the product DVD and then use the Upgrade Configuration wizard to upgrade your installation.

- Follow any instructions on the screen.

Move to New Hardware

To upgrade to TFS 2012 and move to new hardware:

- Verify that your operating systems and your hardware are supported by TFS 2012.

- Backup your TFS data to restore on the new machine.

- Select which account TFS will use. The default is the Network Service account. You probably want to keep using the old account but you can choose the Network Service account if you want.

- Install SQL Server 2008 R2 or 2012 on the server you have selected. Do not forget to check all SQL Server requirements first. Check http://msdn.microsoft.com/en-us/library/dd631907(v=vs.110) for more details. If you are going to use an existing server, you need to verify that it meets the system requirements.

- Install and configure Reporting Services and Analysis Services on the SQL Server (if not already installed). Keep in mind that the report server cannot be shared with other applications since TFS requires a dedicated report server. If your report server is not on the same server as Team Foundation Server, the account that you used to install Team Foundation Server needs to be added to the Administrators security group on the report server. This enables setup to configure the report server to work with Team Foundation Server.

- You can set up SQL Server for reporting during an upgrade in any of the following ways:

- You do not have to use the same report server that you used with your earlier version of Team Foundation Server.

- You do not have to use the same SQL Server instance for the report server as the one that you used for the Database Engine.

- You can use multiple servers to host the report server.

- Verify that you have configured all instances of SQL Server to meet the minimum requirements of Team Foundation Server.

- Set up your SharePoint Products if you want to continue using a portal. Do this before launching the TFS upgrade. You cannot install SharePoint Foundation 2010 during Team Foundation Server upgrade, as you can during installation. For SharePoint you have the following ways to install:

- Install SharePoint on the same new hardware you will use for TFS

- Install SharePoint on different hardware from the new TFS

- Continue to use a SharePoint installation that is installed and running on different hardware from where you will install the new TFS.

- Install Team Foundation Server and use the Team Foundation Server Configuration tool to restore the data you backed up earlier in this checklist and configure the upgrade.

- If you installed Microsoft Office SharePoint Server 2007 or SharePoint Server 2010, you must configure the enterprise application definition that you created for Team Foundation Server.

Once the steps in any of these two preceding upgrade paths are complete, please follow the steps from http://msdn.microsoft.com/en-us/library/dd631898(v=vs.110) to configure the upgrade. Notice that you have different steps depending on from which previous version you are upgrading.

Enabling Features After Upgrade

After we have upgraded our TFS instance we can continue to work in the project but some new TFS features will not be available to us before we enable the capabilities. After enabling these new features we get access to the following TFS 2012 features:

- Code Review: Initiate code review and give responses.

- Feedback: Initiate feedback requests and track feedback responses.

- My Work: Manage work, suspend and resume tasks.

- Planning Tools: Manage the product backlog and plan sprints using the backlog page (including iteration and capacity planning), manage spring activities using the task board page.

- Storyboard Integration: Access the storyboards by linking them to work items.

- Hidden Types Category: Specify the set of work item types that the Visual Studio tools create but not something we work with manually. This includes Code Review Request and Code Review Response, Feedback Request and Feedback Response, and Shared Steps.

If a project has not yet been updated, the error in Figure 31-5 is shown where we can configure the features using a wizard.

Figure 31-5.  Error message for missing configuration of upgraded team project

Click the Configure feature to initiate the wizard that enables the new features. The wizard analyzes the team project to see whether the process template used is compatible with what the wizard can upgrade to. Figure 31-6 shows the configuration wizard process.

Figure 31-6.  Configure Features for TFS 2012 Wizard (continued)

If the project cannot automatically be upgraded we can still do the upgrade by hand. For a description of the required steps see http://msdn.microsoft.com/en-us/library/ms194972(v=vs.110).

Team Foundation Server Administration

A Team Foundation Server will not run and maintain itself (nor will any other server product). You need to plan for maintenance and operations at the same time as you plan for the deployment. Be sure to involve the operations team already at the deployment planning stage (at the latest) so that they can advise on how to implement TFS into your infrastructure. As we can understand from the TFS architecture in Chapter 30 there are numerous ways to deploy TFS and many things to consider.

Some of the tasks that need to be taken care of after (and during) a TFS installation are

- Managing the service accounts and the services that were used during deployment.

- Setting up security by managing users and user groups, both server-level as well as collection-level security

- Managing permissions and security for any other roles necessary.

- Creating and maintaining team project collections so that projects and resources can be well supported

- Setting up permissions on servers that are integrated with TFS. For instance, a separate SQL Server or a SharePoint server. This requires managing groups and permissions between these participating servers.

- Adding servers and resources necessary for the deployment.

- Creating disaster recovery routines, including backing up and restoring data, both on TFS server, SQL Server databases as well as on SharePoint servers.

It is essential that the administrators understand the architecture of TFS and also the basic concepts that we have covered in this book. This includes the features of TFS, the tools TFS offers both to developers, project managers, administrators and others.

Administrators also need to understand how TFS and its components communicate so that the correct settings can be applied in the network. There are also Windows server security concepts and restrictions they need to be aware of, like access control, user account administration, and so on. This should not be a problem for a good operations team, but make sure you won’t run into problems with something you could easily avoid beforehand.

Also make sure that the competence on SQL Server and SharePoint (if you will use it) is high, especially if this is the first SQL Server/SharePoint deployment in your organization. If you are going to use Lab management and Hyper-V for testing you need to make sure that operations understand and can manage a Hyper-V server.

TFS includes an administration console (see Figure 31-7) where common administration and tasks for TFS can be performed. From this console you can manage administrative permissions, your team project collections, your build environment, and your virtual lab environment, as well as checking your TFS deployment status and reading various TFS log files.

Figure 31-7.  The TFS management console

Here are some of the things the management console can help you with:

- Managing the SharePoint web applications. This requires the TFS server extensions for SharePoint and that you have added the SharePoint Web applications as resources that are available for team projects and team project collections.

- Configuring reporting resources for your team project collections and the projects in those collections.

- You can monitor your TFS server logs.

- Create and manage team project collections. This includes organizing the team projects into a specific team collection so that the structure matches the development needs.

- You can start and stop TFS services for operational maintenance. You can also change the accounts or passwords that are used as service accounts in TFS.

- The console can be used to manage server level users and user groups. You can create your own custom groups if you do not want to use the default.

If the TFS operations team prefer to, they can also use the TFS command line tools to manage the TFS deployment. They can of course script the parts they need as well. The command line tools are installed during the TFS installation.

Adding TFS Collections After Installation

As we saw previously we can choose to upgrade an existing TFS instance as part of the TFS 2012 installation. But what if we have already setup our TFS 2012 and want to upgrade an existing TFS later on? Do we have to reinstall the entire TFS setup to do the upgrade?

Fortunately for us there is a command-line tool that allows us to upgrade an existing TFS to 2012—the TFSConfig.exe utility. To upgrade a TFS 2008 or 2010 database we issue the following command:

TFSConfig Import /SQLInstance:ServerName /CollectionName:NewCollectionName /confirmed

Where

- ServerName is the name of the SQL Server instance hosting the TFS database to be upgraded.

- NewCollectionName is the name of the new TFS collection.

- confirmed indicates we confirm we have backups of the source database in case something goes bad during the upgrade.

Attaching a TFS database

If we need to move a TFS database from one TFS instance to another we can use the Team Foundation Server Administration Console to do this and go through the following steps:

- Select the collection from the administration console on the source server and select Detach.

- Back up the database on the source server.

- Copy the backup set to the target server.

- Restore the backup set on the target server.

- Go to the administration console on the target server, select Team Project Collections and Attach Collection. The wizard in Figure 31-8 opens where we point to the restored collection database and lets the wizard connect it to the TFS instance.

Figure 31-8.  Attaching a Team Project Collection

Note  If the attached database is an earlier version, it will be upgraded as part of the attach process.

Enabling Web Features

Another behavior you may notice when working with a TFS 2012 and its web client is that some features may not be available to all users. It turns out that the web client has an access model that allows us to define the functional authorization for our users. If for instance a user does not see the “View Backlog” option on the team’s home page (see Figure 31-9) this is likely due to restrictions in the access configuration for that user.

Figure 31-9.  User not having access to the Backlog view in TFS web access

Figure 31-10 shows how to assign a user to the appropriate permissions in the TFS Web Access. The permissions enable the correct features for the user.

- Limited: View work items created by self.

- Standard: Limited plus standard features and the Agile Boards.

- Full: Standard plus Backlog and Sprint Planning Tools and Request and Manage Feedback.

Figure 31-10.  Adding users to Full access in the TFS web client

Similarly we may not have setup our iterations properly. For instance, if we have not selected iterations for our project, the TFS web client cannot render the team board (see Figure 31-11). To resolve this problem we need to click the “Select team’s iterations” link and configure the iterations for our team.

Figure 31-11.  Error in configuration of Team Board

Enabling Email Alerts

Some features in TFS 2012 require we enable email alerts, for instance, the Feedback tool and the alert subscriptions.

Figure 31-12.  Error message for missing email alert configuration

To enable alerts we can use the Team Foundation Server Administration Console. Select the TFS application tier and scroll down to the Email Alerts Setting to bring up the configuration dialog. From the settings dialog we can provide the address to our SMTP server as well as the sender address used when sending out alerts (see Figure 31-13).

Figure 31-13.  Email Alert Settings

The Email alert settings dialog has been extended with an Advanced SMTP settings section. This new feature allows us to configure the identity for the SMTP authorization.

Summary

In this chapter we have looked at the various ways we can configure a new TFS installation. We have seen how simple it is to set up a basic setup, as well as what needs to be done to get an advanced configuration in place.

Many existing TFS installations will need to be upgraded and we have looked at how this can be achieved. In general, we recommend moving to new hardware when upgrading TFS because that lets us fall back to the old instance in case something goes wrong.

Finally, we covered some of the administrative tasks related to maintaining a TFS 2012 server. In particular we looked at how we can enable different capabilities in TFS by running the corresponding configuration.

This concludes the architecture, installation, and administration sections of the book. Next we will look at how reporting works in TFS and how we can use the data in TFS to analyze the performance of our team.

CHAPTER 32

TFS Reporting

One of the most powerful aspects of TFS is the fact that data from the different disciplines in the team is stored in one central database. As an ALM platform this is very valuable because we can use this data to gather knowledge of work done over the entire process, not just in an isolated practice. This is something few other tools on the market offer and can be extremely powerful for us as software makers. Unfortunately many teams are unaware of these capabilities of TFS which is something that needs to change.

In the previous chapters on Metrics and ALM Assessments we have looked at important KPIs and how the information from work in each phase in the ALM process is stored in the data warehouse in TFS.

In this chapter we will first drill into the data warehouse and reporting architecture in TFS. Having gained an understanding of the way information is stored in TFS we can then move on and look at the different ways of getting to the information and create reports over the metrics that matters to us.

Architecture

TFS is built for scale and to handle large volumes of data used by many users. Hence, there must be a saleable solution for reporting to support the analysis of the data. To solve this challenge we will not do heavy reporting on the online system, but instead use a separate data warehouse for analysis.

As you can see from Figure 32-1, data is replicated from the various real-time SQL Server tables in TFS to a set of reporting databases.

Figure 32-1.  TFS reporting architecture

Warehouse adapters transform the data from the operational store into the relational warehouse database. The adapter does not translate all data into the warehouse, typically only data that makes sense to do trend analysis are transformed (for instance work item history or test case steps are not available in the data warehouse).

Finally, data is aggregated from the relational warehouse into a Microsoft SQL Server Analysis Services OLAP cube. Online Analytical Processing (OLAP) is a business intelligence solution that enables data mining in large data volumes by restructuring the data in a format more suitable for analytics.

Note  Work Item data is added to the warehouse based on the reportable attribute on the work item field definition. The reportable attribute can have the following values:

- Detail: A field tagged as Detail is moved into the relational warehouse but not into the OLAP cube.

- Dimension: Data for this field moves both to the relational warehouse and the OLAP cube as an attribute that can be used to filter data. A typical example would be State or Iteration.

- Measure: The Measure fields are available as aggregated values in the OLAP cube, for instance Remaining hours and Completed hours.

- None: Field is not available for reporting in the warehouse.

Note  Go back to Chapter 7 if you want to learn how to customize a work item definition and set the reporting properties.

Data Sources for Reporting

As we saw in the previous section reporting data can come from primarily the following three sources:

- The Operational Store: The operational TFS database should not be used for reporting. This is just like most production databases because we do not want to impact the performance of the system with long-running reports. The exception here is reporting based on work item queries, where the data of course comes from the operational store.

- The Relational Warehouse (TFS_Warehouse): The relational warehouse is a SQL database that contains all the warehouse information. This database can be queried using SQL, which makes it more accessible for reporting.

- The SQL Server Analysis Services OLAP Cube (TFS_Analysis): The OLAP cube is a multidimensional database that aggregates data from the relational warehouse for more efficient reporting. This data store is ideal for trend analysis and well suited for use with Microsoft Excel.

Warehouse Adapters

Data in the warehouse is transformed from the operational store using a warehouse adapter. Because each tool in TFS typically has its own schema there is an adapter for each store that gets transformed into the relational warehouse.

Note  The warehouse adapter architecture is extensible. If you would like to get your custom data into the TFS warehouse (for instance code metrics or custom test results), you can create your own warehouse adapter to do so. It is outside the scope of this book to look at the details, instead have a look at http://msdn.microsoft.com/en-us/library/bb286956.aspx if you are interested in learning more.

Relational Data Warehouse Data Models

The relational data warehouse contains the data transformed from the operational store and is available for us as data in different schemas. We have previously gone through the details on the relational data warehouse in each of the Metrics and ALM Assessment chapters:

- Chapter 9—Project management

- Chapter 19—Development practices

- Chapter 24—Testing practices

- Chapter 29—Release management

Analysis Services OLAP Cube

The OLAP cube contains aggregated data from the TFS data sources, which is great to use for trend analysis. It is easy to use Excel and the PivotTable and PivotChart to create reports that answer questions such as the number of bugs per tester per day in a sprint.

To use the cube for analysis in Excel or SQL Reporting we need to understand the structure of the data in Analysis Services. Figure 32-2 shows the fundamental structure of elements in an OLAP cube.

Figure 32-2.  The structure of an OLAP cube

The cube consists of:

- Attributes: Attributes are connected to columns in a dimension.

- Dimensions: A dimension associates attributes to show aggregate results. You can use the dimension to show a trend or as a filter. The dimension can be shared among different fact tables (for instance Person or Date dimension are shared between fact tables).

- Fact: A fact is the data in the cube and can contain a large number of data rows. A row in a fact table contains either the value of a measure or a foreign key to a dimension.

- Measure: Measures are the values that represent columns in a fact table.

- Measure Group: A collection of measures tracked by the OLAP database.

- Perspective: A perspective is a selected part of the cube that can be used to make it easier to find the data for analysis.

The TFS OLAP cube consists of the following measure groups:

- Build: Contains metrics from the build process to help us get status reports, such as which builds failed and which succeeded, and trend reports, such as the build times over the past week.

- Code Churn: This measure group contains data from the version control system to get information, such as which files are changed most frequently or how much code has been changed in an area since the last release.

- Code Coverage: Metrics from code coverage during test runs to help us understand which code has the lowest code coverage or which test runs test the most code.

- Test: Various test metrics from test results and test runs that help us create reports, such as what is the test status for a particular PBI or how many tests are failing over time.

- Work Item: Data about work items in general that gives us answers to questions, such as what is the sprint burndown trend or how many PBIs were active in each sprint over the course of the project.

Note  The data warehouse is updated on a configured schedule. By default data is pulled from the operational store and written to the relational warehouse within two minutes. The SQL Server Analysis Service cube is then updated every two hours. You can change the refresh frequency by using the Warehouse Web Service as described on MSDN: http://msdn.microsoft.com/en-us/library/ms244694.aspx but do so cautiously because shortening the refresh time generates more load on the TFS server.

Setting Up Access to Report Data

To create reports we need to have permission to query TFS and the TFS warehouse. This list shows what is required for the different tools.

- Work Item Queries: For this we only need regular access to TFS because we are going over the standard TFS services to get data.

- Excel OLAP reports: To use Excel to do OLAP reports we need access to the Analysis Services warehouse.

- SQL Server Report Builder and Report Designer: For these reports we need access to the relational warehouse database or the Analysis Services cube.

Setting Up Access to the Relational Warehouse

To get access to the relational warehouse to create reports we need to be added to the TfsWarehouseDataReader security role in the Tfs_Warehouse database. To setup access follow these steps:

- 1. Start SQL Server Management Studio and connect to the Tfs_Warehouse relational database.

- 2. Use the Object Explorer and drill down to the Roles and open the TfsWarehouseDataReader properties.

- 3. Add the user (Dave in this example) to the TfsWarehouseDataReader role (see Figure 32-3). The user needs to have a login account to SQL Server and be added to the users of the database or else we cannot select the user in the role members dialog.

Figure 32-3.  Grant access to relational warehouse database

Setting Up Access to the Analysis Services Cube

Setting up access to the Analysis Services cube is similar to setting up access to the relational warehouse, this time we need to be in the TfsWarehouseDataReader role

1. Start SQL Server Management Studio and connect to the Analysis Services.

2. Use the Object Explorer and drill down to the Roles in the Tfs_Analysis database and open the TfsWarehouseDataReader properties.

3. Add the user (Dave in this example) to the TfsWarehouseDataReader role (see Figure 32-4).

Figure 32-4.  Grant access to the analysis services database

Monitoring the TFS Data Warehouse

The reporting system depends on a number of components working correctly together. The data that the warehouse adapters expect needs to be well-formed to be processed. Occasionally things will happen that make the warehouse stop updating. The most common reason to problems in the warehouse transformation is conflicts in work item fields between projects. When that happens, fields without problems will still be processed but the ones in conflict will not get updated in the warehouse until the problem has been resolved. To help troubleshoot warehouse problems we recommend the following resources:

- Resolving Schema Conflicts That Are Occurring in the Data Warehouse, http://msdn.microsoft.com/en-us/library/ee921480.

- Administrative Report Pack for Team Foundation Server 2010, http://blogs.msdn.com/b/granth/archive/2010/07/12/administrative-report-pack-for-team-foundation-server-2010.aspx.

Creating Reports

Data from the TFS can be extracted in many ways but unfortunately there is no best way to do this, instead we should try to find the best tool for what we want to achieve. The following list is the tools we have available for creating reports;

- Work Item Queries: Work item queries are simple to create but still a great way to generate real-time reports based on work item data.

- Excel Agile Workbooks: Excel workbook was introduced with TFS 2010 and is a great example on how we can use an Excel workbook as a shared source of reports, typically something that users in a specific role would use to get their follow-up be consistent. The specifics in the Agile workbooks include reports for backlog and iteration planning, something that has become partly superfluous with the new agile Web Access.

- Web Access Reports: The new Web Access features some nice reports on velocity and sprint burndown (as discussed in chapter 8 on agile reporting) on the team’s home page. We can add custom tiles to the Web Access page for other reports but we have to do that as a custom implementation.

- Excel Reports: Excel is a great tool in general for working with work item queries and the analysis capabilities and the possibility to hook up to the data warehouse cube makes it really powerful for ad-hoc as well as trend analysis.

- Create Reports in Microsoft Excel: This is a built-in feature in the Excel TFS add-in that we can use to quickly generate reports from a flat list work item query.

- SQL Server Report Builder: Excel is a great tool but of course it requires the user to have Excel installed to view the reports. If we want really accessible reports, we need to move to SQL Server Reporting Services and create custom reports. The SQL Server Report Builder is the first step to Reporting Services and gives the end-user the power to create powerful reports without knowledge of a development tool. It has its limitations but it’s a great start.

- SQL Server Report Designer: The final option is the full-fledged solution for advanced reporting. To create reports using the Report Designer, we use Visual Studio and the Business Intelligence Studio (BIDS) to design the reports. With BIDS we get precision control over the layout as well as the data sources, which allows us to merge data from all of TFS in a report.

The next question to ask about reporting is which data source should we use for creating the reports? Table 32-1 shows the combination of tools and data sources.

Table 32-1. TFS Reporting Tools with Relationship to Data Sources

Reporting Tool

Work Item Query

Relational Warehouse

OLAP Cube

Excel

Yes

Yes

Yes

Report Builder

No

Yes

Yes

Report Designer

No

Yes

Yes

Let’s now take a look at the different tools and see how we use them to create our custom reports.

Work Item Queries

Work item queries might be simple but they are still very powerful as a quick way of getting to live information about project activities.

Using work item queries in Excel is great for things such as

- Managing the product backlog

- Getting a list of open bugs

- Analyzing relationships between work items

Here are a couple examples. The first is an example on a basic product backlog report in Excel. Select the work item query.

Figure 32-5.  Opening a work item query in Excel

Next, add columns as needed and use Excel to filter data in the report.

Figure 32-6.  TFS reporting in Excel

Another example is a report of closed product backlog items without test cases. This report can be created using a link query such as the following:

Figure 32-7.  Reporting over links using a work item query in TFS Web Access

But this is about as detailed we get with work item queries. If we need more details or other TFS data, we need to move up in the tool chain.

Excel Reports

Excel reports against the OLAP cube are easy to create once you get a grip on the data in the cube. A great thing with Excel is that it is the best tool to play with the cube to get familiar with the data. So what would be better than to show an example on how to do this then? Let’s create a build status report!

1. Open Excel and use the Data tab to connect to the TFS Analysis Service database (see Figure 32-8).

Figure 32-8.  Connecting Excel to Analysis Services

2. Enter the credentials as needed; we use Dave here as he was added as a TfsWarehouseDataReader previously (see Figure 32-9).

Figure 32-9.  Connecting Excel to Analysis Services, authentication

3. Select the Cube or Perspective to report on. We need a little of several, so let’s go for the whole Team System cube (see Figure 32-10).

Figure 32-10.  Connecting Excel to Analysis Services, selecting database and cube for analysis

4. Next we can create a pivot table and a pivot chart. We want to create both in this case (see Figure 32-11).

Figure 32-11.  Connecting Excel to Analysis Services, selecting how to view the data

5. Now we can start adding fields to the report by adding fields from the Field List to the four areas of the report (see Figure 32-12).

Figure 32-12.  Working with Excel and Analysis Services, adding fields

6. By adding Project Path, Year-Week-Date, and Build Flavor to the report filters, the Build Status to the Legend Fields, the Date to the Axis Fields, and the Build Details Count to the Values and we get the graph in Figure 32-13. And, yes, you can filter directly in the graph to customize the report content!

Figure 32-13.  Working with Excel and Analysis Services, complete PivotChart

7. If you are happy with the result (of course you should be), save the Excel spreadsheet in a central location to make it accessible by others (the teams’ SharePoint site would be a great place). The users of the report need to have access to the TFS warehouse to run the data access queries.

SQL Server Report Builder

The SQL Server Report Builder is a report authoring tool targeted for business users. It is similar to the way we created a report in Excel against the Analysis Services OLAP cube but the end result will be a native SQL Server report that can be accessed by anyone in the team without installing Excel or having personal access to the reporting data.

To show how the Report Builder works, we will again create a Build status report.

1. Download and install the SQL Server Report Builder 3.0 from http://www.microsoft.com/en-us/download/details.aspx?id=6116.

2. Start the Report Builder and create a new report using the Table or Matrix Wizard.

3. Choose to create a new dataset using SQL Server Analysis Services, connecting to your server hosting TFS analysis services database (see Figure 32-14).

Figure 32-14.  Working with Report Builds, connecting to a data source

4. Design a query using the OLAP cube. Add the Build Details Count measure, the Build Status the Date dimensions to the result pane. Add the Team Project Path hierarchy to the filter pane and filter on the Expense Reporting project. Add the Date Week-Month-Date hierarchy to the filter pane and filter on the date range from January to June (see Figure 32-15).

Figure 32-15.  Working with Report Builds, adding data fields

5. Add the count field to the detail pane in the wizard and complete the process. Remove the resulting table and insert a chart using the Chart Wizard. Select the newly created dataset, add a Column chart, add the Date field to the Categories, add the Build Details Count to the Values, and the Build Status to the Series (see Figure 32-16).

Figure 32-16.  Working with Report Builds, mapping fields to areas

6. Finish the wizard and cleanup the generated layout. Press the Run button to preview the report (see Figure 32-17).

Figure 32-17.  Working with Report Builds, the complete report

7. Save the report. The resulting file can now be published to the Reporting Service portal.

SQL Server Report Designer

The ultimate solution to TFS reporting is to use the SQL Server Report Designer to develop custom reports. With the SQL Server Report Designer we can create advanced logic behind the reports such as creating any number of data sets for the reports or use custom .NET code to transform the data to fit the report. The same principles as for the earlier tools apply but with the Report Designer we need to be familiar with development using Visual Studio as well.

To get you started with custom SQL Server report we will create a build status report from scratch and deploy it to the TFS SQL Server Reporting Services portal.

1. To create a custom report we first need to install the SQL Server Business Intelligence Studio (part of SQL Server) and create a Report Server Project (see Figure 32-18). Name the project TFS Reports; typically you will have one project for all your custom TFS reports.

Figure 32-18.  New Business Intelligence project for reporting

2. Next we need to set up the data sources for the report. By using a Shared Data Source we can share the report data sources between reports in TFS. By default TFS adds the following data sources:

- a. TFS2010ReportDS. Data source connected to the relational warehouse database.

- b. TFSOlapReportDS. Data source for the Analysis Services database.

By reusing these names, our report will be easy to install later and will not need any custom database, which will make it easier to maintain the reports. Of course, if you need to get to other data you will create a data source specific for that purpose.

In the General tab, name the data source and set up the connection. We will connect to the Analysis Services cube to get the data for this report (see Figure 32-19).

Figure 32-19.  Creating a reporting services report data source

3. Next we can add a new report to the project. The Report Wizard walks you through the process of setting up the initial report parameters (see Figure 32-20).

Figure 32-20.  Creating a new reporting services report, select data source

4. Next we use the Query Builder to select which data is fetched for the report. This is very similar to the steps we previously saw when creating a report using the Report Builder. Add the Build Details Count measure, the Build Status and the Date dimensions to the result pane. Add the Team Project Path hierarchy to the filter pane and filter on the Expense Reporting project. Add the Date Week-Month-Date hierarchy to the filter pane and filter on the date range from January to June (see Figure 32-21).

Figure 32-21.  Creating a new reporting services report, query designer

5. Finish the wizard. We want to create a chart for this report so the table created is not of any interest to us this time.

6. Clean up the report area, remove the report table and make some room for the chart. Use the toolbox to add a stacked bar chart to the report. Map the data fields to the chart by adding the Date field to the Categories, the Build Details Count to the Values, and the Build Status to the Series (see Figure 32-22).

Figure 32-22.  Creating a new reporting services report, adding a chart

7. Use the Preview function to test the report during development. As long as the data sources are connected correctly you will get live data in the review mode.

8. Save the report and you are ready to deploy it to the Reporting Services portal for use by your team!

Deploy a SQL Server Report in Reporting Services

To deploy a report in Reporting Services we simply need to upload the report.rdl to the Reporting Service portal and configure the report parameters as required by the report.

Navigate to the Reporting Services portal and to the project where you want add the report.

Figure 32-23.  Uploading a reporting services report to the report portal

Next upload the file. The report now needs to be configured with the appropriate data source mapping and parameter. Open the report properties and finish the configuration, for instance selecting the data source as in Figure 32-24.

Figure 32-24.  Configuring a Reporting Services report

After the report has been configured we can run the report and verify that it works as expected (see Figure 32-25).

Figure 32-25.  Running a Reporting Services report

Those were the fundamental steps for creating a SQL Server Reporting Service report. Of course we can do many other things with these reports, including creating nested reports, adding parameters, and so on, but from a TFS data warehouse perspective this is what you need to get started.

Summary

Getting information about the project status is critical to most projects. By defining the metrics for analysis early we can make sure in the end we can also get the reports.

In this chapter we looked at the reporting capabilities in TFS and saw that we have a range of options for generating reports. We can use work item queries to quickly extract the current state of activities. That same list can then be loaded into Excel and turned into advanced, appealing reports using the graphing and pivoting capabilities in Excel. We can also use Excel to create trend reports from the TFS data warehouse. For more advanced scenarios we can use SQL Server Reporting Services and create the reports for detailed analysis or reports that collect data from multiple sources. We have additional features in the SQL Server Reporting Services, such as permission control and automatic scheduling of reporting.

CHAPTER 33

Working in Heterogeneous Environments

Organizations develop software using different environments: Some applications or components are built in Java and some in .NET. One of our current customers has to support its platform on both Windows and Mac OSX (a Java version of the Windows application). This means that the customer uses both Visual Studio and Eclipse for development.

Working in heterogeneous environments presents difficulties. For instance, how can we share our work items between the development environments? How can we share reports and statistics about the progress of our projects when working in different development environments? Sure, we can use web access, but switching context from the development environment to the web access costs time. It would be better if we could access the work items and reports directly from within the development environment, just as we do in Visual Studio.

Having most information about source control, builds, work items, and so on in a single repository would greatly improve the ALM process in a heterogeneous environment. This would also enable us to get rid of some applications that were used for supporting one platform only, hence cutting down on maintenance costs. Let’s face it, having two source control environments just because we have two development environments costs time and money to support and maintain. It’s simpler and more cost effective to use only one.

What is Team Explorer Everywhere?

Team Explorer Everywhere (TEE) can help us with problems we might encounter in a heterogeneous environment. TEE enables us to access most features of TFS inside our Eclipse environment, no matter if we use Windows or Mac as a development platform. The authors use it for accessing TFS from inside Eclipse on Mac OSX and it works great.

Originally a company called Teamprise developed TEE. On November 9, 2009, Microsoft announced that it had acquired Teamprise and that its plug-in for Eclipse would be developed as a Microsoft product. The result is what we now call Team Explorer Everywhere.

TEE gives us a single repository for source control, work-item tracking, reports, statistics and information, and much more. This greatly improves the ALM process and cuts down on the need for maintaining several systems supporting our different development environments.

A Little Bit about Licensing

In March 2012. Brian Harry announced (http://blogs.msdn.com/b/bharry/archive/2012/03/08/even-better-access-to-team-foundation-server.aspx) that before March 8 2012, Team Explorer Everywhere users had to purchase both a Client Access License (CAL) and the Team Explorer Everywhere software. Visual Studio Team Explorer users only had to purchase a CAL, since the Visual Studio Team Explorer software always had been a free download for users who had a license to access a TFS server. As of March 8, 2012, it is the same for Team Explorer Everywhere. We only need a CAL to TFS in order to use TEE. These are great news for organizations with a mixed development environment.

According to Harry, this change means that now that Microsoft has eliminated the licensing of TEE, it can finally set up a standard Eclipse update site that will enable us to keep Team Explorer Everywhere up to date, just as we do the rest of our Eclipse extensions.

Other exciting scenarios, according to Harry, are:

- No matter if we use Visual Studio-based Team Explorer or the Eclipse-based Team Explorer Everywhere, we can access TFS without any extra cost besides the CAL.

- If we use Team Foundation Server “Standard Edition,” we use either the Visual Studio-based Team Explorer or the Eclipse-based Team Explorer Everywhere with no additional purchases.

- Team Foundation Server Express enables a team of up to five people to experience much of the power of TFS for free. We can now use TFS Express along with Team Explorer and/or Team Explorer Everywhere all for no charge.

For some organizations, these are really great advantages, especially for those that have not already integrated their Eclipse environment into TFS. Now they don’t need to purchase TEE separately.

Features Available in Team Explorer Everywhere

Most features of TFS are available to Eclipse users through TEE (http://msdn.microsoft.com/en-us/library/gg413271.aspx). These include team projects, version control, team builds, and project management (including work item features). Some features, however—such as the ability to create team projects and set permissions—are not available from the Team Foundation Server plug-in for Eclipse. For those features, we must use Team Explorer in Visual Studio. We can live with these limitations, but it would be great if these were included as well.

Reports are not available inside TEE, so we need to use web access for these.

There is even a cross-platform command-line utility available for those of us who like and use the command prompt for our work. This utility works on UNIX and Linux as well, extending the use of TFS in heterogeneous environments.

Installation in Eclipse

Follow these steps (http://www.microsoft.com/download/en/details.aspx?displaylang=en&id=28983) to install Team Explorer Anywhere in your Eclipse environment on Mac OSX (the same steps apply to a Windows Eclipse version).

Open Eclipse. On the Help menu, click Install New Software (Figure 33-1) and the Install dialog box appears.

Figure 33-1.  Installing new software in Eclipse

- Click Add (Figure 33-2) and the Add Site dialog box appears.

- In Name, type Local Team Explorer plug-in archive.

- Click Archive.

- Specify the location of the archive file (TFSEclipsePlugin-UpdateSiteArchive-11.0.0-Beta.zip) and click Open.

Figure 33-2.  The install dialog opens

Click OK. In the list of features in the Install dialog box (Figure 33-3), select the check box that corresponds to Team Explorer Everywhere. Double-click Next.

Figure 33-3.  Select Team Foundation Server plug-in for Eclipse

If you agree to the Microsoft Software License Terms, Accept them and then click Next.

You must restart Eclipse when prompted (Figure 33-4).

Figure 33-4.  Restart Eclipse after installation

Connect to TFS

When Team Explorer Everywhere is installed and Eclipse is restarted, it is time to connect to a TFS server.

From within a Java project, right click the project and select Team/Share Project, which creates a connection to a TFS server.

Figure 33-5.  Connecting to TFS from Eclipse

In the next screen (Figure 33-6), you select which repository you want to use to share your project. Select Team Foundation Server and click Next.

Note  Sharing your project only means that you want to store your code in a source control repository. To access the features of Team Explorer Everywhere, you need to perform these actions.

Figure 33-6.  Select Team Foundation Server

Before you can continue, review and agree to the license agreement and then enter the address to the Team Foundation Server (Figure 33-7).

Figure 33-7.  Enter the name or URl of the TFS server

After pressing OK, you will be presented with the login screen to the Team Foundation Server. When you are done signing in, press Close (Figure 33-8).

Figure 33-8.  Close the dialog box

You are now presented with a form for selecting a TFS collection and one or more Team Projects (Figure 33-9).

Figure 33-9.  Select a TFS Collection and one or more Team Projects

Next we select which workspace we want to use for our project. Add one of your own or edit the default if you want to change anything (Figure 33-10).

Figure 33-10.  Select a workspace

Select which Team Project you want to use for storing your source code (Figure 33-11) and select Next.

Figure 33-11.  Select where to share your Java project

If you are happy with the changes, click on Finish to add your Java project to TFS.

Using TEE from Eclipse

Once TEE is installed, we can access it from within Eclipse (Figure 33-12). Let us start by checking in the source code we began this section with.

Figure 33-12.  Source Control options in Eclipse using TE

From the Pending changes menu, select Check In. This will trigger the check-in process, and we will have the same rules and policies available to us in TEE as we would in a Visual Studio environment. This means we can enforce the policies we need our developers to comply with in a cross-platform environment. Confirm the check-in by clicking Yes in Figure 33-13.

We can also see that we have several more options in our Team menu after installing TEE (Figure 33-14).

Figure 33-13.  Review the pending changes and select Yes

But where is Team Explorer itself? Team Explorer does not show by default. You need to go to Window, Show View, Other . . . , and select Team Explorer from the next form.

Figure 33-14.  New options are available from the Team menu

Now you will have Team Explorer inside Eclipse just as you would in Visual Studio (Figure 33-15), and you can access work items, builds, and more.

Figure 33-15.  Team Explorer from within Eclipse

Click on Work Items to access the work item queries. In Figure 33-16, we can see all work items returned by the All Work query.

Figure 33-16.  We can reach our Work Item Queries inside TEE just as in Visual Studio Team Explorer

We can open a work item from inside Eclipse, and it looks just like it does in Visual Studio (Figure 33-17). We have the same layout, the same tabs, and the same information.

Figure 33-17.  Viewing a work item from inside Eclipse on Mac OSX

In TEE, we can access some important settings for our projects (see Figure 33-18), such as project security, areas and iterations, and check-in policies.

Figure 33-18.  The project settings in TTE in Eclipse

All in all, we can access the most important features of TFS from within our Eclipse environment. If we want access to reports, however, we must go to the web-access page for the project.

Ant and Maven 2 Build Improvements

Previously there was not support for Ant or Maven 2 builds in TEE and TFS, making the ALM process a bit flawed. Microsoft has now changed this. The Team Foundation Build Extensions (http://visualstudiogallery.msdn.microsoft.com/3e3b3492-d78a-4829-9657-fc1cadba4ccb) provide the ability to execute Ant or Maven 2 builds from Team Foundation Server 2012 and publish the results of the build along with any associated JUnit test results back to Team Foundation Server. This very useful feature makes visibility and traceability in a heterogeneous environment much better, helping us improve an ALM process in mixed-development organizations.

Once we have installed the build extensions on the necessary build-agent machines, we must create a build definition that makes use of them. The easiest way to do this is to use Team Explorer Everywhere 2012 to create the build definition. We will not go into the specifics in this chapter; please check out the section on Releasing in Chapter 26.

When communicating with Team Foundation Server 2012, an MSBuild-based Ant or Maven build definition will make use of the UpgradeTemplate workflow process to execute a TFSBuild.proj script that actually calls Ant or Maven. This allows the entire process to be easily edited from Eclipse or from the command line.

Please note that these extensions will not work with earlier versions of TFS, but you can find extensions for TFS 2005–2010 at http://visualstudiogallery.msdn.microsoft.com/2d7c8577-54b8-47ce-82a5-8649f579dcb6.

Summary

In this chapter, we looked at Team Explorer Everywhere, a Microsoft product that enables us to access the features of TFS from inside our Eclipse environment in either Windows or OSX and hence allows us to include our Java developers into our TFS ALM solution. This is useful for organizations with a mixed-development environment.

In the next chapter we look at Team Foundation Services, an implementation of TFS in the cloud.

CHAPTER 34

TFS In The Cloud

Maintaining and operating (server) applications in an IT infrastructure is costly. You need staff for installation, configuration of the applications, and monitoring. TFS is no exception. As the use of TFS increases in a company, there is usually a need for scaling up and out the server platform to handle the load. The more important TFS becomes in an organization, the greater the need for maintenance and good backup/restore solutions.

Companies increasingly have started to use cloud-based solutions for much of their infrastructure functions. Vendors such as Google, Microsoft, and others offer cloud services for other companies and organizations. Microsoft, for instance, offers the Office365 concept including email, shared calendars, IM, video conferencing and document collaboration.Moving part or all of an organization’s IT services to a cloud vendor makes it possible to reduce the number of functions in the internal IT infrastructure. Hence, the cost associated with maintaining and operating these services will be reduced.

To address these issues, Microsoft has made Visual Studio Team Foundation Server 2012 available in the cloud, running under Windows Azure. This chapter will cover TFS in the cloud and the pros and cons associated with this. Let’s start by looking at the Windows Azure platform.

Windows Azure Overview

The Windows Azure Platform is a cloud application platform that allows Microsoft datacenters to host and run applications. It provides a cloud operating system called Windows Azure that serves as a runtime for the applications and provides a set of services that allows development, management, and hosting of applications outside of the organization. All Azure Services and applications built using them run on top of Windows Azure.

Windows Azure has three core components:

- Compute

- Storage

- Fabric

Let us take a look at these.

Compute

Compute provides a computation environment with

- A web role customized for web-application programming, as supported by Internet Information Services (IIS) 7 and ASP.NET. Web roles run in full IIS 7.0.

- A worker role that is useful for generalized development and that may perform background processing for a web role.

- A virtual machine (VM) role thatprovides a user-customized image to make it easier to move existing Windows Server applications to the Windows Azure hosting environment.

Storage

Storage focuses on providing scalable storage:

- Blob service, for storing text or binary data

- Queue service, for reliable, persistent messaging between services

- Table service, for structured storage that can be queried

Relational Database functionality is offered through SQL Azure, a version of SQL Server that runs on the Azure platform.

Fabric

Fabric is the networking underpinnings of the Windows Azure platform and uses high-speed connections, and switches to connect nodes consisting of several servers together.

Fabric resources, applications, and services running are managed by the Windows Azure Fabric Controller service. It acts as the kernel of the Windows Azure distributed-cloud operating system, providing scheduling, resource allocation, device management, and fault tolerance for the nodes in the fabric. It also provides high-level application models for intelligently managing the complete application lifecycle, including deployment, health monitoring, upgrades, and de-activation (http://en.wikipedia.org/wiki/Azure_Services_Platform).

The Azure Platform

The Windows Azure platform uses a specialized operating system, called Windows Azure, to run its fabric layer. This is a cluster hosted at Microsoft’s datacenters and it manages the computing and storage resources of the computers. It also provisions the resources (or a subset of them) to applications running on top of Windows Azure. Basically, Windows Azure can be described as a cloud layer sitting on top of a number of Windows Server systems. These use Windows Server 2008 and a customized version of Hyper-V, Windows Azure Hypervisor, to provide virtualization of services.

The platform includes five services:

- Live Services for communication and collaboration services

- SQL Azure (formerly SQL Services) for data-storage services

- AppFabric (formerly .NET Services) for cloud-computing services

- SharePoint Services for web-content-management and document-management services

- Dynamics CRM Services for customer-relationship-management services

As developers, we can use these to build the applications that will run in the cloud. A client library, in managed code, and associated tools are also provided for developing cloud applications in Visual Studio. Scaling and reliability are controlled by the Windows Azure Fabric Controller, which prevents the services and environment from crashing if one of the servers crashes within the Microsoft datacenter, and provides the management of the user’s web application, such as memory resources and load balancing.

TFS on Azure

Note  Keep in mind as you read this chapter that as of this writing, TFS on Windows Azure is still in preview. This means that much might change before the first version is released. The URL http://tfspreview.com will change, and many features might be different. The good news is that the version available at the time of writing is very stable, and open for everybody.

As mentioned, Microsoft now offers TFS as a cloud service. Instead of connecting to a local server, we connect to a cloud version of TFS. One difference though is that TFS in this case stands for Team Foundation Service, not server. This is to show that it is a service that Microsoft provides, not an application. Figure 34-1 shows our Eclipse development environment in Mac OSX connecting to a cloud-based TFS instance.

Figure 34-1.  Connecting to a web URL instead of an on-premises TFS server using Eclipse in Mac OSX

Getting Started

Getting started with Visual Studio Team Foundation Service in the cloud is easy. Microsoft provides a web-based signup procedure that enables teams to quickly get started with the service.

This is how you get started. Browse to http://tfspreview.com and see the screen in Figure 34-2. A video will tell you more about the cloud version of TFS.

Figure 34-2.  The Team Foundation Service logon page (at the time of writing)

Proceed by clicking Create Account. Fill in all details of the Account Signup screen and click Sign Up (see Figure 34-3). TFS will ask for a Windows Live ID. If you don’t have one you can create one from the web page directly. You also need to fill out the Server URL that all TFS users need to use when they connect to your instance of TFS. Once you have signed, in you are ready to get started (Figure 34-5). From the screen in Figure 34-4, you can start creating your TFS projects, add team members, watch tutorials, download necessary software, and more.

Figure 34-3.  Signup screen (at the time of writing)

Figure 34-4.  You can manage your TFS instance From this management screen

Figure 34-5.  Creating your first project

The first thing you need to do is to create a team project. Do this by clicking on Create Team Project. This will start the Create Team Project wizard (Figure 34-5). Choose the name of the project and which process template you want to use. You are presented with three options for the process template:

- Microsoft Solutions Framework for Agile

- Microsoft Solutions Framework for CMMI

- Microsoft Visual Studio Scrum

Once you are done, click Create Project and wait for the wizard to finish. This will take a few minutes, and then you are presented with a completion screen.

Once the project is created, you can go to the team project home page (Figure 34-6), where you are presented with a web page that looks exactly like the web-access page in a Team Foundation Server. This makes the experience really seamless if you are used to a local TFS.

Figure 34-6.  The team project home page

To access the new project from your Visual Studio development environment, enter the URL to the cloud TFS in the connect server properties (Figure 34-7). Now you are ready to develop using a cloud service.

Figure 34-7.  Connecting to the Team Foundation Service URL in Visual Studio 2012

You can now add your team members to the service. As TFS is hosted in the cloud, Visual Studio Team Foundation Server 2012 is available to all your team members regardless of location. They just need to sign in and provide their credentials.

This solution is ideal for geographically dispersed teams or for collaboration with stakeholders on remote locations. If you have a network connection, you can access Visual Studio Team Foundation Server 2012 from anywhere.

Documentation and Help

The tfspreview site offers extensive documentation (Figure 34-8). The documentation will take you through every step from creating your first TFS project to connecting Visual Studio to checking in code to creating work items to setting up agile project management, and so on.

Figure 34-8.  The documentation will help even an inexperienced user set up a hosted TFS instance

Services Offered With TFS in the Cloud

Using TFS in the cloud is very much like having a local TFS server on our site. We can do pretty much everything that we can do with a local install, including:

- Source control

- Work-item tracking

- Process templates

- Testing

- Build service

Let’s look at some of these to see what does and does not differ from a local TFS server.

Source Control

Source control works just as it does in a local TFS installation, which is good. We can access the source-control system from Visual Studio as we would in an on-premises scenario and from the web-management page.

Work-item tracking

Work-item tracking is available just as on a local TFS server. There are no real disadvantages to using the work-item system in TFS in the cloud. You can achieve the same traceability in the same way as you can on a local install.

The biggest drawback here is that we currently cannot change the work items provided in the service. This means we are limited to the work items Microsoft has selected for us and the information available in them.

Process Templates

This service differs a bit from the local TFS installation. For automation of high-level processes, which is one of the important ALM pillars (see Chapter 2), it is essential to be able to create new templates, adjust templates, or maybe delete templates. Being able to adjust TFS to support our process of choice is a great benefit of TFS. In the cloud version (as of this writing), however, we do not have the option to tailor TFS to accommodate our needs. We are stuck with the process templates that Microsoft offers and cannot add any of our own. Neither can we adjust the templates to add more information in, for example, the work items, or change workflows and so on.

Microsoft, we hope, will change this in the final or future versions. There is nothing wrong with the templates that Microsoft offers (MSF for Agile, MSF for CMMI and Microsoft Scrum), but it is a pity that we cannot change anything of this ourselves for now.

Testing

At this point, there is no special hosted TFS services for testing. We can, however, connect Microsoft Test manager to a cloud-based TFS and store our test cases and test results there, which is great. On the other hand, what would we be looking for as cloud-based testing services? Several interesting areas to consider include:

- Using the cloud to host a test environment

- Automatic deployment to Azure

- Load testing on a cloud-based test rig

For more details on testing in the cloud, switch back to Chapter 23, which has a section about lab management using Azure VMs. Chapter 22 talks about load testing on Azure VMs, and Chapter 27 discusses deployment to Azure as well as how to implement a build-deploy-test workflow on an Azure VM.

Hosted Build Service

With the Hosted Build service, Microsoft has made it possible to use a shared pool of hosted build machines (though we can still install and manage build machines on-premises with hosted TFS). We can do more than just build—as with on-premises TFS, we can run a default workflow that include compilation, testing, and more. We can also create a custom workflow that does whatever we like.

The build service works by maintaining a pool of machines that can expand and shrink as needed. When we start a build, a VM is allocated from the pool to run our build. The build is run, the build output is copied off the build machine, and then the VM is restored and it is returned back to the pool for someone else to use. Yes, the machine is wiped clean after every build, so we need not worry about someone accidentally getting our code from a previous build.

This means that we can avoid having to setup a local build environment if we don’t want to. For some (perhaps smaller) organizations, this can be very useful as they don’t have to invest in hardware and maintenance of the build environment, but instead can use the cloud version and still get the benefits of the build features.

Chapter 26, on build automation, shows how to work with the hosted build service, and Chapter 27, on deployment, explains how to implement a complete build-test-deploy workflow using cloud services.

What about Reporting?

Getting good reports from your ALM tools is essential to get visibility into the project and its status. In Team Foundation Server, there are lots of reports out of the box, and also ways we can create our own reports. The same reports are available when we use Team Foundation Service, however. We get the burndown chart and the velocity chart, as shown in Figure 34-6, but not much else.

Another drawback is that we cannot create our own reports, so we need to be happy with the ones Microsoft provides. This is a weak point for TFS in the cloud so far, at least from an ALM perspective. It would be better if Microsoft could extend the reporting functionality in Team Foundation Service.

Benefits of TFS in the Cloud

Let us take a look at some of the benefits of using TFS in the cloud. From the ALM overview in Chapter 2, we remember that collaboration is a key concept. Using TFS in the cloud can help us enhance collaboration, especially with external contributors and developers who travel frequently. It can also enhance collaboration in an organization with geographically spread locations.

Using TFS in the cloud, we do not have to set these users up in our local network. Neither do we need to provide VPN access and other security solutions for remote users ourselves, or to perform operating-system or software upgrades, make backups, patch the server, or respond in the middle of the night to hardware or power failures. This reduces the cost of maintaining such infrastructure, regarding network and hardware costs and also regarding staff costs, since Microsoft (or your selected vendor) takes on the burden of infrastructure security and monitoring.

By using TFS in the cloud, we can quickly and easily add new users to our TFS environment. We can also provide access to TFS wherever our users are located. All they need is an Internet connection and the correct rights in the TFS environment. In the end, this simplifies collaboration in our projects, since we do not have many barriers to get us started.

Since we have access to work-item tracking, build services, test tools, and almost everything available in a local TFS server, we can also get traceability and visibility in the same way as with a local TFS installation.

Another benefit is that the cloud based TFS version will be updated frequently, even more often than the updates will be offered to a boxed TFS version. Microsoft will push out updates as they create them on the cloud service.

Concerns with TFS in the Cloud

One disadvantage with TFS in the cloud is that we can neither add our own process template nor make adjustments to the existing templates. This means that we cannot adjust TFS to all our needs in the company. If we need more information in a work item, we cannot add fields or change work flows to suit our needs. Instead, we are stuck with the structure that Microsoft offers.

Another concern is the weak support for reports, which makes visibility suffer from an ALM perspective.

Summary

All in all, Team Foundation Service is a great addition to any development organization that does not want to maintain and support a Team Foundation Server installation on its own premises.

Team Foundation Services provides most of the ALM features we should strive for—with one big exception. We do get traceability, visibility, collaboration, and so on, but also we get (so far) limited control over the automated process, since we cannot change the process templates that are available to our teams. Or change these templates ourselves. One huge benefit of the hosted service is that it can evolve much faster than what we are used to, so expect the gaps we see today between the hosted service and TFS on-premises to close quickly. And then we will see the latest innovations appearing on the service first.

Index

A

ADO

Agile project management

ALM assessment

backlog

estimation

Kangreen organization

code sharing

high-level organizational chart

pilot project

psychologists, software

metrics

project startup phase

available time calculation

backlog building

backlog grooming

backlog items, adding

backlog order updating

capacity planning

DoD

epics

expense report project

forecast

initial sprint planning

PBI

Poker Planning/Story Points

project cost

release planning

requirements

risk assessment

team building

time plan

velocity

work item removal

requirements

scrum (see Scrum)

sprints

backlog grooming

daily scrum

planning meeting

reports and queries

retrospective

review

Agile software development

Agile testing process

acceptance criteria

definition

capability tests Q4

exploratory testing Q3

functional tests Q2

unit and component tests Q1

planning

automated testing strategy

test proportion

test specification and matrix

platform support

Application lifecycle management 1.0

Application lifecycle management 2.0

Application lifecycle management 2.0+

Application lifecycle management (ALM)

APM view

artifacts, traceability

assessment (see also Application platform optimization)

CIO view

development efforts, visibility

high-level processes automation

PPM

rangers build customization guide

roles

SDLC view

service management/operations view

tools

ALM 1.0

ALM 2.0

ALM 2.0+

configuration and change management

executable code building

issue management

production monitoring

release management

software modeling

software testing

system/application designing

system/application developing

Application platform capability assessment

code analysis

contributor

corrective measures

information gathering

interview

medium score

organization maturity level

owner

report

requirements handling

team assessment

architecture and design

deployment and operations

development

engineering and user experience requirements

governance

vs. individual assessment

project planning and management

software configuration management

testing and quality assurance

value of

versions

Application platform optimization (APO)

application platform capability assessment See (see Application platform capability assessment)

business productivity infrastructure model

capabilities

definition

maturity levels

goals

infrastructure optimization model

Application portfolio management (APM) view

Architecture analysis and design

ALM assessment

metrics

Architecture explorer

class view

dependency graphs

complex code identification

containment relationships

creation

projects and file types

sample graph

solution view

Automated testing

coded UI tests

action recording

architecture

associated automation tab

code generated

command line, TCM

core elements

data-driven

map editor

MTM

supported platforms

test case

test explorer

test plan

TFS build

unit tests

validation

Visual Studio project creation

supported test type, Visual Studio 2012

test environment

controller installation

infrastructure

test agent installation

test controller registers

Visual Studio lab management

traceability

query-based suite, MTM

test case

web performance test

add behavior logic

add validation rule

counter set

execution result

load pattern

load test designer, Visual Studio

load test running

load test scenarios

network and browser mix

request details configuration

run settings

scenario

test mix pattern

Visual Studio test designer

web test recorder

B

Bug status report

Bug trend report

Build agent

Build automation

build explorer

configuration

advanced parameters

basic parameters

default templates

high-level activities

required parameters

continuous integration

cloning

gated check-in builds

nightly build changes

running private build

creation

build definition (see Build definition)

continuous integration

Visual Studio team explorer

customization

build process template (see Build process template)

built-in build activities

TFS build extensions

extensions

code activity

solution development

XAML activity

feedback

build notifications

team explorer and web access favorites

hosted TFS

logging in

parameters

queueing

requirements

TFS build architecture

build agent installation

build controller installation

build service installation

components

high-level process

logical setup

web Access

Build controller

Build definition

build delete options

defaults settings

general settings

MTM working

process settings

retention policy settings

trigger settings

workspace settings

Build-deploy-test (BDT) workflows

BuildMessageImportance

Build notifications

alert

configuration

events

tool tray application

Build process template

activities addition

An AssemblyInfo file

assembly versioning

FindMatchingFiles

properties setting

workflow toolbox

WriteBuildMessage

custom assemblies location

custom build process checking

custom build workflow testing

implementation process

new build definition creation

new build template

process arguments

process variables

Business environment

collaboration gap

communication

geographic spread

globalization

organization, development process

PPM

project health

budget

goal fulfillment

time, delivery

project size

tools synchronization

Business intelligence (BI)

Business intelligence studio (BIDS)

Business process management (BPM)

Business software

Business system development

business rules

information

processes

C

Capability maturity model integration (CMMI)

Chief information officer (CIO) view

Class diagrams, UML

Expense Report class

attributes

operations

modeling elements

multiplicity

relationships

Client Access License (CAL)

CoCreateInstance

Code activity

build definition

build log in result

build workflow

code snippet

core implementation

process

SQL database

Code analysis

check-in policy

configuration

Enable Code Analysis on Build flag

FxCop community tool

GlobalSupressions.cs file

incorrect identifier casing

rule sets

TFS builds

window

Code churn data model

Code clones

Code coverage

builds

concept

to project

results

runsettings

testsettings

Code metrics

Code review

checkboxes

Incoming Review queue

initiation

reasons

review comments

source code history

Team Explorer

CollectionAssert

COM

ConnectionString

Continuous delivery

continuous integration (see Continuous integration)

null release cycle

optimized delivery process

typical delivery process

release management

Continuous integration (CI)

agile development

components

build automation

database integration

deployment

feedback

inspection

testing process

definition

implementation

advantages

disadvantages

rules and recommendations

Corporate Executive Board (CEB)

D

DataAccessMethod

Data-driven automated tests

Data management

Data warehouse model

build quality

build changeset tables

build coverage tables

build details tables

build project tables

current work item tables

developer practice

software testing

test results data model

test run coverage data model

work item

catagory tables

changeset tables

history tables

link history tables

test result tables

DefaultTemplate.11.1.xaml

Definition of Done (DoD)

development team

project startup phase

Dependency injection

Deployment

build-deploy-test process

build-deploy-test (BDT) workflow

build definition

build report

commands

configuration dialog

lab build template see Lab build template)

LabDefaultTemplate.11.xaml

Lab Management environment

script

test plan

topology

Visual Studio 2012 Lab Manager

database

PowerShell

running automated tests

build server

test environment

test failure

tools

WebDeploy

Windows Azure

Developer practices

ALM assessment questions

custom reporting

metrics

code analysis warning

code coverage

code metrics

compiler warnings

quality indicators report

Directed Graph Markup Language (DGML)

Domain Driven Design (DDD)

Drop folder

E, F, G

Empirical process control

Excel reports

adding fields

build status

credentials/authentication

database and cube for analysis

pivot table and pivot chart creation

TFS analysis service database connection

Extreme programming (XP)

H

Hand crafted mock objects

I, J

IBuildDetail

Information Technology Infrastructure Library (ITIL)

IntelliTrace

call naviation

call stack

debugger events

debugging information

enabling

exception events

explorer

framework events

performance wizard

in production

InvokeProcess activity

Iron triangle

IT budget spending

development vs. operations

influencing factors

K

Key performance indicator (KPI)

L

Lab build template

customizations

parameters customization

start and shut down

TFS build extensions

workflow activities

LabDefaultTemplate.11.xaml

Lab environment

Layer diagrams

code mapping

creation

dependencies viewing

patterns

sharing models

validation

architecture template creation

in local build

manual validation

TFS build

ListCustomers

Logical application tier

administrative tools

SharePoint products and technologies

SQL server reporting services

test controller

TFSJobAgent

web services

Logical data tier

M

Microsoft Feedback Client

email alerts

feedback recording

feedback request

feedback session

query

remote machine selection

stakeholder selection

web access page

Microsoft Office SharePoint Server

Microsoft solutions framework (MSF), Agile

burndown and burn rate

CMMI

remaining work report

status on all iterations report

stories overview report

stories progress report

unplanned work report

Microsoft test manager (MTM)

automated testing

bug creation

build definition

fast-forward playback

manual testing process

running tests

analyze

assign build

builds integration

data collection

exploratory tests

filtering

recommended tests

test runner

Visual Studio setting

test cases designing

acceptance criteria

definition

formatted steps

grouping and adding fields

moving on

parameters

sprint 1 test plan

suite status

summary

test configurations

tested backlog items tab

tester assign

test plan

creation

definition

moving on

property

query-based suite

requirement category and sprint 1

requirements-based suite

sprint 1 complete structure

static suite

TFS artifacts

TFS connection

verifying bugs

Mock frameworks

N

Null release cycle

O

Object factories

Online analytical processing (OLAP)

P, Q

PivotChart

PivotTable

Planning poker technique

PowerPoint Storyboarding

animation

creation

placeholder addition

Product Backlog Item

project web access page

storyboard shapes

software development

user stories

workshop and stakeholder meetings

PreEmptive analytics (PA)

configuring

feedback driven development

Inputs and Analytics attributes

installation

TFS

Process template customization

areas and iterations

Process Template Editor

Process Template Manager

reports

Team Explorer

work item

fields

queries

types

workflow

XML files modification

Product Backlog Item (PBI)

Profiling

event-based profilers

instrumenting profilers

outputs

statistical profilers

Visual Studio 2012

concurrency

instrumentation

.NET memory

performance analysis report

performance comparison reports

performance session function details

performance wizard

profiling session

sampling

tier interaction

Project Management Institute (PMI)

Project portfolio management (PPM)

Project success

performance variance

person-months

project length

Robert C. Glass

size characteristics

Standish Group

team size

ProviderName

PsExec activity

R

Rational unified process (RUP)

advantages

analysis and design discipline

business modeling discipline

configuration and change management discipline

deployment discipline

environment discipline

failure causes

implementation discipline

lifecycle

construction phase

development processes

elaboration phase

inception phase

transition phase

principles

project management discipline

requirements discipline

roles

task

test discipline

work product

Reactivations report

Release management

agile release planning

ALM assessment questions

custom reporting. (see Data warehouse model)

definition

metrics

release notes (see Release notes)

software development process

standard report

build quality indicators

build success over time report

build summary report

versioning (see Versioning)

Release notes

definition

technical writers

Visual Studio 2012

build report

Excel and TFS data warehouse

model

TFS build extensions build notes

Reportable attribute

Reporting, TFS

ALM platform

architecture

data access

analysis services database

relational warehouse database

tools

data sources

data warehouse monitoring

OLAP cube

relational data warehouse

report creation

excel agile workbooks

excel reports (see Excel reports)

Microsoft excel

SQL server report builder and designer (see SQL server report)

tools and data sources

web access reports

work item queries

reporting services report

warehouse adapters

Rollback

S

Scrum

complexity

definition

documentation

empirical process control

inspection

planning

process

product backlog

reports

roles

development team

product owner

scrum master

rugby approach

scaling

skeleton

sprint

backlog

planning meeting

retrospective

review

template

reports and queries

web access and portal

work item types

transparency

SOA

Software configuration management (SCM)

Software development lifecycle (SDLC) view

Software testing

ALM assessment questions

custom reporting

metrics

standard reports

bug status report

bug trend report

reactivations report

test case readiness report

test plan progress report

Spiral model

SQL server report

builder

business users

complete report

data source

mapping fields to areas

query design

SQL server report builder 3.0 installation

table or matrix wizard

designer

chart

data sources

query designer

reporting services report creation

SQL server business intelligence studio installation

Standish Group

Storyboards. See PowerPoint Storyboarding

Story points

StringAssert

Symbol server

System center virtual machine manager (SCVMM)

T

TableName

Team explorer everywhere (TEE)

Ant and Maven 2 builds

definition

Eclipse environment

pending changes menu

project settings

software installation

source control options

team explorer

team menu

work item queries

work item view

features

licensing

team foundation server

closing process

collection and team projects

name/URI

selection

share project selection

workspace selection

Team Foundation Background Job Agent (TFSJobAgent)

Team foundation server (TFS)

add-ins

administration

architecture

command-line tool

database

email alerts enable

maintenance and operations

management console

SQL Server and SharePoint

web features enable

windows server security

agile project management (see Agile project management)

architecture

logical application tier (see Logical application tier)

logical data tier

optional components, separate servers

team foundation client tier

branch hierarchy

build

activities

agent installation

components

controller installation

extensions

high-level process

logical setup

service installation

changeset tracking

cloud

benefits

disadvantage

hosted build service

process templates

reporting

source control

testing

work-item tracking

collaboration

community TFS build manager tool

development tool

extensibility

feedback tracking (see Microsoft Feedback Client)

functions and services

heart of

implementation planning

complex implementation

medium implementation

physical server location

ports and network protocols

service accounts

simple implementation

system requirements

installation

advanced configuration

application-tier

basic

configuration center

setup page

integrated development environment integration

Merge tool

Merge Wizard

Microsoft ecosystem

Microsoft office

My Work view

new branch creation

parallel development

power tools

PreEmptive analytics (PA)

process template

proxy

reporting (see Reporting, TFS)

source control

annotate tool

changesets

check-in policies

check-out settings

conflict resolution

diff and merge tools

explorer

Get operation

Get Specific dialog

history

labels

pending changes

rollback

shelvesets

team explorer

Visual Studio environment settings

workspaces

team foundation build

traceability

build management

high-level processes automation

release management

version control

work items

track action

upgrading

enabling features

keep existing hardware

new hardware

version control, logical topology

visibility

Visual Studio 2012 editions

Visual Studio 2012 suite

Web Access product

web client

Windows Azure platform (see Windows Azure platform)

work item tracking system (see Work item tracking system)

Test case readiness report

TestClass

TestContext

Test driven development (TDD)

Test Explorer

TestMethod

Test plan progress report

Traceability

TrackBuildMessage extension method

U

Unified Modeling Language (UML)

advantages

drawback

Visual Studio 2012

activity diagram

class diagram (see Class diagrams, UML)

component diagrams

Model Explorer

sequence diagrams

TFS 2012

toolbox with symbols

use case diagram

Unit test

automatic tests

external files

mock implementation

principle

basic rules

inversion of control

mock objects usage

separation of concerns

test dependencies

test driven development

Visual Studio (see Visual Studio unit tests)

UpgradeTemplate.xaml

User experience (UX)

V

Versioning

build

code

single branch pattern

TFS branching and merging guide

typical branch pattern

requirements

test cases

clone

existing test plan

work items

Visual Studio 2012

build report

editions

Excel and TFS data warehouse

Lab Manager

model

profiling

concurrency

instrumentation

.NET memory

performance analysis report

performance comparison reports

performance session function details

performance wizard

profiling session

sampling

tier interaction

release management

suite

supported test type

test lab management features

TFS build extensions build notes

UML

activity diagram

class diagram (see Class diagrams, UML)

component diagrams

Model Explorer

sequence diagrams

TFS 2012

toolbox with symbols

use case diagram

Visual Studio fakes

Visual Studio lab management

architecture

automated testing

command line

MTM

test case

test plan

capabilities

automated testing environments

build-deploy-test workflow

cloning environment

improved manual testing

network isolation

components

SCVMM

test agent

test controller

Kangreen lab

hyper-V and SCVMM

SCVMM environment

standard environment

template creation, SCVMM

test controller

TFS configuration

topology

lab environment

cloning

design

properties and definition

SCVMM

standard environment

test settings for

Vmware

manual testing

environment selection

environment setting

environment viewer, test runner

snapshot test

test plan

Visual Studio unit tests

assertions

code coverage (see Code coverage)

data-driven tests

elements

test calling order

Test Explorer

test run

third-party test frameworks

unit test creation

VSTS. See Process template customization

W, X, Y, Z

Waterfall model

Web access customization

Product Backlog Add Dialog

Product Backlog List

team Board

WITadmin

Windows Azure platform

compute

connect server properties

Create Team Project wizard

documentation

fabric

logon page

management screen

scaling and reliability

services

Signup screen

storage

team project home page

web-based signup procedure

Work item data

Work Item Query Language (WIQL)

Work item tracking (WIT) system

Bug form

process templates

queries

traceability

work items