Categorising Automated Tests

Categorising automated tests based on what they cover is very common. You call a test that covers a class with no/mocked dependencies a unit test. Tests that cover end to end journeys are then called integration tests or something similar.

This separation doesn’t help much in deciding which group of tests to run on a specific machine and when.

A more useful way of categorising is based on whether the test needs the solution to be deployed before it can run. This is simple to manage as you have only two categories. The boundaries are very clear also. More importantly, this allows you to get quick feedback by running the tests that do not require a deployment more frequently.

Advertisements
Categorising Automated Tests

Ix Operators

If you haven’t heard the term Ix, read my earlier post.

If you want to learn more, watch this video.

EnumerableEx.StartWith

You have an enumerable sequence of T. You want to add an instance of type T to the beginning of that sequence.

Test

[TestMethod]
public void StartWithTest()
{
  var count = 10;
  var zero = 0;
  var originalRange = Enumerable.Range(1, count);
  var newRange = MyEnumerableScratchPad.StartWith(originalRange, zero);
  Assert.AreEqual(zero, newRange.First());
  Assert.AreEqual(count + 1, newRange.Count());
}

Legacy implementation

public static IEnumerable StartWith(IEnumerable originalSequence, T first)
{
  yield return first;
  foreach (var item in originalSequence)
  yield return item;
}

Using Ix

public static IEnumerable StartWith(IEnumerable originalSequence, T first)
{
return originalSequence.StartWith(first);
}

Ix also has another overload for StartWith, which takes a param array of items to be inserted at the beginning of the sequence.

EnumerableEx.ForEach

You have an enumerable sequence and you want to perform an action for each item of the sequence.

Test

[TestMethod]
public void ForEachTest()
{
  var list = new List();
  var range = Enumerable.Range(1, 10);
  MyEnumerableScratchPad.ForEach(range, list.Add);
  CollectionAssert.AreEqual(range.ToList(), list);
}

Legacy implementation #1

public static void ForEach(IEnumerable sequence, Action action)
{
foreach (var item in sequence)
{
  action(item);
}
}

Legacy implementation #2

public static void ForEach(IEnumerable sequence, Action action)
{
  Array.ForEach(sequence.ToArray(), action);
}

Legacy implementation #3

public static void ForEach(IEnumerable sequence, Action action)
{
  sequence.ToList().ForEach(action);
}

#2 and #3 are not ideal as they both force the enumeration of the sequence before the iterating through the items begins. This is not necessarily a problem but there are some cases where you want to perform the action as soon as the sequence yields a new value.

Using Ix

public static void ForEach(IEnumerable sequence, Action action)
{
  sequence.ForEach(action);
}

Because most of the Ix operators including ForEach are ported back from the Rx world, there are some interesting additional overloads for ForEach. For example, there is one that takes action of T (the action that needs to be performed on each item) and then another action that will be performed when the iteration over the sequence is completed.

These three operators (Return, StartWith and ForEach) are among the simplest of the long list of Ix operators but nonetheless they are the ones I use most as they make code more concise and easier to read.

Ix Operators

Interactive Extensions for .NET or "Ix"

[Updated on 02 Jul 11 after the Rx official release]

Interactive Extensions used to be part of the Rx library but they can now be downloaded separately. Rx is now officially released and the stable version can be downloaded from here. Ix is now packaged separately and its experimental release can be downloaded from here. Both of these components are also available on the main Nuget feed.

Even many developers who are using Rx are unaware of this interactive part of the library and I believe it deserves more attention. But let’s start with Rx, just in case you don’t know what it is.

What is Rx?

Rx is a library developed by Microsoft, which allows you to write applications that react to events using a nice and clean implementation of the observable pattern. Rx started off as a Microsoft DevLabs incubation project but it proved to be very popular so its incubation period was ended in April 2011 and since then, it has been moved to the MSDN Data Developer Center.

What is Ix?

I refer to the interactive part of the Rx product family as “Ix”, which deals with enumerable sequences (IEnumerable<T>) whereas Rx focuses on the observable sequences (Observable<T>).

Enumerable and Observable sequences are duals

If you have watched Inside the Rx video on Channel9 or read Bart De Smet’s blog post on this topic, you already know that enumerable sequences are the duals of the observable sequences. When the Rx team created the operators for the observable sequences, they also back ported some of those operators to the enumerable world and that is how Ix was born.

Rx and Ix

The pre-release versions of the Rx product family consisted of various components such as System.Reactive, System.Interactive, CoreEx and System.Observable (for all runtimes except .NET 4.0). In the official release, the component structure is simplified and there are only two assemblies involved: System.Reactive and System.Interactive. All of the types required by those assemblies are now embedded in so for example, the IObservable and IObserver interfaces for runtimes other than .NET 4 now live inside System.Reactive. Same applies to the disposable and scheduler types.

If you are not using Rx family for whatever reason, it is now a good time to do so. I will go as far as saying:

If you are a .NET developer, then you should be using Rx and/or Ix in your projects or you are doing something wrong!

You might say I am taking it too far but I insist! If you are using Rx already, then you know what I am talking about. If you aren’t, you will know it when you start using it. Rx targets various platforms and frameworks including .NET (3.5, 4.0 for both client and full profiles), Silverlight (3, 4, 5), Windows Phone, Xbox and Javascript so you can use it in any .NET project.

How to Get Rx and Ix?

These extensions can be downloaded by following the links on the product landing page on MSDN. You can also find a variety of Rx and Ix components on the official Nuget feed.

What is in System.Interactive?

The primary type in Ix is the EnumerableEx type, which adds a long list of operators (in the form of extension methods) to IEnumerable<T>, plus a few operators for enumerable sequence of numbers (to support average, min, max, etc).When developers look at the list of operators provided by these extensions, most people ask “Why aren’t these included in .NET Framework anyway?”. I think (and hope) that this is just a matter of time before they make their way into the framework. Once you start using these extensions, you will soon find out that you cannot live without them, especially if you use LINQ frequently.

In the next post, I will briefly go through some of the Ix operators to show how it can help you in writing code that is easier to write and read.

Interactive Extensions for .NET or "Ix"

Accessing System.Windows.Application Properties From Other UI Threads

WPF applications have one dispatcher associated with their main UI thread. The main UI thread is the thread the Application class is instantiated on. When you create a new thread in the application, WPF allows you to create a new dispatcher for that new thread. This enables the windows created on the new thread to process the user input and because this is happening on a different thread, the processing of messages doesn’t  impact other windows in the application. This can be very useful if you are developing a responsive application with multiple views. However, you need to understand the impact of having multiple dispatchers on the WPF application model.

Like many UI elements, some of the properties of the Application object can be accessed from the main UI thread only. For example if you try to access the MainWindow or ShutdownMode properties from any UI thread other than the main one, an InvalidOperationException is raised saying you can only access it from the thread that created it.

The Application object also has a property called Windows, which is an enumerated list of windows created in the application. As you can expect, if you try to access this property from another thread, you will receive an error. Now what happens if you marshal the call back into the main UI thread by calling Dispatcher.Invoke (or BeginInvoke) or access the property from the main UI thread? It will only include the windows that were created on the main UI thread and it won’t enumerate the windows created on the other threads. If you need a list of all windows in the application, you need to manage this yourself. If your application creates windows on multiple threads, I don’t think creating a plain list of all windows created in the application would be a good idea anyway as you may need to handle interactions with those windows differently.

Accessing System.Windows.Application Properties From Other UI Threads

Providing a Reason for Stopping Project Collections in TFS 2010

When you stop a project collection in TFS 2010, it asks you to enter a reason why you are stopping that collection:

image

This text will be displayed to the users when they try to use an artifact in any of the projects in that project collection:

image

Note that this only affects those artifacts that are managed by TFS (work items, builds and source control items). Users can continue accessing the reports (handled by SSRS and SSAS) and documents (handled by SharePoint).

This is a minor feature but can be very useful when you need to service the project collection. It is still a good idea to make the users aware of the servicing plans in advance though!

Providing a Reason for Stopping Project Collections in TFS 2010

Model-View-*

I find myself answering questions related to the presentation patterns repeatedly. Since I am a lazy person, I am writing this post to illustrate my understanding and interpretation of the Model-View-* patterns and their variations so I can refer back to this post later.

[Update] I have discussed these patterns and their contexts with my colleagues Josh and Rupert and have updated the post the reflect the result of our conversation. Thanks guys!

I understand the concerns the Model-View-* purists might have around the correctness and/or pureness of these diagrams so I do apologize for potential inaccuracies in advance and I welcome any suggestions / comments!

I am not going to explain every single Model-View-* pattern as you can find better explanations in other places. Instead, I will focus on the relationship between participant components in each pattern.

Let’s start with a list of patterns we are going to cover:

  • MVC: Model-View-Controller
  • MVP (Supervising Presenter): Model-View-Presenter with Supervising Presenter
  • MVP (Passive View): Model-View-Presenter with Passive View
  • MVVM: Model-View-ViewModel
  • MVPoo: Model-View-ViewModel with some unwanted stuff (in the ViewModel)

MVC

This is a typical implementation of the good old MVC pattern. The Controller displays the View, make changes to its state and manage the navigation. The Controller can also update the Model (or ask the Model to update itself) and these updates are subsequently reflected in the View.

* What Happened to the MVP Pattern? *

As you may have noticed, the plain MVP (Model-View-Presenter) pattern is not in the above list. If you want to know why, read Martin Fowler’s note on retirement of the MVP pattern. Basically, he has split the MVP pattern into two: Supervising Presenter and Passive View. By looking at the next two diagrams, you will notice why they need to be separated.

MVPSupervisingPresenter

In the Supervising Presenter variation of the MVP pattern, Controller becomes responsible for implementing the complex view logic where the data binding between the View and the Model is not able to support that complex logic. Data binding is still the primary mechanism for the communication between the View and the Model and the Controller is used only in those scenarios where data binding support falls short.

MVPPassiveView

Passive View is the other variation of the MVP pattern and it brings the level of supervision of the Controller to the next level. The View no longer updates itself based on the Model and Controller becomes responsible for updating the View based on the data in the Model.

MVVM

In the MVVM pattern, the View Model inherits most of the functionality that would otherwise be in the Controller. The user gestures are captured by the View and the View uses the commanding feature of the underlying framework to invoke the operations defined in the View Model. Note that there is a dotted line showing the data binding between the View and the Model. This highlights the ability for the view to be bound to the Model via the View Model. Although the View Model can have its own properties the View binds to, but it is very likely for the View Model to publish the underlying Models as properties so the View is effectively bound to the Model (via the View Model).

MVVM makes an assumption that the framework the application is built on provides strong support for two-way data binding between the View and the Model. This diagram shows how simple the MVVM pattern is but this pattern would be really hard (if not impossible) to implement before WPF because of the lack of strong support for commanding and data binding. Even with WPF, implementing a non-trivial application that follows the design shown above can be tricky and will usually require a lot of additional helpers and constructs if we want to rely on data binding only.

MVPoo

The MVPoo (or M-V-poo to be more precise as defined by its creator, Dr. WPF) pattern acknowledges the fact that there is a difference between the ideal world and the real world so that nice and clean implementation of the MVVM pattern is not always achievable. As a result, there will be scenarios where:

  • View Model needs to update the View with mechanisms other than data binding.
  • The user gesture captured by the View or the events raised by UI components needs to be handled by the View Model with mechanisms other than commanding

So it is very likely that we will end up with a stinky version of the MVVM pattern. The MVPoo pattern acknowledges this fact and says this is okay but we need to try and make the components (with specific attention to the View Model!) less stinky as much as possible (by keeping the number of non-ideal elements/features in the View Model to a minimum).

Model-View-*

Deployment and Test Automation

In Application Lifecycle Management (ALM) consulting engagements with customers, we usually cover automation of various activities in the development process. Most of the development teams I work with have an automated build system in place. The automated build usually goes as far as compiling the solution(s) and then running some unit tests to verify the build. True value of the development process automation can only be realised when we take that final step and automate the deployment and testing process too. It is very hard (if not impossible) to run any integration tests in such environment because there is no deployment so the tests we can run will be limited to those that are related to individual units or those that rely on fake components.

When I ask development teams why they don’t take that final step, this is what I usually hear back:

  • Our environment and deployment process is very complex so we can’t automate it.
  • We have a shared database server we use for development and it is constantly changing so we don’t have a reliable baseline to test against.
  • We don’t store our configuration files in the source control because it contains sensitive information.
  • We use different environments for development and testing and every one of those has its own settings so which one should we keep in the source control?
  • We need some data to be available in the database for testing. The database is changing, the base data set is changing too.
  • If we want to include the deployment and testing in the process, it is going to take at least a few hours.

None of these are a good excuse for not automating the deployment and testing. Points similar to the ones listed above highlight problems somewhere else in the development process so even if we don’t want to automate the deployment and testing, we are still going to be affected by those inefficiencies and problems.

For example, if our argument is that the database schema is constantly changing (by a different team and probably on a different release cycle), this means when we decide to have a release, we have no idea if the latest version of the code is in sync with the database schema and scripts at that point. We may be lucky and manage to release in few hours but next time we may end up spending a day or two sorting out inconsistencies. The typical solution to this is to make sure the database artifacts are always in sync with the source code and are stored together with the code in the source control. Historically, maintaining the database scripts (especially the DDL change scripts) in source control has been a huge problem but today, we have great support from development tools in the market for managing the database artifacts (and their changes). Many of the teams I talk to are simply unaware of the developments we have made in these areas.

The point about having a dependency on the base test data is interesting too. If we don’t have a reliable set of test data for the code we are working with, we will end up copying a database from a previous release and then start making changes to that database to bring it in line with the latest version of the database schema. And guess what? Every developer needs to go through this error-prone process. As the testing team and other developers start testing, they will discover new sets of data that allows them to test new scenarios but since all of the developers have spent a considerable time on creating their “precious” test data, they will hang on to what they have. I am sure this is one of those experiences we have all had and we can spend a lot of time on maintaining our test data instead of implementing features for our product. The easiest solution to this challenge is to maintain a set of base test data and store it together with the source code in the source control. Yes somebody needs to maintain this but we are in a better position as less people need to be worried with maintenance of that data and other developers will get the latest test data when they get latest from the source control. This base test data can also be used by the automated processes for testing purposes.

Implementing an automated deployment and testing process is not trivial but is certainly achievable and there are things you can do to make it simpler. If you ask me to come up with #1 guideline in this area, I will say put all of the artifacts required for deployment in the source control but even more importantly, make sure they are in the right location! Nobody questions why we need to put the source code under source control but people usually miss other artifacts such as database artifacts and build and deployment scripts. The interesting point is that many teams actually put these artifacts somewhere under source control but they put it in the wrong location, which makes them less effective and those artifacts will be unusable when we want to automate deployment and testing. So what is the right location for the artifacts?

The basic idea is that all of the artifacts required for deployment should follow our branching hierarchy.

image

For example, if we have three branches (DEV, MAIN and REL), each branch needs its own copy of the artifacts. Why? Because we want to be able to deploy and test any of those branches independently. As you can see this is a fairly obvious (and simple) solution but many teams don’t do this for various reasons and that makes the automated deployment and testing very hard (or impossible).

image

As I said earlier, even if we are not going to automate deployment and testing, we will still benefit from following this structure and making sure all of the artifacts under each branch are consistent and in sync with one another. Once this structure is in place, the next step is to make sure we have the capability of creating and managing the target environments as we will be deploying and testing over and over again. [I am writing the next few sentences with my “good corporate citizen” hat on…] This is where tools such as Visual Studio Lab Management can help. Many developers and testers I talk to still haven’t heard of this great product or don’t know what it does so if you fall in that category, you may want to have a look at the resources (product team blog, videos) to find out more about it. Please note that at the time of writing this post, Visual Studio Lab Management is still in pre-release status so make sure you read the pre-release license terms (rtf) before installing it.

You will be amazed when you see how easy it is to automate the process of managing the target environments (which usually consists of one or more virtual machines) as part of the build process in TFS 2010.

image

The above diagram shows the list of activities provided out of the box and you can use the workflow editor for the build to customise the end-to-end process.

Deployment and Test Automation

Creating Objects Without Calling the Constructor

When you use the new operator to create an instance of a class, the class constructor is called before the reference to the newly created object is returned. This also applies to the reflection scenarios where you call Activator.CreateInstance, which finds the right constructor based on the number, order and type of the parameters provided.

This is the expected behavior in most cases as you don’t want to work with an object that might be in an invalid state. However, there are cases where you don’t need to run the constructor. A good example is when you are deserializing an object from a byte stream. During deserialization, you will be using the values embedded in the incoming byte stream to initialize the member variables so there is no point in initializing those members in the constructor as they will be overwritten.

The formatters that ship with the .NET Framework (such as BinaryFormatter and DataContractSerializer) follow this approach as you can expect. There is a utility class called FormatterServices, which provides some of the underlying services to the formatters, including but not limited to object creation.

namespace System.Runtime.Serialization { public sealed class FormatterServices { public static void CheckTypeSecurity(Type t, TypeFilterLevel securityLevel); public static object[] GetObjectData(object obj, MemberInfo[] members); public static object GetSafeUninitializedObject(Type type); public static MemberInfo[] GetSerializableMembers(Type type); public static MemberInfo[] GetSerializableMembers(Type type, StreamingContext context); public static ISerializationSurrogate GetSurrogateForCyclicalReference(ISerializationSurrogate innerSurrogate); public static Type GetTypeFromAssembly(Assembly assem, string name); public static object GetUninitializedObject(Type type); public static object PopulateObjectMembers(object obj, MemberInfo[] members, object[] data); } }

 

 

When you call GetUninitializedObject or GetSafeUninitializedObject, an instance of the class is created and returned without calling the constructor of the class. But note that the member variables are set to null and zero. FormatterServices is a public class so you can use its published functionality inside your application but you need to be very careful when using Get(Safe)UninitializedObject. As explained in the MSDN documentation, when you use these methods you should immediately populate all fields to make sure the object is not left in an invalid state.

There are some other interesting operations in FormatterServices. For example, GetSerializableMembers returns a list of members that will be serialized by the classic formatters.

One final point on this topic. If you are deserializing an object from a byte stream and want to run a piece of code to initialize the object before it is used (for example to calculate some of the fields that were not included in the serialized format), you will need to use the serialization attributes. See my earlier post on this subject for more information.

Creating Objects Without Calling the Constructor

Unshelving a Shelveset to a Different Branch in TFS

[Note: In this post, I am assuming that you are familiar with the TFS Branching Guidance. If you haven’t read the guide, I strongly recommend that you do so! I will be referring to the branch names suggested for the basic branch plan.]

Scenario:
After an application is released to the production environment, a defect has been raised by the users. The production support team decides to fix this defect before the next version is released so they start by modifying the appropriate files in the REL branch to fix the defect. When the developer working on the defect finishes his job, he shelves those changes and asks the support team lead to review those changes before they are checked in. The support team lead analyses the changes and because of the risks involved, she talks to the users and they agree to wait until the release of the next version to get the fix. 

Challenge:
When you create a shelveset, the server path of each shelved file is stored with the shelveset so when you try to unshelve that shelveset, the files in the same branch are modified. You cannot unshelve the shelveset to a different branch such as DEV so you will need to do this manually by unshelving to the original branch and then manually copying the modified files across, which is time consuming and error prone. 

Solution:
TFS 2008 Power Tools includes a command line tool (TFPT.exe), which allows you to perform advanced operations in TFS. One of these operations is the unshelve command, which allows you to change the server path of the files in a shelveset during the unshelve operation. This means you can use this command to unshelve any shelveset to a different branch. In our scenario, the shelveset was originally created in the REL branch so the development team can use the TFPT tool to unshelve the shelveset to the DEV branch. 
So let’s assume that the support team have changed the source files in the REL branch and we want to unshelve this shelveset to the DEV branch. In order to do this, we need to: 
– Make sure the TFPT command line tool is installed!
– Go to the command prompt and change the folder to a location that is mapped in the desired workspace.
– Run the following command:

    TFPT.exe unshelve ReleaseHotFix1 /migrate /source:$/MyProject/REL /target:$/MyProject/DEV
          

Where ReleaseHotFix1 is the original shelveset name and $/MyProject/REL and $/MyProject/DEV are the server paths to the REL and DEV branches respectively. 

When you run the unshelve command with the migrate option, the tool will prompt you to choose which version of the file (in origin or target) you want to keep. It will also allow you to perform a file merge if needed, which will be done in the merge tool.

The following diagram shows the sequence of events that need to happen across multiple branches in our scenario: 

  
Events: 

  • S1: Production support team shelve their changes in the REL branch.
  • U1: Development team unshelve the shelveset to the DEV branch by running the TFPT tool and using the shelveset migrate option.
  • C1: After resolving any potential conflicts in the DEV branch, the modified files are checked in.
  • RI1: At some point, the checked-in changeset will be reverse integrated (merged) back into the MAIN branch.
Unshelving a Shelveset to a Different Branch in TFS

Spanish (United States) Locale

Somebody asked me how to use the Spanish locale for the United States. If you are running Windows XP or Windows Server 2003, there is no such locale supported by the operating system. If you try to access this locale in a managed application by writing this line of code:

CultureInfo cultureInfo = new CultureInfo(“es-US”);

you will get an ArgumentException, saying “Culture name ‘es-US’ is not supported”. As a workaround, you can use the “es-MX” locale or create your own locale.

Support for the “es-US” locale was added in Windows Vista and Windows Server 2008 so you can now use the es-US locale (LCID = 21514) and the same line of code will work fine.

This page has a complete list of locale IDs assigned by Microsoft so if you are thinking of creating your own locale, check this list first to make sure there is no conflict with the existing ones.

Spanish (United States) Locale