xUnit.net vs NUnit, a quick pragmatic comparison

At the beginning of our last project, which was a greenfield project on .NET Core, I was responsible to choose a testing framework, isolation framework and all tools and frameworks, related to unit and integration testing. So I started searching for a decent testing framework for .NET Core and came up with 2 major opensource candidates: NUnit and xUnit.net. In this article, I’m going to briefly go through the features of each framework and share my final verdict!

I’ve been using NUnit for many years with no problem whatsoever. So I tend to have a bias towards NUnit. My task was choosing the best testing framework though. So I put my experience and comfort a side and tried not to have any familiarity bias, as learning curve of a new testing framework wouldn’t be that steep.

NUnit
nunit-logo

NUnit is actually ported from a Java testing framework named “JUnit”. It’s a mature framework with a long history started from JUnit since 1998. Widely used in dotnet community, highly documented and at the time of writing this article 19225 questions are tagged as NUnit in stackoverflow.

xUnit.net
xunit

xUnit.net is a relatively new testing framework, written by original inventor of NUnit v2. At the moment, xUnit.net is the latest unit testing tool, and well-accepted by .net foundation. Designed to be extensible, flexible, fast and clean. So far, there are 4008 questions tagged as xunit in stackoverflow.

NUnit xUnit.net
Supported Platforms  UWP, Desktop, Windows Phone, Xamarin Android, Xamarin iOS, ASP.net  UWP, Desktop, Windows Phone, Xamarin Android, Xamarin iOS, ASP.net
.Net Core Support  YES YES
IDE Tools Support  Visual Studio, Visual Studio Code, Resharper  Visual Studio, Visual Studio Code, Resharper
CI Tools Support  TeamCity, VSTS, MSBuild, CuiseControl.net, Bamboo, Jenkins  TeamCity, VSTS, MSBuild, CuiseControl.net, Bamboo, Jenkins
Parallel Execution  YES  YES
Execution Isolation Level Per test class Per test method
Extensible Test Attributes  NO (Test and TestCase attributes are sealed)  YES (Theory and Fact attributes are extensible)

As you see, both NUnit and xUnit are strong, mature and widely adopted by .net community. So it’s not easy to say either of them is not good enough.

To me, the way that xUnit.net runs the tests is admirable. By default, it creates a new instance of the test class for each test method. This means that test methods are completely isolated and do not interfere each other. So it actually mitigates the risk of dependent test methods which is a bad practice. However, if in some rare cases you need to share the context among several test methods, there is a way to do that.

Another pro of xUnit.net over NUnit is more extensibility and flexibility. We can inherit from Theory, Fact and other attributes. Personally I don’t care about this feature though. As it never happened to be useful in any of the gigs I’ve done. But anyway, xUnit.net designers have had extensibilty in mind.

NUnit on the other hand, has a longer history. There are many more projects already done via NUnit, you can find many more documents, samples, and discussions about NUnit in developers’ communities. i.e, you can find 5 times as much as xUnit.net questions for NUnit in stackoverflow website.

The bottom line is: I’d rather xUnit.net over NUnit, mainly because of the way that it runs the tests, whereby test methods run in separate test class instances in isolation, it can also increases the level of parallelism. I like It’s nice style of coding as well.(using constructor/Dispose instead of ugly TestFixtureSetup and TestFixtureTearDown attributes)

N.B: I have also made a Powerpoint presentation to talk about the chosen tools and frameworks plus a quick introduction to TDD. You can see the slides in here.

Advertisements

Does TDD really matter?

Test Driven Development (TDD) has been around for a quite a while (since 2003) and nowadays, somehow works like a buzzword in developers’ resumes.

Many decent companies around the world would highly consider TDD skills, experience, and more importantly TDD tendency of their job applicants, in their recruitment process. Sometimes, their candidates’ TDD adherence even matters more than any other skill such as knowing about new frameworks and technologies. But why is that? Does TDD really matter in real world projects or it’s yet another buzzword in developer’s recruitment?

unit-testing-joke

In this article I’m not going to bother writing yet another article about TDD, as there is already more than enough. The thing that I’m going to focus here is: what difference it would make on the software design and software and code quality as well as software developer’s level of expertise.

Let’s have a quick look into TDD to see what is it about and then we’ll get back to our main point:

By definition Test Driven Development or TDD is a software development process that relies on the repetition of a short development cycle. Each cycle consists of the following steps:

  • Add some tests
  • Run the tests and check if they all fail
  • Write some code to pass the tests
  • Run the tests and check if they all pass
  • Refactor the code if needed
  • Repeat the cycle for the next requirement

test-driven-development

First of all, once we’re talking test, we mean automated test as opposed to end-to-end manual test. Which means we’re writing code to test the actual code!

Generally speaking, we’ve got the following 3 types of automated tests (specifically functional tests):

  • Unit testing : which tests one and only one method(function) regardless of all the dependencies of the method (such as database, file, network resource, etc.)
  • Integration testing : that tests a combination of methods and components to see if they work properly while integrated with each other.
  • End-to-end (E2E) automated testing (using tools such as Selenium or Protractor in AngularJS)

In TDD we could have all 3, traditionally though, we mean Unit and Integration testing once talking automated tests.

At first as a developer, it doesn’t make sense to write a code that tests something which doesn’t actually exist. But that’s one of the main points. The tests actually represent the specs we are about to implement. All the components in our software are there because we are expecting them to do something. So before writing the code in a TDD manner, we need to be clear about specifications (expected behavior) of the system. If we have clear specs we would be able to write the code that exactly needed.

So let’s highlight the first point in TDD : be clear about the software module’s specs . Why is it important? To answer this question you need to experience working in both TDD and non-TDD teams. Well … I can bet you on this, If you do TDD for a while and then get back to a non-TDD environment you can feel the chaos, reworks and ping pong game between the Business Analysts (BAs) QA (End to end testing team) and developers. Which is a bloody vicious and tedious game.

In the future I’ll be writing about BDD (Behavior Driven Development) which is a kind of evolution in TDD in terms of having clear specs, I’m not going to get into more details on that topic here though.

When you write test and then the code, you are actually covering the code’s health and integrity with your unit tests. This means that if you develop a component and in the future another developer comes and changes the component to add a new feature, and his code breaks some functionalities of your code, he will realize the problem as soon as possible. So he’d be able to fix it easily and briskly. Even if he’s a careless developer and doesn’t run the tests before pushing to the code repository, using a CI (Continuous Integration) server, everyone in the team will receive a notification email which indicates the change has broken the code and caused a test to fail.

So another benefit we can get from TDD is : protecting our code from breaking changes and enabling the team to find the bugs at the very beginning of their occurrence. Remember though, that we need to do code review plus Continuous Integration (CI) in conjunction with unit testing and integration testing  to achieve a robust mechanism to protect the code against breaking changes.

Note that, Code review plays a crucial role in this process, for instance if a developer changes the code and tests in a way that tests are wrong and pass with a wrong code then we’re screwed! :))) Therefore we can say that reviewing tests is even more substantial than reviewing the actual code!  (I’ll dedicate separate articles on Code Review and CI/CD quite soon.)

To me, the most important plus of doing TDD is its impact on design and quality of the code. Writing code in a TDD fashion is subject to writing testable code which takes a different style of software design. At first, testability might seem trivial, but quite a lot of value is buried behind it.

A testable code, tend to adhere SOLID design principles which are fundamental object oriented design principles and the root of many design patterns and best practices. SOLID is actually an abbreviation of 5 basic principles:

  • (S): Single responsibility Principle
    Which says: each method should be only responsible to do one and only one thing.A class should be responsible to do a single job.
  • (O): Open/Close Principle
    Your code should be open to extension and close to modification. Which means too extend the software’s functionality, we should be adding code rather than modifying the current code.
  • (L): Liskov Substitution Principle
    Derived classes must be completely substitutable with their base classes. (This principle is subject to more explanation which is not relevant to this topic)
  • (I): Interface Segregation Principle
    Classes should not be forced to depend upon interfaces that they don’t use.(This principle is subject to more explanation which is not relevant to this topic)

  • (D): Dependency Inversion Principle
    Depend upon abstractions instead of concrete classes. Which means we need to write a code that all dependencies are interfaces or pure abstract classes (with no implementation) rather than concrete types (non-abstract classes). In fact, Dependency Injection which is a widely used design pattern is based on this principle. I’ll cover this topic later on, in another topic.

Writing testable code, forces developers to stick to the Single responsibility principle. As we need to write unit test for our methods and each unit test should test only one thing. So a method with multiple responsibilities would be hard and cumbersome to test.

We have to apply Open/Close principle while doing TDD, because if we do so, we can just add new code and write test for that specific extension rather than modifying the code and its pertinent tests. Therefore, a developer that breathe in a test driven environment, tend to adhere Open/Close principle by using best practices and design patterns (such as strategy, decorator, bridge, etc.)

More importantly, the need for testability makes developers to use dependency injectionin code to be able to replace the actual dependencies with a fake implementation. For instance consider a repository class that relies on a database context object (say it’s using Hibernate session classes or Entity Framework’s DbContext object). If this database context is injected into the repository, we could unit test the repository by faking the DB context object and without any need to connect to the actual database. Technically, it’s called testing in isolation. Once we’re unit testing, we should isolate the component or the system under test (SUT) from all of it’s dependencies.

You can see a code I’ve pushed into github, as a sample of unit testing repository classes to make a better sense of how dependency injection facilitates unit testing.
To see the sample repository class click here and to see the unit test class for that repository click here!

Using dependency injection, would result in a clean component decoupling in our app which is a crucial factor in software design. And test driven attitude would force developers to write loosely coupled classes to be able to test them in isolation.

The point I’m trying to make in this article is that a test driven attitude is not just about writing automated tests for our code, but rather leads to an immense impact on our design and coding style as well. In fact, a developer with TDD skills and experience is the one who a professional development team can count on.

To wrap up, I would say “Hell YES!!! TDD really matters and is a must for a developer who wants to work in a professional development team.” So if you want to be a real developer, start learning or brush up your TDD expertise, rather than whining about the companies that excessively care about TDD!