Saturday, 24 October 2015


Observations of the space at present. 

Tongue in cheek of course


I don't usually offer much opinion in what I do. I  sit on the fence alot, maybe I shouldn't . However, when I do, I try to base what I know on what I really know, not on what other people tell me I need to know.  I'm sure what I am observing has been commented on before, and maybe other people agree or not, but reading a post on Twitter and then using it as your own keen insight  or shallowly cited insight is more of an indication of your lack of ability instead of  just keeping schtum or just saying "I dont know", where more respect would probably be gained. 

The above practice seems to be rife in software development at the moment.  I'd be a hypocrite if I said I hadn't fallen into the trap, but - like a recovering alcoholic - I have realised the errors of my ways and I am standing up and saying my name is Colin and I am a twitosopher (at times)  and shallow citer (at times). 

Shallow Citing. (Shiting?) 

The practice of citing something when you don't really have the full picture of what it is you are citing and/or name dropping at the same time .  I have only really noticed  this recently, maybe its  a larger organisation based thing which has pervaded the rest of the industry or  just that people are more aspirational or deluded in the places I have worked recently.   

I work in enterprise development and the most cited things are the testing triangle and Fowler (as in Martin Fowler).  The Shiter often uses a cite to gain the upper hand in an argument.  I mean just what is there be said if someone who is the Oracle on a subject has said it, what can I - as mere mortal - offer in the way of comeback or alternative.  It can be the argument/debate ender -  I'd encourage it not to be and challenge the citation for all its worth.  People have with me and have seen me coming up short.   

Is this the same where everyone works?  Don't get me wrong, things are often correctly and relevantly cited but, dig a little deeper , and asking the person just what it is they are referring to can often prompt some hand-wringing awkwardness.

Again, I have been guilty of this and have been caught out red-handed one more than one occasion. I felt embarrassed and felt that there's an element of charlatanism at play,  and I came to the conclusion that unless I can confidently back up what I am citing, then don't cite it - unless a disclaimer of some sort is provided.   

So for all the pseudo intellectuals out there, most people have got your number, but it is entertaining to watch you in all of your pumped-up, self-important glory, so carry on shiting. 

Monday, 31 August 2015

Sql Static Code Analysis (revisited)

Static Code Analysis.

You may remember my opening gambit in a post about SSDT code analysis talking about coding standards back in May 2014 (but most probably not)  It went something like....
Coding standards (and more latterly styles) in any organisation of any size are liable to incite religious wars and makes zealots out of even the most considered or apathetic of us.   Inconsistency and subjectivity,  lack of enforcement and heavy handedness all contribute to the contempt often levelled at them... blah blah

Contributing to solving the problem.

As part of one of our projects we had some funding to invest some time  and dev into SQL static code analysis, after researching what was available we had the green light to incorporate analysis written to leverage the SSDT code analysis framework into our own CI/CD ALM framework.  The appetite for this waned somewhat with the DB community having their own take on what they wanted to use, but the code lives on and some small bits of work continue to be done on it, when possible. 

Anyway, permission was given to publish our efforts.  I wasn't the sole contributor but continued/continue to run with what we had, looking to match , at least, the capabilities of other offerings in the arena but also crucially trying to ensure quality- by providing what amounts to integration testing -   and some time-saving features to developing code analysis rules.  The source code for all of this work is now published on GIT (with permission),but can be used as a basis for your own SQL static code analysis needs. 

It incorporates the work we did as well as the adoption and adaptation of the testing framework suggested by the SSDT code analysis team themselves.  

The code lives in a Visual Studio solution (initially written in VS2012) and is written in C#,  it was written against SSDT March 2014 however there are subsequent updates to SSDT currently at SSDT June 2015

You can find it here: Git repo

Hopefully it might provide you with a start to your own custom analysis needs. 

Wednesday, 11 March 2015

Async/Await Revisited

I put up an article back in November '14 about using async/await for a toy project I had created.   I have now had a chance to use it for "real work" and, as a result, have had a chance to research the language feature in more depth and have come up with a list of considerations, resources, gotchas and tid-bits.  

Must Reads

There is a lot of literature available for this feature and plenty of MSDN articles and Stack Overflow related help, however, I'd highly recommend the following books for greater insight into
TAP, TPL and async/await: 
Blogs by the latter two also cover this feature in great detail.

They all overlap in their coverage on async/await. As an aside, the first book I would recommend to any serious C# developer anyway, I'd say its on a par with the "C Programming language by Kernighan and Ritchie" which is a book I should probably buy again for posterity (something to show the grandchildren, when they are programming using some super tools) 

Considerations in an ASP.NET application

Using async/await in an ASP.NET application (WebAPI, MVC and Webforms)  is slightly different to using it in  a WinForms or WPF application.  

As I am sure you are aware, the UI thread is the main thread in desktop GUI applications. If async/await is used and the UI needs to be updated in a continuation after awaiting, then its is necessary to ensure that the UI thread is the one which is posted back to, in order to ensure updates are made on the UI thread and there are no cross threading issues. 

Thankfully, this is handled by default because of  the use of a SynchornisationContext.   This has the UI thread associated with it and allows work, from worker threads, to be posted back to the UI thread.   This incurs performance costs, both with the context switching of threads and the posting of work to the UI thread, but  this is inescapable, as the UI thread is the only one which can update the UI! 

In an ASP.NET the application threading model is rather different.  

The thread used for an initial request to an ASP.NET application will - likely - not be the same thread that is used for doing the work in  a continuation after an await. 

Because of this there is no requirement to use the SynchronisationContext, unless contextual information such as HttpContext or other information passed down the way on the initial request, is needed.  

The recommendation, therefore, is to use a method called ConfigureAwait - passing false -  whenever it is deemed unnecessary to have any of the context available from the initial request. (other contexts such as the execution context are still flowed - which has security settings etc).

In the snippet above we have a controller operation set up to work in an async manner.  The awaitable is configured not to use the SynchornisationContext.   This has smaller performance benefits in an ASP.NET application but is still a consideration to be aware of.

There is a gotcha I encountered in upgrading a WebAPI application from .NET 4 to .NET 4.5 (as opposed to creating a .NET 4.5 app from the off) concerning the use of the LegacyAspNetSynchronisationContext.  Ensure that if upgrading an ASP.NET application the following entry is added to the web.config (it is added by default if you create a new .NET 4.5 application)


Without it I had to use ConfigureAwait(false) everywhere to stop use of the aforementioned context, otherwise the application would fall over when trying to use it, I suspect because the thread initially associated with the context was no longer available.

Cost of async/await

Given all of the machinery produced as part of providing the async/await capability - the state machine - there are costs to bear in mind.   Some of the profiling detailed in "Async in C# 5.0", showed that a call to an async method can be up to 10 times slower, even when no asynchronous work is performed.   

So, if asynchronous work is not needed or is only needed in edge cases, paying this price is worth more than cursory  consideration. 

Interoperability with legacy asynchronous models. 

Interacting with older asynchronous programming models is possible using a number of approaches.

A helper method is available to create Tasks (and then await them) from the older APM model.(asynchronous programming model - BeginXXX, EndXXX methods and  IAsyncResult) 

The method is called "FromAsync".

"FromAsync"  takes the BeginXXX and EndXXX methods of the older APM style model  and returns a Task/Task<T>, with the Task containing the result when EndXXX completes. 

Alternately the EAP model can be interacted with by providing a "puppet task" by using a TaskCompletionSource  and hooking up the original async EAP method and callback event to properties on this class. The TaskCompletionSource also supports co-operative cancellation allowing a cancellation token to be passed in to control the execution of the operation being wrapped.

The TaskCompletionSource provides a wrapper to the older EAP methods and, when wired up correctly, will respond to the callback event when async work is completed and transition the state of the wrapped Task<T> and provide the result, if any. If there are any exceptions, or cancellations, the state and result can be set for these too.  

Once created, the Task<T> on a TaskCompletionSource can be awaited using async/await, thereby hiding consumers from the EAP or APM completely, which helps consistency and readability of code upstream. 

Unit Testing 

Unit testing async/await is pretty straightforward.  As far as I am aware the latest versions of  NUnit, MsTest and XUnit all support the ability to  execute an async bit of code without problem. (more on NUnit in a bit). As well as this the latest version of Moq has some nice extension methods to help out here too.   In my limited reading on unit testing support for async/await it is generally facilitated by having a some kind of context and related SynchronisationContext in order to ensure that methods can be awaited correctly.

Stephen Cleary says that it might be worthwhile testing synchronously and asynchronously as well as testing that exceptions thrown in an async method are appropriately handled. 

In the test below, we utilise the ReturnsAsync extension that Moq provides instead of writing

Worthy of note here and a potential future gotcha lies within the signature of the method above which is, "public async void when...."

Apparently in NUnit 3.0  (although I am using the 3.x Alpha above without problem) support for asnyc void will die and the method above would continue executing synchronously after the await (instead of waiting if we had introduced a delay) and potentially appear to have passed with an exception or failure rearing its head on another thread at some point in the future, causing unpredictable results. So instead the async method should have  return type of  Task or Task<T> just as you would write normally in production code.

Anyway  truly testing code asynchronously we can introduce a Task.Delay(x) or Task.Yield() to relinquish the thread momentarily, as below:

Here Task.Yield()  allows the currently executing thread to be returned to the thread pool and the call to service1.GetResult() executes truly asynchronously, awaiting Task.Delay(x) would work here too.  This kind of testing would be useful if wanting to test a timeout set on a cancellation token, for example.

Lastly, exceptions can be tested asynchronously too.
Unlike normal exceptions, exceptions which occur in an async method are placed on the Task and the await operation essentially unwraps the exception and rethrows it in the calling code (preserving - mostly -  the stack trace)
The accepted answer in this S/O post and a helper method in Stephen Cleary's book provide a way to test for async exceptions. Apparently NUnit will support this at some point.    Moq allows exceptions to be thrown asynchronously  ( using the ThrowAsync method). 

Mixing async and blocking code

Don't. There is the potential to cause deadlocks if mixing blocking and async code.  

Consider the following example: 

If a thread is executing in a method which awaits some Task - which is not yet complete - upon returning the same Task to the calling method, if the same thread is then blocked by calling a blocking method such as Task.Wait(),  a deadlock will occur when the awaited Task completes.  

This is because, when the awaited Task completes, by default, and owing to the SynchornisationContext captured when the await was invoked (which has the initial thread associated with it)  the thread associated with the SynchronisationContext would be used to schedule the continuation after the await, but as this is blocking due to the call to Task.Wait(), which is waiting to transition to unblock, with the transition only able to occur when the continuation occurs there is no way forward. 

This can be addressed by calling ConfigureAwait(false) on the Task being awaited, with the cost of another thread being created or, even better, by ensuring that all methods are marked with async and awaited appropriately(as well as dropping the call to Task.Wait()) and relying on the "waiting" provided by awaiting as opposed to explicitly blocking.  

As a rule of thumb, in order to facilitate scaling, blocking code should probably be avoided if at all possible.


The primary benefit of using async/await is the ability to use less resources, notably less memory but also the cost of context switching between threads.  

This benefit is primarily gained when using async/await with I/O (network or disk)   as the I/O work is performed by hardware and I/O completion threads independently of worker threads and  the thread pool.   

The magnitude of scaling can be quantified by load testing a  synchronous and asynchronous version of an application  (which my team did for a WebAPI component - which give expected results).   Keeping an eye on user requests/sec, failures and requests queued, the benefits really are tangible.   The ASP.NET and IIS performance counters can be used to gather information to ascertain the numbers. 

To get a good feel for the differences in resources used between sync and async, is to lower the I/O completion port threads and available thread pool threads to mimic contention more quickly and keeping any eye on the threads in use, you would definitely expect to see, as we did in the load testing we performed that the async version of the application used less resources than its synchronous counterpart.

This blog sums up getting a handle on the differences between  sync vs async/await in terms of resources used.


Async programming has been made almost trivial by the introduction of the async/await keywords in C# 5.0.

Bearing in mind a few of the gotchas I identified, there shouldn't be too many problems putting it in to practice in production and in unit testing.   

However, I wouldn't necessarily go async-ho and start adding it everywhere, I think that there should be careful consideration on whether or not it is going to provide tangible benefits before peppering code with the language feature, especially as it is pervasive and has some performance costs (and other costs such as learning curve, .NET upgrade impact, unit test updates, and adapting older async APIs - my point being its not free, cheaper in a greenfield project yes, but most certainly not cheaper in a legacy code base).   

To this end some load and performance profiling should be considered as well as not ruling out configuring IIS, the .NET runtime, database and network resources to remove the potential for bottlenecks, by increasing the number of threads available or adding memory or whatever else may be deemed necessary. (this is probably more applicable to retro-fitting this in a legacy codebase)

But anyway..... 

Happy asyncing.