Sunday, 10 September 2017

End to End testing using Reactive Extensions (RX)

A change of tack is needed with my posts.  The limited people who have read my blog have been too kind to say that I talk lots of unnecessary shit.  I read back over my posts and I am musing like some kind of pseudo-intellectual who is smiling smugly and patting  himself on the back over some kind of  new insight into stuff which is really just a regurgitation. So I should stop it and keep the musings to a minimum from here on in. I've rewrittten this post to try and give something valuable that can be taken away and used.

Requisite Knowledge


General C#, Generics, Messaging Patterns, NServiceBus, Linq, Understanding of asynchrony, Task<T>, Unit testing.


Reactive Extensions 

I decided to use Rx to help end to end test a payment method service.  I wont go into a lot of detail about Rx, and instead refer you to an absolutley fantastic reference, which can be found at reactivex.io. The visualisations -  especially -  help bring to life what Rx does and can help stimulate ideas about how it can be used for you.    Rx.NET is what the code examples (in C#), here, are programmed against. 

The problem

I need to be able to test that my payment method service (service) is working by ensuring that, given various input,  it is producing the correct output (messages) in a timely fashion. 

I need a way to be able to ask the following questions  "Given xyz input, has a payment accepted event occured?", and if so "Does it relate to this payment reference and has it arrived within10 seconds?".  If so, great my service is working, if not, then it isn't.  

The service to be tested comprises a front end (RESTful like API) and a backend server (producing messages).  From a testing perspective, appropriate input should be supplied via the API and then various messages from the server should occur as a result.  This is going to need end to end testing.   (Qualification:  So I am saying we need  end to end testing IF we want to exercise a fully working service with all its real world external dependencies AND this is necessary as the API and server reside in different processes.  We could  still component test the API and backend server individually by swapping out external dependencies  but this wouldn't provide us an integrated test .  There are advocates and detractors  of  end to end testing along with its tradeoffs.

My experience is that, if not insanely difficult or expensive  (in time, complexity, and maintenance) it is worth it for the confidence (edit: and coverage) an end to end test can bring.


The services in our platform use NServiceBus (NSB) to send and receive what are usually domain events (edit: synonomous with messages), we can leverage this to get a handle on these messages and inspect them or lack of them.

To be able to ask the questions above I need to be able to observe (listen) on a channel for the messages  (an observable sequence) and then run a test asking relevant questions. 

In this instance the channel will be an event handler sitting atop an associated "bus" (set up not shown here)  - an abstraction NSB provides to represent the medium on which messages are passed. 


The handler:

public class GenericMessageHandler : IHandleMessages<object>
    {
        public void Handle(object message)
        {
            Observer.OnNotify(message);
        }
    }


The handlers job is to listen to messages which arrive at the handler and notify the Observer. The Observer is a facade for what is known in Rx.NET as a ReplaySubject, which is a special type of Observer (unfortunately my facade reuses/overloads the term Observer), the Observer has or will be setup with a bunch of subscribers which  will be the tests (or a way to set the results of the tests**) to run for a certain type of message.

The ReplaySubject allows buffering so that if a "would be" subscriber (which we will be using to hook up tests)  is not yet subscribed, and messages have started occuring on the channel (that is: arriving at the handler and being pushed into the ReplaySubject buffer)  then when the subscriber is subscribed all messages which have happened will be replayed to the subscriber.  

This special property (subject to limited buffering) means its not  important that a subscriber is hooked up straight away, as long as the Observer (the ReplaySubject) is getting messages and, so, in this way the chances of missing something on the channel are reduced. 

It is worth nothing that  NSB uses publish/subscribe so some setup has to be provided to ensure that messages of certain types are subscribed to and therefore will arrive at the handler. This code is omitted here. ( available in git shortly).

The Observer facade has 2 users:
  • the channel
  • unit tests
The channel pushes in messages to the Observer
The unit tests will be set up to "ask the questions" and provide the tests against those messages and these tests will be "attached/subscribed" to the Observer.




The Observer looks like this:


public class Observer
    {
        private static readonly ReplaySubject<object> MessageObservable = new ReplaySubject<object>();

        public static void OnNotify(object message)
        {
            MessageObservable.OnNext(message);
        }

        public static Task<T> SubscribeToMessageTest<T>(Func<T, bool> test, string failureMessage, TimeSpan timeout) where T : class
        {
            var tcs = new TaskCompletionSource<T>();

            MessageObservable
               .Timeout(timeout)
               .FirstAsync(message => RunTest(message, test))
               .Subscribe(
                   message => OnSuccess(tcs, message),
                   exception => OnFailure(exception, tcs, failureMessage));

            return tcs.Task;
        }

        private static bool RunTest<T>(object message, Func<T, bool> messageTest) where T : class
        {
            var convertedMessage = ConvertMessage<T>(message);
            if (convertedMessage != null)
            {
                return messageTest(convertedMessage);
            }
           
            return false;
        }

        private static T ConvertMessage<T>(object message) where T : class
        {
            var messageAsType = message as T;
            return messageAsType;
        }

        private static void OnSuccess<T>(TaskCompletionSource<T> tcs, object message) where T : class
        {
            tcs.SetResult(ConvertMessage<T>(message));
        }

        private static void OnFailure<T>(Exception ex, TaskCompletionSource<T> tcs, string failureMessage)
        {
            Console.WriteLine(failureMessage);
            var messsageException = new PublishTestException(failureMessage, ex);
            tcs.SetException(messsageException);
        }
    }



The most interesting method is:
Task<T> SubscribeToMessageTest<T>(Func<T,bool> test, string failureMessage, TimeSpan timeout)


At a highlevel then: a test is passed in which has previously been setup to be  executed against a specific type of message (when a message occurs on the channel and appears in what is known as an observable sequence) 
A TaskCompletionSource is created (tcs) whose  result is going to be set with a message if the associated test is successful or, an exception if it takes longer than a certain time.  A test is always unsuccessful if it can't be satisfied within the specified timeout period. 
The task from the TaskCompletionSource is then returned. 


The parameter "test" is a Func<T, bool>  (a generic predicate) to be hooked up to test elements in an observable sequence (the messags from the channel), to  MessageObservable (which is a ReplaySubject<object>. 

This is done with the lines:
MessageObservable
.Timeout(timeout)
.FirstAsync(message => RunTest(message, test))

Firstly,  via Timeout,  a timeout is set (if this is exceeded an exception is thrown)
Secondly, via FirstAsync, return ONLY the elements from the observable sequence  (the messages from the channel) which pass the test passed to RunTest.  The RunTest method does some conversion on the message, and runs the test. 


The next 3 lines:
.Subscribe(
  message => OnSuccess(tcs, message),
  exception => OnFailure(exception, tcs, failureMessage));


Subscribes the message  to a couple of methods which set the state of  the TaskCompletionSource (tcs) declared at the top of the method. 
OnSuccess takes the message itself and puts it into the result of the tcs.  
OnFailure occurs when a timeout occurs and the exception is put on to the exception on tcs. 
Not forgetting that the task associated with tcs was returned earlier and so consumers of the code would be notified as and when this occurs.


The SubscribeToMessageTest method is facaded with a PublishTest<T>:

    public class PublishTest<T> where T : class
    {
        private Task<T> task;

        /// <summary>
        /// Test a message that satisfies a predicate, before a timeout occurs.
        /// </summary>
        /// <typeparam name="T">The type of the message to test.</typeparam>
        /// <param name="predicate">A predicate with tests you need to determine the correct message has been published.</param>
        /// <param name="failureMessage">A message to aid debugging the test failure.</param>
        /// <param name="timeout">Timespan indicating how long to wait for a message to be received.</param>
        /// <returns>Task which should be awaited or Result inspected.</returns>
        public PublishTest(Func<T, bool> predicate, string failureMessage, TimeSpan timeout)
        {
            this.task = Observer.SubscribeToMessageTest(predicate, failureMessage, timeout);
        }

        /// <summary>
        /// Block whilst waiting for test to verify. Use where async/await is not supported.
        /// For example, SpecFlow does not currently support async/await.
        /// </summary>
        /// <returns>Message associated with successful test.</returns>
        public T Verify()
        {
            return this.task.Result;
        }

        /// <summary>
        /// Verify the test at a later point in time by calling .Result or using async/await on the Task returned.
        /// </summary>
        /// <returns>Task with message assocated with successful test.</returns>
        public Task<T> VerifyAsync()
        {
            return this.task;
        }
    }




A test can be written Which is then used like so in a test:

    [Then(@"a PayPalPaymentIntentAcceptedInternalV4 is published")]
        public void ThenAPayPalPaymentAcceptedIsPublished()
        {
            new PublishTest<PayPalPaymentIntentAcceptedInternalV4>(
                message => message.PaymentReference == this.context.PaymentReference,
                "No PayPalPaymentIntentAcceptedInternalV4 message was published matching the specified test.",
                this.context.MessageVerifyWaitTime).Verify();
        }



The lambda - which provides the actual test -  (message => message.PaymentReference...) and the type "PayPalPaymentIntentAcceptedV4" are used in tandem to qualify that firstly a test for this message type should take place and that the message is associated with input supplied earlier at the API, which would have the same payment reference.  The lambda/test is  ultimately, what is passed to RunTest in the Observer class (which was listed earlier). 

**As I am writing this I can see some improvements I would like to make to this code and will add an edit (also adding proper code snippets).  I have left out some of the set up code around  NServiceBus  and the handler and setup to subscribe to messages.  Also the other parts of the tests are omitted for brevity.  ( To be included in git). 
 


















Wednesday, 12 April 2017

Micro Post: Big Glass of Sand

Monolithic architecture is classified as a Big Ball of Mud
Microservices architecture is therefore a Big Glass of Sand

I said it first , right here.

Tuesday, 7 March 2017

Testing, Testing, 1.2.3.4.5.6..ad infinitum

Before testing...

I've now been in development/engineering/coding/hacking or whatever you want to call it for more than 10 years.  Testing has always been important, but it  - in my line of work - is now at the forefront of everything I do.   

For the business this is great, every line of code, component and group of components is covered by some set of tests. Be it unit, integration, component...the overloaded terms go on.   This means that a set of requirements -  to some degree  - have been determined to have been met by these tests and the business is getting what it wants (all manner of development and agile practices not withstanding - they all get us to the same place)

It struck me today though that earlier in my career, when testing was still important but lines were  - perhaps - a little more blurred and things were a bit more hit and miss, things were less clinical, less contrived and quite frankly less boring.  This is obviously not a good model for delivering quality software everytime but sometimes I yearn to literally hack out some code,  build it  and just experiment when things aren't working quite right.  Yes this can be done guided by tests and within the confines of the TDD cycle but, for me, having not come from a TDD beginning, feels a little stifiling if I was developing a hobby project (not something I was contributing to on GIT) I, dare I say it, would not necessarily use TDD - but make no mistake, at this time, developing quality software for my employers I would not develop without TDD, it gives me confidence I never had in the past. Boring yes, Confidence inspiring yes.

This kind of work is left now for spiking  or hobby projects, but it feels that, and although this term is bandied around,  software engineering  really is coming of age. 

Engineering, Not art/craft 

I often hear that software development is a craft or an art and that it is creative - this may well be the case for UI related disciplines but business software is the geography teacher of practices, boring and wearing tweed.    Once all the requirements are down and how these are going to be met are sorted  - that's it. Software is being driven out in a very prescriptive manner.  The biggest debates that happen in my team are not religious wars about coding standards - we have static code analysis, not religious wars about performance - we have resources to burn,  not religious wars about  general designs issues we have SOLID (and other guiding design guidelines), but no - we have wars about how we are going to test something and where.  ATDD brings this war right to the inception of a story/feature (sorry I often get these the wrong way around) and then we have a list of work todo and a checklist to say when its done.   This is all very organised and delivers value,  but I'd never realised how boring this actually is when you break it down. 

Summary

This might sound negative,  and boring is generally not considered good, but it is a matter or perspective.  Boring is good, as  I said before, for the business.   Now, I enjoy going to work and doing a good job, but I am wondering what it is I actually like doing anymore, maybe I have become boring and that's what I now like.  That's what my Mrs thinks anyway...

 


 

Tuesday, 6 December 2016

1..2…3…Freeze….Peak



I haven’t written anything for ages as  I’ve been the busiest I have ever been at my current employer – a large E-Commerce Website -  as a result of preparation (planned and unplanned) for its largest ever “Peak” weekend, my company’s name for the Black Friday weekend.    

Sorry this is a long read so grab a coffee or skip to the Summary, or don't read it!

I’ve started writing this on the train coming back from London where I was required to provide extra-support for the Payments Platform (Domain)  - where I ply my trade – a group of Microservices and Legacy components supporting Payment processing for the Website and sitting very closely to the Orders Domain. 

I expected to be bleary eyed and wired to the hilt, full of coffee and adrenaline as me and my colleagues worked like mad to pull out all the stops to make our giant incredible machine work properly. But no, as our CIO put it, everything just worked (well 99% of the time – more on this later). 


The freeze part of the title refers to the fact that, in the weeks up to Peak, as we tried to ensure stability of our software, only the most critical software releases were allowed out.  I , like many of my peers, do not think this is great.  This is costly in many ways; not least, all of the feature development still going on actively within the company is shelved and starts to gather dust and not being released means the software is accruing debt on many fronts.  However, I can’t be too hasty in judgement, this way of working comes from something along the way biting my company really hard during this crucial trading period, this mentality is understandable if not immediately completely excusable.  Hopefully, this will get better as the company gains more maturity, confidence and control over its systems.

Application Insights

As part of the preparations it became clear that our services did not have sufficient monitoring capabilities.   Services were performance tested in quasi-production like environments and so we had some idea about how they might perform, but only an idea.    So we had some expectations about performance but very little in the way of helping us see operational health .  Measuring and keeping  track both would have been impossible without telemetry and logging of some sort.    

We already had some monitoring available via dashboards powered by Grafana. A number of dashboards exist showing Order flow, and, as a result, part of an Orders path through Payments, but only at a high level.  With various counters representing the number of Orders at a particular status, e.g. arriving from the Website, currently being billed, and shipped to the warehouse, this is used to help try to generalise the performance and health of backend systems as Orders wing their way through the Website, through Order and Payment Processing and much more and then, eventually, out to the Warehouse(s). 

If problems start occurring, by some thresholds being exceeded or not met,  represented by numbers and traffic light boxes, focussed investigations into issues can take place in a particular area.   The problem with this is that in between the different statuses there are a vast swathe of software services in locations all over the world involved: including New-World Microservices, Legacy systems, stuff in the sky, Messaging, RESTful APIs, document DBs,  SQL DBs, No-Sql Dbs, Edge caching, and hardware on premise.  Application support have lots of fun trying to diagnose errors and, where they are at a loss ,they hand over to software development teams/developers who mostly run around in blind panic at the same time.

More recently with the release of a updated versions of the Website (which is multi-platform/device) New Relic was enlisted in helping see how the various APIs consumed by the Website were behaving and this has helped to start to build a more fine grained picture of the operational behaviour of APIs   ( and by association the Websites) where various HTTP statuses can be obtained and detailed analysis about response times, payload sizes, customer locations, can be gleaned, but that’s about it in terms of insight into what an API is doing. There is yet another monitoring tool called Kibana (sitting atop of the ELK stack) which is used for some of our Azure based services, largely this gives transparency to telemetry just like New Relic. My money’s on New Relic in this area though.

So, the introduction of performance counters, custom and otherwise were conveniently placed at the door of our APIs, on the 3rd party calls within them and in and around the messaging infrastructure used in the backend processing for our services.  Relevant logging was added too, to ensure that in the event of failure or exceptional behaviour we had some trace of what was going on.  So, having all of this information, but not at your fingertips is a bit of a nightmare. 

Consolidating this all into one single place is achieved with dramatic effect using Application Insights hosted in Azure.    Simply adding appropriate libraries in the service and hooking it up to an App Insights Resource Group in Azure and boom you are presented with a bewildering array of options to present you with your telemetry.    Coupled with this you can also leverage Analytics to perform queries on all of the data collected and herein lies a problem.  Making sense of all of this information is hard.  Certainly, there are some headline acts easily noticeable but more subtle problems like a semi dodgy 3rd party call, or reasonably flaky database writes can be eked out but not after some considered tweaking with Queries etc.    Also without reasonable SLAs from the business in most places and the fact that a message queueing system is used, what alerts should be used and when is a question we are still asking ourselves.

Long nights and Instability

So getting to a good place a lot of pain needs to be had.  And boy did we experience our fair share up to peak.  With SAN migrations causing mayhem with our messaging,  Windows Cluster failovers  misbehaving and deleted Orders in Azure there was plenty of time to practice and gain experience in trying to solve problems  quickly and still meet “customer promise”  a term used to describe meeting cutoff for various delivery options, which is not taken lightly. 

Graph Watching First Blood Part III

So the culmination in all of the Black Friday prep is a series of graph watching sessions, with eyes trained on graphs rolling around in multi-tabbed browsers and lots of suggestions and conjecture about what this trend is saying and what that trend is saying.   I was personally involved in 4 of the 5 days of Peak support, remotely on Thursday, Black Friday some of Saturday  and on premise (by demand) on Cyber Monday.  Certainly, at the beginning of the event focus was very high and everyone was apprehensive about what might occur and what problems might need to be addressed.  Previous year’s Peak had seen all manner of catastrophes including site outages and various problems.


   
The biggest problem this year, then,  from a Payments perspective, was that a Legacy component on dealing with timeouts with a Payment Service Provider couldn’t reliably recover and compensate  from the timeouts and so Orders would be left in a state where we didn’t know whether they had been billed.  Simply retrying the Orders meant risking a cancellation occurring as the legacy component is pretty dumb, so a quick support tool was knocked up and like a scene from Lost, a button was pushed every so often to make sure these were processed properly. 

The highlight though is that everything, generally, worked.   On Cyber Monday, due to a failure with a voucher service Orders dropped off a cliff.  Fixing the problem and spreading the word to our persistent and patient customers meant that between 9-10pm an average of 55 orders as second were being processed and all of this would land at Payments door step. The system coped admirably and a backlog of messages, of which there were up to about 40k at one point (spread across our system) were slowly but surely dealt with.  

Summary

The Payments Platform have now got a lot a data regarding capabilities during spiky load periods, a greater operational understanding and has breathed a big sigh of relief having been able to support Black Friday.   Thing is, its going to be even bigger next year and we are just about to start preparations in updating our Platform to be even more resilient, scalable and available.  Looking forward to it.  We will have a lot more control of our systems come this time next year and a maturity and experience to put what we have learnt to good use.  

Up Next....Reactive Extensions or maybe even HAL (Hypermedia Application Language)




Sunday, 5 June 2016

Learning to TDD is like learning to drive a car

Its true

I'm sure this comparison has been made but I'll have to tire it out a bit more. When I was learning to drive I was pre-occupied with where to put the gear stick, where it actually was, when to use the clutch and so on and so forth and being flustered all the time.

But, quite quickly, with continued practice and with growing familiarity and confidence with a car, I could think about where I wanted to go instead of every detail of what I needed to do, to get there, and now I drive like Lewis Hamilton.

The parallels between learning to drive a car and learning to test drive application development are striking.

Before  I could start to appreciate the nuances of TDD and be able to decide what works for me (for example choosing when to use classic and mockist TDD) I had to become comfortable with some pretty basic things including a good refactoring tool - ReSharper, a good test runner and Unit testing framework like NUnit and a good isolation (mocking) library, like Moq (all of these tools have their proponents and detractors)  but this is what I use for the majority of the TDD I do at the moment, this could change, but the principles won't.

I've been TDD'ing - or at least - trying hard to TDD over the last couple of years and it has been a grind. 

I've read TDD by example by Kent Beck about 10 times and watched plenty of videos and read blogs and - in spite of being told that practice makes perfect - did not practice anywhere near enough - I just didn't have the opportunities in work time alone to do this (but that's another story - and that is in spite of being a TDD first org).

Taking my lack of practice by the horns after reading Sandro Mancuso's book -  The Software Craftsman) this coupled with Roy Oshrove's string calculator kata,  Growing Object Oriented Software Guided by Tests  (GOOS) by Steve Freeman and Nat Pryce, Mocks aren't stubs  by Fowler and Sandi Metz's take on what and what not to test
a few things changed and things have, seemingly, finally clicked.

TDD Bigger than the sum of its parts

Using the tools I mentioned above will get you writing unit test's pretty quickly but not TDD'ing.  Using the tools and TDD'ing will get you TDD'ing but most likely in haphazard manner.

Firstly, TDD'ing at all requires confidence in the tools, mentioned above, and not worrying or having to think about how to use  them - its not that they are extremely complicated to use, but things can get bloody confusing.  Using Moq, we can create Mock objects which are Stubs or we can use Mock objects to verify behavior with them.   Of course, this can be done without Moq, but it is great at creating Mocks and Stubs, quickly.

Using a good refactoring tool really helps speed up things too, but - again - to use one fairly well there is a bit of a big curve learning useful shortcuts but then once this has been done learning when to pull objects into existence or when they should be Mocked or Stubbed needs to be learnt.  

So we have the tools and we need to learn how to use them and when to use them and why to use them.  This is a lot to learn and then apply. However, when all of this starts to fall into place, wonderful things start to happen and the productive ebb and flow of  TDD, which before, was likely staccato comes to the fore. 

With this, and something  I forgot to mention, pairing with people when TDD'ing loses the scare factor, worrying about how to use the tools is a bit of a disadvantage and can disrupt the flow of pairing, it can make you feel pretty stupid if someone is already pretty comfortable, even if you are far from it.  Nobody likes to appear like a novice - not knowing how to find a gear or the accelerator if comparing to driving.   

So the upshot of having the requisites in place is that shaping applications, driving the design by pulling in collaborators (either by Mocking or Stubbing them) with a quick refactoring short cut can all be done under the TDD cycle with confidence and very nearly without a thought as to how you are using the tools to get to your end goal - you are just using them as a means to an end. This is a really fulfilling feeling. 

Summary

I am writing about my experiences with TDD as I've struggled with it, and watching people around me get it and not having that same Eureka moment (or moments) has added to my anxiety, stress and worry that I would never be able to TDD effectively.  But I think I am just about there, I always like TDD'ing,  I now think that I can do it pretty well.  And on this note if other people are having trouble with it, stick with it, practice a bit more - read a bit more too, make sure you are reading the right things,  some things may likely only make sense when you have some of the other fundamentals boxed off.

Happy TDD'ing.



Monday, 25 April 2016

Newline character is 1 character

I will never forget the above

When using substring and you can't work out why "\n" is not 2 characters refer back to this title.  Does it make me look stupid, Yes, hopefully I will work this out sooner next time.  To be fair to myself its like and and. 

That is all. 


Tuesday, 19 April 2016

A Beautiful Solution?

^^Perhaps^^

For various reasons me and my team hadn't been spending a lot of time at the coal face, coding.  From spikes, POCs and analysis to release activity -  coding opportunities have been few and far between. Thankfully this changed over the last couple of weeks.

As part of the Payments Platform we have been given the opportunity to chip away at some of the Legacy processes to provide more flexibility in our Payments offering.  Part of this process see's us hoping to consolidate some existing logic to manage the integration with a 3rd party Payment Solution Provider more flexibly.  Specifically around handling particular types of payment which are handled differently based on certain criteria such as a customers Billing country, what currency they are paying with and the card type they are using.  

Stuck

In essence the work we are carrying out is a redesign and re-factor of some logic which is spread out in a number of places.  Bringing it all in, has been both interesting and frustrating.

Maybe its because of the lack of practice but I couldn't quite nail a solution (to a part of the problem) until about a week after starting the work.  

I'd started out test first, using TDD to drive out an initial solution, which I was sort of happy with, and then re-factored - went a bit mad - duplicated loads of tests and then had lost the essence of a simple solution.  As I wasn't happy, I made a V2 of my work, left all of - what had become sociable tests - in place and started again.   I ended up with real clarity, no over-engineering and a solution  I am really happy with.

The essence of the problem

When filtering data there are a number of options available.  


Typically if the data is stored in SQL (perhaps relationally - but not always) , querying and filtering can easily be performed in a stored procedure (or data layer). Certainly at the cost of having business logic in the DB (usually a pretty poor choice), amongst other trade-offs.   

Conversely a denormalised bit of data stored in a document store or - again  - in SQL can be retrieved and then worked on in memory and up in the vernacular of the UL of a domain  using LINQ and , giving a nice opportunity to easily test behaviour and nail requirements around a business critical process.

The existing solution, we are augmenting, used some pretty meaty  LINQ queries.  Now, arguably for some, this may suffice and do the job, but due to the number of things needed to filter on and various conditions, some pretty "all encompassing" LINQ 'WHERE' statements were being used with comments.  Coupled with there were no tests for some of the behaviour, some of it seemed coincidental.

The comments alone are enough to make you question the clarity of a design and are well documented as a marker of a code smell in a lot of situations (but not all of course).

Me and my colleagues also noticed there were lots of null checks thrown in too.  We all thought - we can do better than this!  I lobbied for this task and was certain I'd knock something out quickly.  I struggled!

My first approach saw me using the decorator pattern or at least - a bastardised version of it.  The basic concept is illustrated below.  In fact the approach can be visualised as moving some data through a pipeline of filters, each filter being stateless and all working toward refining some data to a reduced set. I talked about this kind of approach when mentioning folding in an earlier post. This is a functional programming technique.





In code my first attempt was facilitated by the code below: 

In the snippet above I have an interface that takes the seed collection (the original query) in the first argument - only the first filter in the chain would work on this.

The other arguments are values which are needed to drive the filtering, all of this information has to go down so that each filter can use what it needs. This could be simplified by applying Extract/Introduce Parameter Object to the parameters. 

Subsequent filters just work on the results from previous filters who have their Execute method invoked within a filters Execute method.  The other filter being a dependency of course.

Only one filter implementation is shown here, the currency filter.  Its Execute method dictated by the interface, applies any filtering and returns the result for the next filter to do its thing (if there is one configured).


Each filter is stateless, in the end after applying a number of filters, an input set is reduced to just one matching value based on various criteria.

Setting up a "chain" of filters would look like:


Now, this worked but after review by my colleague, the fact I'd gone out of my way to encapsulate the filters as separate classes, kind of indicated that the filters should be usable by him as standalone autonomous bits of logic and this is exactly what he wanted to do as he had some similar filtering which only required some of the filters I'd made. Fair enough.

With a few tweaks this would work. The problem was that with the dependency of the filter up in the constructor without some kind of null object pattern or specific handling to check for nulls or handle setting up the chain of filters, things would break if composed in the wrong way.  That is, for example, if he wanted to use a filter but didn't want to chain a specific child filter.

As well as a set of scenario driven tests for the domain service, I'd written tests for each filter and in so doing  duplicated lots of test.  I'd not quite implemented what I desired.

I started looking at Haskell list comprehensions and thinking about a functional way to tackle the problem and  came across a post  on S/O which was exactly what I was looking for.  In essence I could use a left-fold and have my filters called one after another with a LINQ extension method called Aggregate which - like my solution - takes a seed but - unlike my solution - handles 'accumulating' or keeping track of the result of applying filters without the need to hand-crank any other classes. Composition looks like below. A list of methods are used to impose filtering with the heavy lifting done by a bit of LINQ as we can see below in the next snippet.

Below, the  Aggregate function is called which takes a seed (our initial data to be filtered - number 1 in the diagram above), a lambda (or anonymous function) representing an accumulator method, which matches the signature of the methods in the filters list (numbers 2 and 3 in the diagram above).



It then applies each of the methods - composed above - in turn. Running each filter and passing its result, via the accumulator to the next filter, eventually pushing out a much more refined result adhering to the nip and tuck that the filters imposed along the way. 

Summary

I really like the elegance of this solution.  Yes it is simple (you may wonder why so many lines were devoted to something which may not have even been given a second glance on what you are working on)  and there are yet simpler ways to tackle our problem, the filters could just be looped over for example (or even revert to clunkier LINQ queries).  It just struck me as, erm, nice.  You may well disagree.  If so, please let me know  in the meantime I might do a V3 and just filter down in the persistence layer :)