Tuesday, 31 October 2017

Book Study; 99 Bottles of OOP by Sandi Metz

Review to follow, just finishing it.  Basically its good. Perhaps that is enough of a review.

Sunday, 10 September 2017

End to End testing using Reactive Extensions (RX)

A change of tack is needed with my posts.  The limited people who have read my blog have been too kind to say that I talk lots of unnecessary shit.  I read back over my posts and I am musing like some kind of pseudo-intellectual who is smiling smugly and patting  himself on the back over some kind of  new insight into stuff which is really just a regurgitation. So I should stop it and keep the musings to a minimum from here on in. I've rewrittten this post to try and give something valuable that can be taken away and used.

Requisite Knowledge


General C#, Generics, Messaging Patterns, NServiceBus, Linq, Understanding of asynchrony, Task<T>, Unit testing.


Reactive Extensions 

I decided to use Rx to help end to end test a payment method service.  I wont go into a lot of detail about Rx, and instead refer you to an absolutley fantastic reference, which can be found at reactivex.io. The visualisations -  especially -  help bring to life what Rx does and can help stimulate ideas about how it can be used for you.    Rx.NET is what the code examples (in C#), here, are programmed against. 

The problem

I need to be able to test that my payment method service (service) is working by ensuring that, given various input,  it is producing the correct output (messages) in a timely fashion. 

I need a way to be able to ask the following questions  "Given xyz input, has a payment accepted event occured?", and if so "Does it relate to this payment reference and has it arrived within10 seconds?".  If so, great my service is working, if not, then it isn't.  

The service to be tested comprises a front end (RESTful like API) and a backend server (producing messages).  From a testing perspective, appropriate input should be supplied via the API and then various messages from the server should occur as a result.  This is going to need end to end testing.   (Edit 1:  So I am saying we need end to end testing IF we want to exercise a fully working service with all its real world external dependencies AND this is necessary as the API and server reside in different processes.  We could  still component test the API and backend server - each individually  - by swapping out external dependencies  but this wouldn't provide us an integrated test .  There are advocates and detractors  of  end to end testing along with its tradeoffs. 
(Edit 2:  I am choosing to state end to end testing over  "external" component testing whereby external colloborators (I/O, Http) are included in the mix - because the Microservice which logically comprises an API, and other backend messages servers are involved - bit of a minefield!)

My experience is that, if not insanely difficult or expensive  (in time, complexity, and maintenance) it is worth it for the confidence (edit: and coverage) an end to end test can bring.


The services in our platform use NServiceBus (NSB) to send and receive what are usually domain events (edit: synonomous with messages), we can leverage this to get a handle on these messages and inspect them or lack of them.

To be able to ask the questions above I need to be able to observe (listen) on a channel for the messages  (an observable sequence) and then run a test asking relevant questions. 

In this instance the channel will be an event handler sitting atop an associated "bus" (set up not shown here)  - an abstraction NSB provides to represent the medium on which messages are passed. 


The handler:

public class GenericMessageHandler : IHandleMessages<object>
    {
        public void Handle(object message)
        {
            Observer.OnNotify(message);
        }
    }


The handlers job is to listen to messages which arrive at the handler and notify the Observer. The Observer is a facade for what is known in Rx.NET as a ReplaySubject, which is a special type of Observer (unfortunately my facade reuses/overloads the term Observer), the Observer has or will be setup with a bunch of subscribers which  will be the tests (or a way to set the results of the tests**) to run for a certain type of message.

The ReplaySubject allows buffering so that if a "would be" subscriber (which we will be using to hook up tests)  is not yet subscribed, and messages have started occuring on the channel (that is: arriving at the handler and being pushed into the ReplaySubject buffer)  then when the subscriber is subscribed all messages which have happened will be replayed to the subscriber.  

This special property (subject to limited buffering) means its not  important that a subscriber is hooked up straight away, as long as the Observer (the ReplaySubject) is getting messages and, so, in this way the chances of missing something on the channel are reduced. 

It is worth nothing that  NSB uses publish/subscribe, so some setup has to be provided to ensure that messages of certain types are subscribed to and therefore will arrive at the handler. This code is omitted here. ( available in git shortly).

The Observer facade has 2 users:
  • the channel
  • unit tests
The channel pushes in messages to the Observer
The unit tests will be set up to "ask the questions" and provide the tests against those messages and these tests will be "attached/subscribed" to the Observer.




The Observer looks like this:


public class Observer
    {
        private static readonly ReplaySubject<object> MessageObservable = new ReplaySubject<object>();

        public static void OnNotify(object message)
        {
            MessageObservable.OnNext(message);
        }

        public static Task<T> SubscribeToMessageTest<T>(Func<T, bool> test, string failureMessage, TimeSpan timeout) where T : class
        {
            var tcs = new TaskCompletionSource<T>();

            MessageObservable
               .Timeout(timeout)
               .FirstAsync(message => RunTest(message, test))
               .Subscribe(
                   message => OnSuccess(tcs, message),
                   exception => OnFailure(exception, tcs, failureMessage));

            return tcs.Task;
        }

        private static bool RunTest<T>(object message, Func<T, bool> messageTest) where T : class
        {
            var convertedMessage = ConvertMessage<T>(message);
            if (convertedMessage != null)
            {
                return messageTest(convertedMessage);
            }
           
            return false;
        }

        private static T ConvertMessage<T>(object message) where T : class
        {
            var messageAsType = message as T;
            return messageAsType;
        }

        private static void OnSuccess<T>(TaskCompletionSource<T> tcs, object message) where T : class
        {
            tcs.SetResult(ConvertMessage<T>(message));
        }

        private static void OnFailure<T>(Exception ex, TaskCompletionSource<T> tcs, string failureMessage)
        {
            Console.WriteLine(failureMessage);
            var messsageException = new PublishTestException(failureMessage, ex);
            tcs.SetException(messsageException);
        }
    }



The most interesting method is:
Task<T> SubscribeToMessageTest<T>(Func<T,bool> test, string failureMessage, TimeSpan timeout)


At a highlevel then: a test is passed in which has previously been setup to be  executed against a specific type of message (when a message occurs on the channel and appears in what is known as an observable sequence) 
A TaskCompletionSource is created (tcs) whose  result is going to be set with a message if the associated test is successful or, an exception if it takes longer than a certain time.  A test is always unsuccessful if it can't be satisfied within the specified timeout period. 
The task from the TaskCompletionSource is then returned. 


The parameter "test" is a Func<T, bool>  (a generic predicate) to be hooked up to test elements in an observable sequence (the messags from the channel), to  MessageObservable (which is a ReplaySubject<object>. 

This is done with the lines:
MessageObservable
.Timeout(timeout)
.FirstAsync(message => RunTest(message, test))

Firstly,  via Timeout,  a timeout is set (if this is exceeded an exception is thrown)
Secondly, via FirstAsync, return ONLY the elements from the observable sequence  (the messages from the channel) which pass the test passed to RunTest.  The RunTest method does some conversion on the message, and runs the test. 


The next 3 lines:
.Subscribe(
  message => OnSuccess(tcs, message),
  exception => OnFailure(exception, tcs, failureMessage));


Subscribes the message  to a couple of methods which set the state of  the TaskCompletionSource (tcs) declared at the top of the method. 
OnSuccess takes the message itself and puts it into the result of the tcs.  
OnFailure occurs when a timeout occurs and the exception is put on to the exception on tcs. 
Not forgetting that the task associated with tcs was returned earlier and so consumers of the code would be notified as and when this occurs.


The SubscribeToMessageTest method is facaded with a PublishTest<T>:

    public class PublishTest<T> where T : class
    {
        private Task<T> task;

        /// <summary>
        /// Test a message that satisfies a predicate, before a timeout occurs.
        /// </summary>
        /// <typeparam name="T">The type of the message to test.</typeparam>
        /// <param name="predicate">A predicate with tests you need to determine the correct message has been published.</param>
        /// <param name="failureMessage">A message to aid debugging the test failure.</param>
        /// <param name="timeout">Timespan indicating how long to wait for a message to be received.</param>
        /// <returns>Task which should be awaited or Result inspected.</returns>
        public PublishTest(Func<T, bool> predicate, string failureMessage, TimeSpan timeout)
        {
            this.task = Observer.SubscribeToMessageTest(predicate, failureMessage, timeout);
        }

        /// <summary>
        /// Block whilst waiting for test to verify. Use where async/await is not supported.
        /// For example, SpecFlow does not currently support async/await.
        /// </summary>
        /// <returns>Message associated with successful test.</returns>
        public T Verify()
        {
            return this.task.Result;
        }

        /// <summary>
        /// Verify the test at a later point in time by calling .Result or using async/await on the Task returned.
        /// </summary>
        /// <returns>Task with message assocated with successful test.</returns>
        public Task<T> VerifyAsync()
        {
            return this.task;
        }
    }




A test can be written Which is then used like so in a test:

    [Then(@"a PayPalPaymentIntentAcceptedInternalV4 is published")]
        public void ThenAPayPalPaymentAcceptedIsPublished()
        {
            new PublishTest<PayPalPaymentIntentAcceptedInternalV4>(
                message => message.PaymentReference == this.context.PaymentReference,
                "No PayPalPaymentIntentAcceptedInternalV4 message was published matching the specified test.",
                this.context.MessageVerifyWaitTime).Verify();
        }



The lambda - which provides the actual test -  (message => message.PaymentReference...) and the type "PayPalPaymentIntentAcceptedV4" are used in tandem to qualify that firstly a test for this message type should take place and that the message is associated with input supplied earlier at the API, which would have the same payment reference.  The lambda/test is  ultimately what is passed to RunTest in the Observer class (which was listed earlier). 

**As I am writing this I can see some improvements I would like to make to this code and will add an edit (also adding proper code snippets).  I have left out some of the set up code around  NServiceBus  and the handler and setup to subscribe to messages.  Also the other parts of the tests are omitted for brevity.  ( To be included in git). 
 


















Wednesday, 12 April 2017

Micro Post: Big Glass of Sand

Monolithic architecture is classified as a Big Ball of Mud
Microservices architecture is therefore a Big Glass of Sand

I said it first , right here.

Tuesday, 7 March 2017

Testing, Testing, 1.2.3.4.5.6..ad infinitum

Before testing...

I've now been in development/engineering/coding/hacking or whatever you want to call it for more than 10 years.  Testing has always been important, but it  - in my line of work - is now at the forefront of everything I do.   

For the business this is great, every line of code, component and group of components is covered by some set of tests. Be it unit, integration, component...the overloaded terms go on.   This means that a set of requirements -  to some degree  - have been determined to have been met by these tests and the business is getting what it wants (all manner of development and agile practices not withstanding - they all get us to the same place)

It struck me today though that earlier in my career, when testing was still important but lines were  - perhaps - a little more blurred and things were a bit more hit and miss, things were less clinical, less contrived and quite frankly less boring.  This is obviously not a good model for delivering quality software everytime but sometimes I yearn to literally hack out some code,  build it  and just experiment when things aren't working quite right.  Yes this can be done guided by tests and within the confines of the TDD cycle but, for me, having not come from a TDD beginning, feels a little stifiling if I was developing a hobby project (not something I was contributing to on GIT) I, dare I say it, would not necessarily use TDD - but make no mistake, at this time, developing quality software for my employers I would not develop without TDD, it gives me confidence I never had in the past. Boring yes, Confidence inspiring yes.

This kind of work is left now for spiking  or hobby projects, but it feels that, and although this term is bandied around,  software engineering  really is coming of age. 

Engineering, Not art/craft 

I often hear that software development is a craft or an art and that it is creative - this may well be the case for UI related disciplines but business software is the geography teacher of practices, boring and wearing tweed.    Once all the requirements are down and how these are going to be met are sorted  - that's it. Software is being driven out in a very prescriptive manner.  The biggest debates that happen in my team are not religious wars about coding standards - we have static code analysis, not religious wars about performance - we have resources to burn,  not religious wars about  general designs issues we have SOLID (and other guiding design guidelines), but no - we have wars about how we are going to test something and where.  ATDD brings this war right to the inception of a story/feature (sorry I often get these the wrong way around) and then we have a list of work todo and a checklist to say when its done.   This is all very organised and delivers value,  but I'd never realised how boring this actually is when you break it down. 

Summary

This might sound negative,  and boring is generally not considered good, but it is a matter or perspective.  Boring is good, as  I said before, for the business.   Now, I enjoy going to work and doing a good job, but I am wondering what it is I actually like doing anymore, maybe I have become boring and that's what I now like.  That's what my Mrs thinks anyway...

 


 

Tuesday, 6 December 2016

1..2…3…Freeze….Peak



I haven’t written anything for ages as  I’ve been the busiest I have ever been at my current employer – a large E-Commerce Website -  as a result of preparation (planned and unplanned) for its largest ever “Peak” weekend, my company’s name for the Black Friday weekend.    

Sorry this is a long read so grab a coffee or skip to the Summary, or don't read it!

I’ve started writing this on the train coming back from London where I was required to provide extra-support for the Payments Platform (Domain)  - where I ply my trade – a group of Microservices and Legacy components supporting Payment processing for the Website and sitting very closely to the Orders Domain. 

I expected to be bleary eyed and wired to the hilt, full of coffee and adrenaline as me and my colleagues worked like mad to pull out all the stops to make our giant incredible machine work properly. But no, as our CIO put it, everything just worked (well 99% of the time – more on this later). 


The freeze part of the title refers to the fact that, in the weeks up to Peak, as we tried to ensure stability of our software, only the most critical software releases were allowed out.  I , like many of my peers, do not think this is great.  This is costly in many ways; not least, all of the feature development still going on actively within the company is shelved and starts to gather dust and not being released means the software is accruing debt on many fronts.  However, I can’t be too hasty in judgement, this way of working comes from something along the way biting my company really hard during this crucial trading period, this mentality is understandable if not immediately completely excusable.  Hopefully, this will get better as the company gains more maturity, confidence and control over its systems.

Application Insights

As part of the preparations it became clear that our services did not have sufficient monitoring capabilities.   Services were performance tested in quasi-production like environments and so we had some idea about how they might perform, but only an idea.    So we had some expectations about performance but very little in the way of helping us see operational health .  Measuring and keeping  track both would have been impossible without telemetry and logging of some sort.    

We already had some monitoring available via dashboards powered by Grafana. A number of dashboards exist showing Order flow, and, as a result, part of an Orders path through Payments, but only at a high level.  With various counters representing the number of Orders at a particular status, e.g. arriving from the Website, currently being billed, and shipped to the warehouse, this is used to help try to generalise the performance and health of backend systems as Orders wing their way through the Website, through Order and Payment Processing and much more and then, eventually, out to the Warehouse(s). 

If problems start occurring, by some thresholds being exceeded or not met,  represented by numbers and traffic light boxes, focussed investigations into issues can take place in a particular area.   The problem with this is that in between the different statuses there are a vast swathe of software services in locations all over the world involved: including New-World Microservices, Legacy systems, stuff in the sky, Messaging, RESTful APIs, document DBs,  SQL DBs, No-Sql Dbs, Edge caching, and hardware on premise.  Application support have lots of fun trying to diagnose errors and, where they are at a loss ,they hand over to software development teams/developers who mostly run around in blind panic at the same time.

More recently with the release of a updated versions of the Website (which is multi-platform/device) New Relic was enlisted in helping see how the various APIs consumed by the Website were behaving and this has helped to start to build a more fine grained picture of the operational behaviour of APIs   ( and by association the Websites) where various HTTP statuses can be obtained and detailed analysis about response times, payload sizes, customer locations, can be gleaned, but that’s about it in terms of insight into what an API is doing. There is yet another monitoring tool called Kibana (sitting atop of the ELK stack) which is used for some of our Azure based services, largely this gives transparency to telemetry just like New Relic. My money’s on New Relic in this area though.

So, the introduction of performance counters, custom and otherwise were conveniently placed at the door of our APIs, on the 3rd party calls within them and in and around the messaging infrastructure used in the backend processing for our services.  Relevant logging was added too, to ensure that in the event of failure or exceptional behaviour we had some trace of what was going on.  So, having all of this information, but not at your fingertips is a bit of a nightmare. 

Consolidating this all into one single place is achieved with dramatic effect using Application Insights hosted in Azure.    Simply adding appropriate libraries in the service and hooking it up to an App Insights Resource Group in Azure and boom you are presented with a bewildering array of options to present you with your telemetry.    Coupled with this you can also leverage Analytics to perform queries on all of the data collected and herein lies a problem.  Making sense of all of this information is hard.  Certainly, there are some headline acts easily noticeable but more subtle problems like a semi dodgy 3rd party call, or reasonably flaky database writes can be eked out but not after some considered tweaking with Queries etc.    Also without reasonable SLAs from the business in most places and the fact that a message queueing system is used, what alerts should be used and when is a question we are still asking ourselves.

Long nights and Instability

So getting to a good place a lot of pain needs to be had.  And boy did we experience our fair share up to peak.  With SAN migrations causing mayhem with our messaging,  Windows Cluster failovers  misbehaving and deleted Orders in Azure there was plenty of time to practice and gain experience in trying to solve problems  quickly and still meet “customer promise”  a term used to describe meeting cutoff for various delivery options, which is not taken lightly. 

Graph Watching First Blood Part III

So the culmination in all of the Black Friday prep is a series of graph watching sessions, with eyes trained on graphs rolling around in multi-tabbed browsers and lots of suggestions and conjecture about what this trend is saying and what that trend is saying.   I was personally involved in 4 of the 5 days of Peak support, remotely on Thursday, Black Friday some of Saturday  and on premise (by demand) on Cyber Monday.  Certainly, at the beginning of the event focus was very high and everyone was apprehensive about what might occur and what problems might need to be addressed.  Previous year’s Peak had seen all manner of catastrophes including site outages and various problems.


   
The biggest problem this year, then,  from a Payments perspective, was that a Legacy component on dealing with timeouts with a Payment Service Provider couldn’t reliably recover and compensate  from the timeouts and so Orders would be left in a state where we didn’t know whether they had been billed.  Simply retrying the Orders meant risking a cancellation occurring as the legacy component is pretty dumb, so a quick support tool was knocked up and like a scene from Lost, a button was pushed every so often to make sure these were processed properly. 

The highlight though is that everything, generally, worked.   On Cyber Monday, due to a failure with a voucher service Orders dropped off a cliff.  Fixing the problem and spreading the word to our persistent and patient customers meant that between 9-10pm an average of 55 orders as second were being processed and all of this would land at Payments door step. The system coped admirably and a backlog of messages, of which there were up to about 40k at one point (spread across our system) were slowly but surely dealt with.  

Summary

The Payments Platform have now got a lot a data regarding capabilities during spiky load periods, a greater operational understanding and has breathed a big sigh of relief having been able to support Black Friday.   Thing is, its going to be even bigger next year and we are just about to start preparations in updating our Platform to be even more resilient, scalable and available.  Looking forward to it.  We will have a lot more control of our systems come this time next year and a maturity and experience to put what we have learnt to good use.  

Up Next....Reactive Extensions or maybe even HAL (Hypermedia Application Language)




Sunday, 5 June 2016

Learning to TDD is like learning to drive a car

Its true (well sort of)

When I was learning to drive I was pre-occupied with where to put the gear stick, where it actually was, when to use the clutch and so on and so forth and being flustered all the time.

But, quite quickly, with continued practice and with growing familiarity and confidence with a car, I could think about where I wanted to go instead of every detail of what I needed to do, to get there, and now I drive like Lewis Hamilton.

Before  I could start to TDD fairly well (In my .NET C# world) and be able to decide what works for me (for example choosing when to use classic and mockist TDD) I had to become comfortable with some pretty basic things including a good refactoring tool - ReSharper, a good test runner and Unit testing framework like NUnit and a good test isolation (mocking) library, like Moq (all of these tools have their proponents and detractors)  but this is what I use for the majority of the TDD I do at the moment, this could change, but the principles won't.

I've been TDD'ing - or at least - trying hard to TDD over the last couple of years and it has been a grind. 

I've read TDD by example by Kent Beck about 10 times (but never to the end) and watched plenty of videos and read blogs and - in spite of being told that practice makes perfect - have not practiced anywhere near enough - I just dont't have the opportunities in work time alone to do this (but that's another story - and that is in spite of being a TDD first organisation).

Taking my lack of practice by the horns after reading Sandro Mancuso's book -  The Software Craftsman this coupled with Roy Oshrove's string calculator kata (and many others like it),  Growing Object Oriented Software Guided by Tests  (GOOS) by Steve Freeman and Nat Pryce, Mocks aren't stubs  by Fowler and Sandi Metz's take on what and what not to test 
a few things changed and things have, seemingly, finally clicked.
(Edit: 31/10/17  I  am reviewin Sandi Metz's TDD eBook, 99 Bottles of OOP and the first impressions are really good) 



TDD Bigger than the sum of its parts

Using the tools I mentioned above will get you writing unit test's pretty quickly but not TDD'ing.  Using the tools and TDD'ing will get you TDD'ing but most likely in haphazard manner.

Firstly, TDD'ing at all requires confidence in the tools, mentioned above, and not worrying or having to think about how to use  them - its not that they are extremely complicated to use, but things can get bloody confusing.  Using Moq, we can create Mock objects which are Stubs or we can use Mock objects to verify behavior with them.   Of course, this can be done without Moq, but it is great at creating Mocks and Stubs, quickly.

(Edit 31/10/17 Perhaps I have overstated having to use these tools.  Using things like Resharper are really helpful, but not the be all and end all of course.  And of course you could write your own Stubs and Mocks -  I guess my point was that these things help speed up development somewhat and getting up to speed with them takes a while but also at my company I had to learn these at the same time. In retrospect  perhaps it would be better to not use these types of tools AT ALL - when starting out - to really get some of the fundamentals down, especially the really intricate part of slowing down TDD and being very thorough and organised and applying a regimented approach, which  I have come to appreciate more as time has gone on - and  having read more about - even just recently.)


(Edit 31/10/17 However,... ) Using a good refactoring tool really helps speed up things too, but - again - to use one fairly well there is a bit of a big curve learning useful shortcuts but then once this has been done learning when to pull objects into existence or when they should be Mocked or Stubbed needs to be learnt.  

So we have the tools and we need to learn how to use them and when to use them and why to use them.  This is a lot to learn and then apply. However, when all of this starts to fall into place, wonderful things start to happen and the productive ebb and flow of  TDD, which before, was likely staccato comes to the fore. 

With this, and something  I forgot to mention, pairing with people when TDD'ing loses the scare factor, worrying about how to use the tools is a bit of a disadvantage and can disrupt the flow of pairing, it can make you feel pretty stupid if someone is already pretty comfortable, even if you are far from it.  Nobody likes to appear like a novice - not knowing how to find a gear or the accelerator if comparing to driving.    
(Edit:  31/10/17  Perhaps knowing the tools would be less of a worry if you knew the intent and understood the implications of what people you are pairing with are doing, a tool can be picked up and put down but understanding is a much slower gained but sought after quality) 

So the upshot of having the requisites in place is that shaping applications, driving the design by pulling in collaborators (either by Mocking or Stubbing them) with a quick refactoring short cut can all be done under the TDD cycle with confidence and very nearly without a thought as to how you are using the tools to get to your end goal - you are just using them as a means to an end. This is a really fulfilling feeling. 

Summary

I am writing about my experiences with TDD as I've struggled with it, and watching people around me get it and not having that same Eureka moment (or moments) has added to my anxiety, stress and worry that I would never be able to TDD effectively.  But I think I am just about there, I always like TDD'ing,  I now think that I can do it pretty well.  And on this note if other people are having trouble with it, stick with it, practice a bit more - read a bit more too, make sure you are reading the right things,  some things may likely only make sense when you have some of the other fundamentals boxed off.

Happy TDD'ing.



Monday, 25 April 2016

Newline character is 1 character

I will never forget the above

When using substring and you can't work out why "\n" is not 2 characters refer back to this title.  Does it make me look stupid, Yes, hopefully I will work this out sooner next time.  To be fair to myself its like and and. 

That is all.