Messaging and transient issues.

Messaging and transient issues.


A small post but something worth considering. 

 Recently when discussing a situation where a MongoDB replica set was in the process of failing over, concern was raised about writing data - whilst a new primary was being elected. 

This is going to be a transient issue and issues similar to it - such as temporary server outages and routing  are too .  

 They are going to take a little time to resolve but should resolve fairly quickly. In the meantime, there are a few solutions available whilst this transient issue sorts itself out:  

  1. Do nothing.  In this case give up and find another job  you lazy hacker.
  2. Let the process fall over, report  the failure to users and let them try again via a button click. A users experience might be sullied - in the opinion of some -  but it still could be a reasonable way to recover (this depends on what stakeholders/business think really).  This might not be reasonable for important information which must stored and can not be optionally retried. 
  3. Employ a retry mechanism, simply loop a few (reasonable times) until we get success, or employ an exponential back off to give reasonable time for recovery.  I have done the former using, as someone I know put it, some funky "AOP" shit.  However, I wouldn't recommend doing the AOP stuff for transaction management or bounded retries, because eventual consistency still may never occur and AOP, certainly in a lot of the frameworks I have used, is complete magic. I did have trouble explaining the concepts to some of the less experienced and even seasoned members in my team what that "Transaction" attribute was above my DAL method, and how this was setup on the IOC container and  how code was executed pre and post the method in an execution pipeline .
  4. Use a durable messaging framework like NServiceBus.  The command or event (message) which was sent will fail and if second level retries are enabled will be retried a specified number of times. If unsuccessful within the specified number of retries (which have exponential back off) The message will be placed on an error queue,  relevant administrators will be notified or at least be reporting on this stuff.  The exception and or problem will be noted and hopefully fixed and then the message in the error queue will be replayed bringing everything back into a consistent state.  And all of this with the user completely unaware that four data centers where nuked.

That is all. 

Comments

  1. Blade25 November 2014 at 04:16

    Blade only communicates by pager, budget Nokia handset or telephone box. No need for this messaging rubbish. You can find out more about this here: http://goo.gl/Cre4fE

    ReplyDelete
  2. col27 November 2014 at 12:53

    This comment has been removed by the author.

    ReplyDelete
  3. col27 November 2014 at 12:54

    Hi Blade, as always I am glad for your insightful contribution. I must admit that messaging is not always the answer sometimes smoke signals or morse code can be just as effective too, its all about the context......

    ReplyDelete

Post a Comment

Popular posts from this blog

Book Review: Test Driven Development By Example (Kent Beck)

GitHub I have coded the examples and both are available in GitHub.   Money  and xUnit  Delayed... I am recalling this after a few weeks finishing the book (I should've done this immediately so my memory is hazy, but the code examples should attest to the fact I have read it, if my descriptions of the text are a little ambiguous)  I have read this book on a number of occasions previously, in reference form and once all the way through,  but hadn't worked through the examples in  a code editor.  I think, for real value, you have to  code the examples, after a while it does feel like you are actually pairing, like others who have reviewed this though, I think the humour leaves a bit to be desired but its not too distracting! I have been TDDing for a while but this was a good refresher and reminded me of the maxims and techniques "fake it till you make it",  "obvious implementation" and "triangulation". There are also some nice "red" and &quo
Read more

SSDT March 2014 Static Code Analysis Extensibility

Static Code Paralysis. Coding standards (and more latterly styles) in any organisation of any size are liable to incite religious wars and makes zealots out of even the most considered or apathetic of us.   Inconsistency and subjectivity,  lack of enforcement and heavy handedness all contribute to the contempt often leveled at them.   Sod it. I say sod all that, if you or your company aren't using static analysis tools  then you are missing out on improved code quality and productivity (borne out from a consistent approach) and reducing the most banal and mind numbing conversations and sometimes pissing contests associated with discussing the minutiae of standards.   As an aside, having a few years under my belt  in the industry  I've found that people who "really" moan most about this are usually rabble-rousing morale sappers who like to moan about everything, so it stops them doing this and maybe might lead to them getting on with work instead! Makes thin
Read more

MongoDB

Mongolicious MongoDB training, provided by MongoDB themselves, was up for grabs recently, so I put my name in the hat to get a keener insight into MongoDB and NoSQL databases, having used them only a few occasions. The training was attended by a near 50/50 split of devs and DBAs and this led to some interesting debates and reactions.   Like lots of the stuff I've been introduced to/have started looking at recently (I am late to nearly every technology party there is) they (Mongo and NoSQL) have been around for a while and have become established as a viable alternative persistence solution to the likes of MSSQL.  The main features: Its schema less Document oriented/No support for JOINs  Querying performed using JS/JSON Indexing Replication/Redundancy/High availability Scaling out via Sharding  Aggregation (unfortunately we ran out of time so this is not covered here) Authorisation (again ran out of time on this one)  Its schema less You'l
Read more