tag:blogger.com,1999:blog-53750251315412100122024-02-06T22:18:48.098-08:00We Are DevsExperiences with software development and other musings. colhttp://www.blogger.com/profile/11580119772205731598noreply@blogger.comBlogger25125tag:blogger.com,1999:blog-5375025131541210012.post-70831002702903816182021-12-15T14:57:00.005-08:002022-02-04T09:18:50.855-08:00Review: Logical and Critical Thinking - Future Learn <h2 style="text-align: left;">Logical and Critical Thinking</h2><h3 style="text-align: left;">Looking for a bit of clarity</h3><p>I have (in my own mind) been struggling with both giving my own arguments and interpreting arguments of others - mainly in the workplace - but also elsewhere. </p><p>I undertook this free course:<a href=" https://www.futurelearn.com/courses/logical-and-critical-thinking"> Logical and Critical thinking</a> created - in part- by the University of Auckland.</p><p>My interest was piqued because I am certain that good logical and critical thinking can help drive out more clear solutions and help with more clear testing and coding and generally participate in debates better (in relation to work) and that maybe it could help in other day to day aspects of reasoning. </p><p>I have since read that it appears software engineering can actually foster better logical and critical thinking due to the need to tackle problems of different types and I can see that testing - in effect asserting truths and non-truths when validating behaviour - can also help, however I still think that a formal understanding could help hone these skills further. </p><p>Over 8 "weeks" it takes you over the component parts of an argument, the main types and how to formulate, deconstruct and analyse arguments.</p><p>I'm under no illusion that this is Philosophy for dummies but I found it really thought provoking and far more engaging than I had anticipated.</p><p><br /></p><h3 style="text-align: left;">Great examples</h3><div>The 2 course leaders provide a blow by blow account of some of the following:</div><div><ul style="text-align: left;"><li>Statements</li><li>Fallacious arguments and common fallacies - including a worksheet on the 3 most common, this was an eye opener (more on this in a minute</li><li>Principle of charity</li><li>Standard Form (Premises and Conclusions)</li><li>Non-Deductive and Deductive arguments</li><li>Counter examples</li><li>Determining logically sound and cogent arguments</li><li>Scientific Method</li><li>Moral arguments</li><li>Case study to apply all of the learning from above </li></ul></div><p>It would be pointless to repeat or plagiarise everything I learnt - I've included a few standout takeaways below - but what I can say is that, if anything, after the course and based on the principles learnt above, I think I can articulate my arguments - to help make people believe what I am saying - more effectively. Likewise I am tuning into the validity of peoples' arguments better and this will help me discount or take on board what I am being asked to believe/buy into more effectively. I think the latter is extremely useful when looking for reasons to - or not to do - take a particular approach(es) when weighing up technical solutions (not mutually exclusively to other empirical aids but in addition to) </p><h3 style="text-align: left;">A few examples of applicability to a developers toolkit and software engineering and the engineering process</h3><div>Some of the things I learnt I started applying immediately, I started to see logical fallacies rear their head and came to realise I have been arguing terribly for many years but also listening to terrible arguments from others! We are all at it (well from a logically valid argument point of view anyway) </div><h4 style="text-align: left;">Standard Form</h4><div>During one of the "weeks" on the course me and my team encountered a number of issues in work with one of our components., I was only in for part of the bug hunt due to illness but I was keen to apply what I had learnt, before I had to take time off (thankyou Covid-19).</div><div> </div><div>I found it really effective to try and state the problems we were experiencing in standard form. Writing arguments in this way, allows you to really quickly invalidate a conclusion, as if one of the premises is invalid it can not logically support the conclusion and the argument is not sound.</div><div><br /></div><h4 style="text-align: left;">Affirming the consequent fallacy</h4><div>One thing to look out for when problem finding, and a logical fallacy, is affirming the consequent. It's a bit of shortcut we often take (like framing and cognitive bias). An example would be if you once said, correctly, "the endpoint is switched off, therefore it is not processing any messages" and later - upon seeing a problem with the endpoint incorrectly saying "the endpoint is not processing any messages therefore the endpoint is switched off", here the inference is that there is only 1 reason the endpoint is not processing any messages, but - of course - there can be many reasons; it could be on, but it's not being sent any messages or, perhaps, all the messages it is receiving are failing due to an error. I fall into this trap on occasion, especially on late night support calls, this would likely lead me to mentally closing off lines of enquiry mentally. </div><div>This kind of logical issue is covered in the fallacious arguments section earlier on in the course.</div><div><br /></div><h4 style="text-align: left;">Ad-hominem (at the person) fallacy</h4><div>Often, and I count myself guilty of this at times. A person's character is brought into play when discussing or evaluating their arguments. For whatever reason, a bad encounter, difficult character traits or some other reason comes to bear and something about them is used to attempt to weaken their argument. Obviously this is not truth seeking and it is something to be wary of. Seeing this logical fallacy called out has made me acutely aware of it and want to attempt to not to allow this to cloud my reasoning. </div><div><br /></div><h3 style="text-align: left;">Summary</h3><div>If not this course in particular I think that this is an area - certainly as a developer/engineer - worth exploring. It may help you get your point across or tune in more carefully to help assess good and bad arguments. The course leaders suggested that logical and critical thinking needs to be practiced to be maintained though. </div><div><br /></div><div>Coupled with this course I have started to read the <a href="https://www.amazon.co.uk/Thinkers-Toolkit-Powerful-Techniques-Problem-ebook/dp/B002PYFW4S">The Thinker's Toolkit - 14 powerful problem solving techniques</a> I will review this in another post Its related to the above, but I specifically wanted to understand some useful techniques when attempting to hone in on issues. </div><div><br /></div><div><br /></div><div><br /></div><div><br /></div><p><br /></p>colhttp://www.blogger.com/profile/11580119772205731598noreply@blogger.com0tag:blogger.com,1999:blog-5375025131541210012.post-46365756203868145102021-11-12T18:11:00.012-08:002021-12-15T15:09:56.308-08:00Continuous Workflows and ways of working<h2 style="text-align: left;">Kanban? Be careful...</h2><div><br /></div><h3 style="text-align: left;">Times they are a changin...</h3><div>You may want to skip straight to the summary for a few key observations, the rest of the article is an anecdotal walkthrough of transitioning from SCRUM to Kanban and some of the problems experienced.<br /><br /></div><div>My last post was in January this year regarding some TDD exercises I had undertaken, however I have actually been more involved in helping run our dev team for around 2 years in my first lead role for quite some years. Lead roles where I work are far from strictly technical and I have seen myself - not with the greatest expertise it has to be said - performing BA/PO, SA, and ADM(aka SCRUM master) type responsibilities with a little bit of time left to do the thing I enjoy most. I am writing this with a ruffled SCRUM master hat on. Unlike a famous beer I am probably the worlds worst SCRUM master (Well, maybe not that bad.. but I guess I am saying its not really an area I truly feel I can excel in - its more than just the SCRUM master certification you know...)</div><div><br /></div><div>Our team underwent a lot of change mid-pandemic when Agile Delivery Managers (basically SCRUM masters) were rudely turfed out of our organisation and our team personnel changed significantly at the same time to boot.</div><div><br /></div><div>Using the SCRUM framework was the well defined, and at one point, only way of helping organise work in our organisation and by extension our team. </div><div><br /></div><div>The organisational transition brought with it the chance to start thinking about how we, as a team, were working and assessing if we could (or should) change to a continuous workflow. </div><div><br /></div><div>This was not without some considerable debate and discussions about how to ease into a continuous workflow way of working. Some colleagues were <b>vehemently </b>opposed to moving to a new style of working suggesting that without exhaustive metrics and data to back up our need to change, we shouldn't, but also that we should make sure we were "doing" SCRUM properly first - these aren't entirely unreasonable asks, it would seem.<br /><br />My manager wasn't quite as strongly against change, and ultimately supported our transition, but stated that the preference would be to use data to prompt the need to change. <br /><br />My "opinion" was that even if we were a bit rash moving, setting ourselves up properly in our new system upon transition didn't mean we were being negligent or irrational (avant-garde trendsetter? I think not, more like hit and hope maybe!)</div><div><br /></div><div><div>Noting these opinions and taking a team vote and discussing within our team, each members maturity with Agile methodologies generally and with a view to potentially moving away from the SCRUM framework (with an absence of true empiricism (violations all over the shop eh?) we decided to continue to forge ahead with little supporting data , our lead and cycle times and what this meant were generally understood, but we were driven by slightly different expectations and wants (more in a minute on this). </div><div><br /></div><div>After studying both the SCRUM master course and KMP-1 course and bouyed by the chance to try Kanban, me and the team designed our new workflow and signboard, which represented the way we worked, and slowly but surely started to transition to using it and bumbling our way to using WIP and pre-conditions/gating to move between stages in our workflow and shouting out "pull from the left...." at stand up!</div></div><div><br /></div><div>The board on its own, has generally worked well. The signboard is clear with relevant columns, WIP which has been tweaked once or twice, swimlanes for classes of work and inspection points for entry to and from work columns. We took our time moving over from the sprint board to the sign board, but this is our main view point of the flow of work now. The specifics are kept confined to the sprint board (a level down effectively) . </div><div><br /></div><h3 style="text-align: left;">Agile without the ceremonies</h3><div>Some would hope so! Kanban, isn't actually about this. The nub of it is to visualise pinch points and be able to identify them and get things unpinched as quickly as possible. The nuances come in defining the policies correctly using math and little's law to help define WIP (5 will do + 1 thanks mate...) and quickly inspecting and adapting. Everything that is done in SCRUM but just more loosely defined</div><div><br /></div><div>A key part of Kanban teaching I undertook was what to do around pinch points. In effect retros and meetings often left to the end of a sprint with SCRUM are encouraged to take place as and when needed, there is no mandate for this but as there is no stipulation it makes sense to reflect on a problem when it arises to drive in improvements to help fix processes and get stuff moving.</div><div><br /></div><div>SCRUM is opinionated about stand ups, retros, reviews, backlog refinement, sprint goals etc, sprints and their timeboxing . Kanban seems less opinionated but shares standups and a signboard is about the only real stipulation I can remember.</div><div><br /></div><div>What I personally like about SCRUM is the stipulation of a fixed amount of time within which to do some work, the sprint. When it is done, review and reflection (via a retro) occurs and then the cycle starts again. Coming from a previous employer where we lived in chaos this was very welcome.</div><div><br /></div><div>Kanban (although we still use iterations due to our sign board technology) does seem to (or have the potential to) lead to a never ending cycle, coupled with - in our current spin on this - with fewer inspection points - the strain of this continuity can be felt within the team. </div><div><br /></div><div>Working within SCRUM didn't feel like this. At the end of each sprint - however insignificant the break is to some - I felt it keenly and the break allowed me to psychologically reset. Also a new sprint is a great time to ask questions about why stories are not closed off or a sprint goal has not been met. Not with a view to persecuting the team, but with a view to making things more effective, inspecting and adapting to overcome problems and waste. </div><div><br /></div><h3 style="text-align: left;">Is it a load of faff?</h3><div>At its simplest, basically getting some post it notes up on the board saying what you are doing now and next is all well and good, but the benefits of inspecting a process and impediments and issues must continue thoroughly as otherwise weariness can set in and the perception of not being able to perform correcting action for problems actually becomes manifest and apathy ( I am guilty of this myself) sets in.</div><div><br /></div><div>I have reflected on Agile generally , SCRUM and Kanban and pondered, what's the point? </div><div>Using SCRUM properly at the beginning of my current tenure, we had great focus and a sense of empowerment (bordering on pride) about our sprints. </div><div><br /></div><div>This has all but evaporated and it seems, we are now on one long slog with unclear goals and focus, and that is not to say we are not getting work done, we definitely are, but it feels like a grind. </div><div>Kanban and a continuous workflow is not to blame for this entirely, working from home, Teams meetings, the Pandemic and long running features which have crept in scope are almost definitely major contributors here too. </div><div><br /></div><div>I have observed that the perception of some of the ceremonies and activities within SCRUM are seen as wasteful and not valuable (and, for me, some of this is a fair observation in OUR organisation, some of the ceremonies have definitely not been that effective (we have big long multi team reviews which, for me, are astronomically wasteful - we are talking £10k's of pounds worth of wasteful when you consider the number and time of the people involved) , however I think if taken seriously and used as a guide there is value to having something to guide you, working out how to make various parts effective is something else again. </div><div><br /></div><div>I guess I am kind of thinking of a journey; going full circle from a pretty strict usage of the SCRUM framework to a very relaxed Kanban process losing some clarity and then considering going back to SCRUM to get control. </div><div><br /></div><h3 style="text-align: left;">Back to SCRUM?</h3><div>Writing this post today, I was wondering whether I would emerge with some ideas about the current state of play and how to improve it. Moving back to SCRUM and not being empirically guided may still be a bit negligent. As part of moving to Kanban I did actually create dashboards to capture key metrics (as mentioned above) I looked the other day at some, was shocked, and quickly closed the board (I am joking but regular inspection of this has gone by the wayside too) </div><div><br /></div><div>Story sizes, whilst I am thinking about the current state of play, should really be pretty small and fast moving on a Kanban board, we aren't there and our cycle time is atrocious for all sizes of stories.</div><div><br /></div><div>Our work at the moment doesn't lend itself to a disciplined backlog of small similar sized items ready to be committed to, and being closer to operations than some teams we feel the brunt of lots of expedited stories, still classified as such, but nevertheless contributing to unclear focus at times. </div><div><br /></div><div>The answer to the sub-heading is no, probably not at the moment, but we might need to do something.</div><div><br /></div><h3 style="text-align: left;">In Summary</h3><div>So after all that rambling where was I going. The key things I'd consider are:<br /><ul style="text-align: left;"><li>Can you get any more out of your current workflow framework, can you tweak some of the ceremonies, things you deem to have more value before changing tack</li><li>If your lead time and cycle times are decent, and you can give confident landing zones for work coming your way, do you even need to consider changing. If you don't know the answer to the former then you cant possibly know the answer to the latter, some will say - "most definitely not"</li><li>Are you moving framework to get away from some of the things you dislike. If there is a loss of inspection and adaptation as a result of the move, you may end up becoming less agile</li><li>What type of work are you doing? Although SCRUM is cited as being able to deal with projects of complexity and uncertainty (complex and adaptive), arguably a little less uncertainty may be more suitable for SCRUM in my opinion </li><li>Moving to Kanban - to make it as effective as possible, apparently, you want to get your stories as similar a size as possible. Is your backlog truly "dimensionable" like this. If sizes vary, is SCRUM still more effective? (I don't know to be honest, I do know estimation is still a desirable attribute in both and after calibration story sizes should be relative in SCRUM) So if you move to Kanban and you have stories of different sizes how can WIP be worked out effectively (I think its harder) )</li><li>Are you doing it just for a change? Might work out...Probably want a bit more to it than that...</li><li>Are you running away from the things you don't like (I am guilty of this to an extent) Can you truly get some of the things you don't like changed or improved - have you asked or challenged the status Quo (75 person reviews...anyone...yawn yawn yawn)</li></ul><div>I started repeating myself at the end and had to stop for my sanity...I will refer to this as a reminder next time I am involved in a switcharoo somewhere it might make me think twice... </div></div><div><br /></div><div><br /></div><div><br /></div><h3 style="text-align: left;"><br /></h3><div><br /></div><div><br /></div><div><br /></div><div><br /></div>colhttp://www.blogger.com/profile/11580119772205731598noreply@blogger.com2tag:blogger.com,1999:blog-5375025131541210012.post-30395013646623679222021-01-23T07:34:00.007-08:002021-01-24T13:33:15.512-08:00 Book Review: Test Driven Development By Example (Kent Beck)<div style="text-align: left;"><h3 style="text-align: left;">GitHub</h3>I have coded the examples and both are available in GitHub. <a href="https://github.com/colindooley11/Money-Kent-Beck">Money </a> and <a href="https://github.com/colindooley11/xUnit-Kent-Beck">xUnit </a></div><div><br /></div><h3 style="text-align: left;">Delayed...</h3><div>I am recalling this after a few weeks finishing the book (I should've done this immediately so my memory is hazy, but the code examples should attest to the fact I have read it, if my descriptions of the text are a little ambiguous) </div><div><br /></div><div>I have read this book on a number of occasions previously, in reference form and once all the way through, but hadn't worked through the examples in a code editor. I think, for real value, you have to code the examples, after a while it does feel like you are actually pairing, like others who have reviewed this though, I think the humour leaves a bit to be desired but its not too distracting!</div><div><br /></div><div>I have been TDDing for a while but this was a good refresher and reminded me of the maxims and techniques "fake it till you make it", "obvious implementation" and "triangulation". There are also some nice "red" and "green" patterns described later and I refactored to "pluggable selector "and a "command" pattern in some code in production just a week later (it was a nice refactoring and made some abstractions that were needed, nice and crisp)</div><div><br /></div><h3 style="text-align: left;">Money</h3><div>The money example was harder for me to get to grips with (contrary to Kent saying that the xUnit example is harder) because I found it more abstract and contrived . Its coded in Java and is very easy to follow from a syntax point of view, there are <i>few </i>gaps from Kent in guiding readers when building this example out. </div><div><br /></div><div>When having money work with expressions, the abstractions become a little unclear to me and I don't know if I would ever get to the design presented - and - if I am honest - in my first pass I didn't fully understand it all. It seems that I am not the only person to have found some of the abstractions confusing, this <a href="https://stackoverflow.com/questions/61667891/why-does-class-money-extend-expression-in-kent-becks-tdd-by-example">guy </a>did too.</div><div><br /></div><div>The xUnit example, however, I really enjoyed. </div><div><br /></div><h3 style="text-align: left;">xUnit</h3><div>I stumbled on the driver for the xUnit example and how it ended up the way it did when finding this from Kent. <a href="http://junit.sourceforge.net/doc/cookstour/cookstour.htm">The Junit cooks tour</a>, the arrangement of all the classes and the patterns used for JUnit are clearly the basis for the xUnit example, this adds extra gravitas and realism to the example because JUnit is an important bit of software!</div><div><br /></div><div>The exercise and the steps to help drive out the xUnit framework are still relatively small but they get a bit bigger towards the end and Kent leaves a few more open ended bits to fill in yourself.<br /><br />One thing which is a bit "chicken and eggy" with getting the test framework going (Kent does refer to an "Inception" like approach at the start of the example), and can be overcome by bootstrapping in itself, I suppose, is that Kent uses Python in his example, which has a built in "Assert" method, this is convenient, if using Python, however it is not possible in C# (without building out Assert or using another test framework, which has Assert, to build it out) </div><div><br /></div><div>This means that you can start using Assert (if writing in Python) without creating Assert, this did leave me wondering which order to build Assert out (and this isn't covered in the book). I.e. "should I build Assert first" or "should I bootstrap the test framework and then build out Assert with the test framework" or even " build out the test framework with another test framework". You might think I've lost the plot here , but - if you read the chapter - you will know what I mean. The rambling above aside, I actually built out Assert last and relied upon "Debug.WriteLine" and visual checks to ensure tests passed. I developed and started to back fill Assert in the tests I have uploaded but I was getting a bit confused about how to test it. Take a look, hopefully you can see what I mean. </div><div><br /></div><div>I really did have a smile on my face when you are taken through creating the test framework with a test (WasRun - terrible name :)) which is then tested by the test framework you just created. </div><div>Basically building out "template method" in small steps ensuring small parts of what is used to create a typical test fixture "SetUp", "Run", "Teardown" and eventually a TestSuite, are built up over the example (and thus you are exposed to Template, Command, Pluggable Selector and Composite patterns - all very interesting) . This intimate insight allowed me to imagine how a TestRunner would evolve and enumerate test cases (this isn't actually done in the book or in my coded example) The example could obviously be taken on and used as a basis for your own test framework.</div><div><br /></div><h3 style="text-align: left;">In Summary</h3><div>Its a good book and the key points about writing a failing test and building out the functionality to build it along with going in smaller and bigger steps when you need to, are reiterated many times. The introduction of using a checklist to remind yourself of things you need to give attention to as you reveal the design, as you go, is a good tip too, I've used this on lots of occasions in work (and my colleagues have to) but it tends to fizzle out (clever people - I have read - are usually pretty regimented and consistent in at least some of the stuff they do - I am neither - ask someone who knows me )</div><div><br /></div><div>Having read Sandi Metz's book- <a href="https://sandimetz.com/99bottles">99 bottles of OOP</a>, I noticed that she has even better commentary about cutting over when refactoring (by way of comparison - neither book is about refactoring per-se) so there are obviously other books (Fowler's <a href="https://martinfowler.com/articles/refactoring-2nd-ed.html">Refactoring </a>chief among them) that can help with specific aspects of the TDD cycle (red-green-refactor) but this is a good overall book about approaching TDD. </div><div><br /></div><div><br /></div><div><br /></div>colhttp://www.blogger.com/profile/11580119772205731598noreply@blogger.com0tag:blogger.com,1999:blog-5375025131541210012.post-77772559202607713012017-10-31T16:35:00.001-07:002021-01-23T07:34:55.855-08:00Book Study: 99 Bottles of OOP by Sandi MetzReview to follow, just finishing it. Basically its good. Perhaps that is enough of a review. colhttp://www.blogger.com/profile/11580119772205731598noreply@blogger.com0tag:blogger.com,1999:blog-5375025131541210012.post-12543232839266221022017-09-10T16:09:00.001-07:002019-03-18T17:53:23.705-07:00End to End testing using Reactive Extensions (RX)<span style="font-size: large;">A change of tack is needed with my posts. The limited people who have read my blog have been too kind to say that I talk a bit too much in my posts. So I've read back over my posts and I am musing like some kind of pseudo-intellectual who is smiling smugly and patting himself on the back over some kind of new insight into stuff which is really just a regurgitation. <br /><br />So I should stop it and keep the musings to a minimum from here on in. I've rewritten this post to try and give something valuable that can be taken away and used. <br /><br /><b>Requisite Knowledge</b></span><br />
<br />
<span style="font-size: large;">General C#, Generics, Messaging Patterns, NServiceBus, Linq, Understanding of asynchrony, Task<T>, Unit testing. </span><br />
<span style="font-size: large;"><br /></span>
<br />
<h2>
<span style="font-size: large;">Reactive Extensions </span></h2>
<span style="font-size: large;">I decided to use Rx to help end to end test a payment method service. I wont go into a lot of detail about Rx, and instead refer you to an absolutley fantastic reference, which can be found at <a href="http://reactivex.io/">reactivex.io</a>. The visualisations - especially - help bring to life what Rx does and can help stimulate ideas about how it can be used for you. <a href="https://github.com/Reactive-Extensions/Rx.NET">Rx.NET</a> is what the code examples (in C#), here, are programmed against. </span><br />
<span style="font-size: large;"></span>
<span style="font-size: large;"></span>
<br />
<h2>
<span style="font-size: large;">The problem </span></h2>
<span style="font-size: large;"><b>I need to be able to test that my payment method service (service) is working by ensuring that, given various input, it is producing the correct output (messages) in a timely fashion.</b> </span><br />
<br />
<span style="font-size: large;">I need a way to be able to ask the following questions "Given xyz input, has a payment accepted event occured?", and if so "Does it relate to this payment reference and has it arrived within10 seconds?". If so, great my service is working, if not, then it isn't. </span><span style="font-size: large;"> </span><br />
<br />
<span style="font-size: large;">The service to be tested comprises a front end (RESTful like API) and a backend server (producing messages). From a testing perspective, appropriate input should be supplied via the API and then various messages from the server should occur as a result. This <i>is </i>going to need<i> </i><a href="https://martinfowler.com/articles/microservice-testing/#testing-end-to-end-introduction">end to end testing. </a> (Edit 1: So I am saying we need end to end testing IF we want to exercise a fully working service with all its real world external dependencies AND this is necessary as the API and server reside in <i>different</i> processes. We <i>could </i>still component test the API and backend server - each individually - by swapping out external dependencies but this wouldn't provide us an integrated test . There are advocates and detractors of end to end testing along with its tradeoffs. </span><br />
<span style="font-size: large;">(Edit 2: I am choosing to state end to end testing over "external" component testing whereby external colloborators (I/O, Http) are included in the mix - because the Microservice which logically comprises an API, and other backend messages servers are involved - bit of a minefield!)<br /><br />My experience is that, if not insanely difficult or expensive (in time, complexity, and maintenance) it <i>is</i> worth it for the confidence (edit: and coverage) an end to end test can bring. </span><br />
<br />
<span style="font-size: large;">The services in our platform use <a href="https://particular.net/nservicebus">NServiceBus</a> (NSB) to send and receive what are usually domain events (edit: synonomous with messages), we can leverage this to get a handle on these messages and inspect them or lack of them. </span><br />
<span style="font-size: large;"><br /></span>
<span style="font-size: large;">To be able to ask the questions above I need to be able to <i>observe (listen) </i>on a <i>channel </i>for the messages (an observable sequence) and then run a test asking relevant questions. <br /><br />In this instance the channel will be an event handler sitting atop an associated "bus" (set up not shown here) - an abstraction NSB provides to represent the medium on which messages are passed. </span><br />
<span style="font-size: large;"><br /></span>
<span style="font-size: large;">The handler:</span><br />
<br />
<span style="font-size: x-small;">public class GenericMessageHandler : IHandleMessages<object><br /> {<br /> public void Handle(object message)<br /> {<br /> Observer.OnNotify(message);<br /> }<br /> }</span><br />
<span style="font-size: large;"><br /></span>
<span style="font-size: large;">The handlers job is to listen to messages which arrive at the handler and notify the Observer. The Observer is a facade for what is known in Rx.NET as a ReplaySubject, which is a special type of Observer (unfortunately my facade reuses/overloads the term Observer), the Observer has or will be setup with a bunch of subscribers which will be the tests (or a way to set the results of the tests**) to run for a certain type of message.</span><br />
<br />
<span style="font-size: large;">The ReplaySubject allows buffering so that if a "would be" subscriber (which we will be using to hook up tests) is not yet subscribed, and messages <i>have </i>started occuring on the channel (<i>that is:</i> arriving at the handler and being pushed into the ReplaySubject buffer) then when the subscriber <i>is </i>subscribed all messages which have happened will be replayed to the subscriber. </span><br />
<br />
<span style="font-size: large;">This special property (subject to limited buffering) means its not important that a subscriber is hooked up straight away, as long as the Observer (the ReplaySubject) is getting messages and, so, in this way the chances of missing something on the channel are reduced. </span><br />
<span style="font-size: large;"><br /></span>
<span style="font-size: large;">It is worth nothing that NSB uses publish/subscribe, so some setup has to be provided to ensure that messages of certain types are subscribed to and therefore will arrive at the handler. This code is omitted here. ( available in git shortly). </span><br />
<span style="font-size: large;"><br /></span>
<span style="font-size: large;">The Observer facade has 2 users:</span><br />
<ul>
<li><span style="font-size: large;"> the channel </span></li>
<li><span style="font-size: large;">unit tests</span><span style="font-size: large;"> </span></li>
</ul>
<span style="font-size: large;">The channel pushes in messages to the Observer</span><span style="font-size: large;"><br />The unit tests will be set up to "ask the questions" and provide the tests against those messages and these tests will be "attached/subscribed" to the Observer.</span><br />
<br />
<br />
<br />
<span style="font-size: large;">The Observer looks like this:</span><br />
<br />
<br />
<span style="font-size: large;"><span style="font-size: x-small;">public class Observer<br /> {<br /> private static readonly ReplaySubject<object> MessageObservable = new ReplaySubject<object>();<br /><br /> public static void OnNotify(object message)<br /> {<br /> MessageObservable.OnNext(message);<br /> }<br /><br /> public static Task<T> SubscribeToMessageTest<T>(Func<T, bool> test, string failureMessage, TimeSpan timeout) where T : class<br /> {<br /> var tcs = new TaskCompletionSource<T>();<br /><br /> MessageObservable<br /> .Timeout(timeout)<br /> .FirstAsync(message => RunTest(message, test))<br /> .Subscribe(<br /> message => OnSuccess(tcs, message),<br /> exception => OnFailure(exception, tcs, failureMessage));<br /><br /> return tcs.Task;<br /> }<br /><br /> private static bool RunTest<T>(object message, Func<T, bool> messageTest) where T : class<br /> {<br /> var convertedMessage = ConvertMessage<T>(message);<br /> if (convertedMessage != null)<br /> {<br /> return messageTest(convertedMessage);<br /> }<br /> <br /> return false;<br /> }<br /><br /> private static T ConvertMessage<T>(object message) where T : class<br /> {<br /> var messageAsType = message as T;<br /> return messageAsType;<br /> }<br /><br /> private static void OnSuccess<T>(TaskCompletionSource<T> tcs, object message) where T : class<br /> {<br /> tcs.SetResult(ConvertMessage<T>(message));<br /> }<br /><br /> private static void OnFailure<T>(Exception ex, TaskCompletionSource<T> tcs, string failureMessage)<br /> {<br /> Console.WriteLine(failureMessage);<br /> var messsageException = new PublishTestException(failureMessage, ex);<br /> tcs.SetException(messsageException);<br /> }<br /> } </span></span><br />
<span style="font-size: large;"><br /></span>
<span style="font-size: large;"><br />The most interesting method is:<br /> <span style="font-size: x-small;">Task<T> SubscribeToMessageTest<T>(Func<T,bool> test, string failureMessage, TimeSpan timeout)</span></span><br />
<span style="font-size: large;"><br /></span>
<span style="font-size: large;">At a highlevel then: a test is passed in which has previously been setup to be executed against a specific type of message (when a message occurs on the channel and appears in what is known as an observable sequence) </span><br />
<span style="font-size: large;">A TaskCompletionSource is created (tcs) whose result is going to be set with a message if the associated test is successful or, an exception if it takes longer than a certain time. A test is always unsuccessful if it can't be satisfied within the specified timeout period. <br />The task from the TaskCompletionSource is then returned. </span><br />
<span style="font-size: large;"><br /></span>
<span style="font-size: large;">The parameter "test" is a Func<T, bool> (a generic predicate) to be hooked up to test elements in an observable sequence (the messags from the channel), to MessageObservable (which is a ReplaySubject<object>. </span><br />
<span style="font-size: large;"><br /></span>
<span style="font-size: large;">This is done with the lines:</span><br />
<span style="font-size: x-small;">MessageObservable</span><br />
<span style="font-size: x-small;">.Timeout(timeout)</span><br />
<span style="font-size: x-small;">.FirstAsync(message => RunTest(message, test))</span><br />
<span style="font-size: large;"><br /></span><span style="font-size: large;">F</span><span style="font-size: large;">irstly, via Timeout, a timeout is set (if this is exceeded an exception is thrown) <br />Secondly, via FirstAsync, return ONLY the elements from the observable sequence (the messages from the channel) which pass the test passed to RunTest. The RunTest method does some conversion on the message, and runs the test. </span><br />
<span style="font-size: large;"><br /></span>
<span style="font-size: large;">The next 3 lines:</span><br />
<span style="font-size: x-small;">.Subscribe(<br /> message => OnSuccess(tcs, message),<br /> exception => OnFailure(exception, tcs, failureMessage));</span><br />
<br />
<span style="font-size: large;">Subscribes the message to a couple of methods which set the state of the TaskCompletionSource (tcs) declared at the top of the method. <br />OnSuccess takes the message itself and puts it into the result of the tcs. <br />OnFailure occurs when a timeout occurs and the exception is put on to the exception on tcs. <br />Not forgetting that the task associated with tcs was returned earlier and so consumers of the code would be notified as and when this occurs. </span><br />
<br />
<span style="font-size: large;">The SubscribeToMessageTest method is facaded with a PublishTest<T>:</span><br />
<br />
<span style="font-size: large;"><span style="font-size: x-small;"> public class PublishTest<T> where T : class<br /> {<br /> private Task<T> task; <br /><br /> /// <summary><br /> /// Test a message that satisfies a predicate, before a timeout occurs.<br /> /// </summary><br /> /// <typeparam name="T">The type of the message to test.</typeparam><br /> /// <param name="predicate">A predicate with tests you need to determine the correct message has been published.</param><br /> /// <param name="failureMessage">A message to aid debugging the test failure.</param><br /> /// <param name="timeout">Timespan indicating how long to wait for a message to be received.</param><br /> /// <returns>Task which should be awaited or Result inspected.</returns><br /> public PublishTest(Func<T, bool> predicate, string failureMessage, TimeSpan timeout)<br /> {<br /> this.task = Observer.SubscribeToMessageTest(predicate, failureMessage, timeout);<br /> }<br /><br /> /// <summary><br /> /// Block whilst waiting for test to verify. Use where async/await is not supported. <br /> /// For example, SpecFlow does not currently support async/await. <br /> /// </summary><br /> /// <returns>Message associated with successful test.</returns><br /> public T Verify()<br /> {<br /> return this.task.Result;<br /> }<br /><br /> /// <summary><br /> /// Verify the test at a later point in time by calling .Result or using async/await on the Task returned. <br /> /// </summary><br /> /// <returns>Task with message assocated with successful test.</returns><br /> public Task<T> VerifyAsync()<br /> {<br /> return this.task;<br /> }<br /> } </span></span><br />
<br />
<br />
<br />
<span style="font-size: large;">A test can be written Which is then used like so in a test:</span><br />
<br />
<span style="font-size: large;"> <span style="font-size: x-small;"> [Then(@"a PayPalPaymentIntentAcceptedInternalV4 is published")]<br /> public void ThenAPayPalPaymentAcceptedIsPublished()<br /> {<br /> new PublishTest<PayPalPaymentIntentAcceptedInternalV4>(<br /> message => message.PaymentReference == this.context.PaymentReference,<br /> "No PayPalPaymentIntentAcceptedInternalV4 message was published matching the specified test.",<br /> this.context.MessageVerifyWaitTime).Verify();<br /> } </span></span><br />
<br />
<br />
<span style="font-size: large;">The lambda - which provides the actual test - (message => message.PaymentReference...) and the type "PayPalPaymentIntentAcceptedV4" are used in tandem to qualify that firstly a test for this message type should take place and that the message is associated with input supplied earlier at the API, which would have the same payment reference. The lambda/test is ultimately what is passed to RunTest in the Observer class (which was listed earlier). </span><br />
<br />
<span style="font-size: large;">**As I am writing this I can see some improvements I would like to make to this code and will add an edit (also adding proper code snippets). I have left out some of the set up code around NServiceBus and the handler and setup to subscribe to messages. Also the other parts of the tests are omitted for brevity. ( To be included in git). </span><br />
<span style="font-size: large;"> </span><br />
<span style="font-size: large;"></span><br />
<span style="font-size: large;"></span><br />
<span style="font-size: large;"></span><br />
<span style="font-size: large;"></span><br />
<span style="font-size: large;"></span><br />
<span style="font-size: large;"></span><br />
<span style="font-size: large;"></span><br />
<span style="font-size: large;"></span><br />
<span style="font-size: large;"></span><br />
<span style="font-size: large;"></span><br />
<span style="font-size: large;"><br /></span>
<span style="font-size: large;"><br /></span>
<span style="font-size: large;"><br /></span>
<span style="font-size: large;"><br /></span>
<br />
<span style="font-size: large;"><br /></span>
<br />
<span style="font-size: large;"><br /></span>colhttp://www.blogger.com/profile/11580119772205731598noreply@blogger.com0tag:blogger.com,1999:blog-5375025131541210012.post-76377042458270329982017-04-12T09:03:00.000-07:002017-04-12T09:12:34.390-07:00Micro Post: Big Glass of Sand<span style="font-size: large;">Monolithic architecture is classified as a Big Ball of Mud</span><br />
<span style="font-size: large;">Microservices architecture is therefore a Big Glass of Sand</span><br />
<span style="font-size: large;"><br /></span>
<span style="font-size: large;">I said it first , right here. </span>colhttp://www.blogger.com/profile/11580119772205731598noreply@blogger.com2tag:blogger.com,1999:blog-5375025131541210012.post-59762208230481130342017-03-07T13:45:00.000-08:002017-03-07T13:45:10.272-08:00Testing, Testing, 1.2.3.4.5.6..ad infinitum<h2>
<span style="font-size: large;">Before testing...</span></h2>
<span style="font-size: large;">I've now been in development/engineering/coding/hacking or whatever you want to call it for more than 10 years. Testing has always been important, but it - in my line of work - is now at the forefront of everything I do. </span><br />
<br />
<span style="font-size: large;">For the business this is great, every line of code, component and group of components is covered by some set of tests. Be it unit, integration, component...the overloaded terms go on. This means that a set of requirements - to some degree - have been determined to have been met by these tests and the business is getting what it wants (all manner of development and agile practices not withstanding - they all get us to the same place) </span><br />
<br />
<span style="font-size: large;">It struck me today though that earlier in my career, when testing was still important but lines were - perhaps - a little more blurred and things were a bit more hit and miss, things were less clinical, less contrived and quite frankly less boring. This is obviously not a good model for delivering quality software everytime but sometimes I yearn to literally hack out some code, build it and just experiment when things aren't working quite right. Yes this can be done guided by tests and within the confines of the TDD cycle but, for me, having not come from a TDD beginning, feels a little stifiling if I was developing a hobby project (not something I was contributing to on GIT) I, dare I say it, would not necessarily use TDD - but make no mistake, at this time, developing quality software for my employers I would not develop without TDD, it gives me confidence I never had in the past. Boring yes, Confidence inspiring yes. </span><br />
<br />
<span style="font-size: large;">This kind of work is left now for spiking or hobby projects, but it feels that, and although this term is bandied around, software <b>engineering </b> really is coming of age. </span><br />
<br />
<h2>
<span style="font-size: large;">Engineering, Not art/craft </span></h2>
<span style="font-size: large;">I often hear that software development is a craft or an art and that it is creative - this may well be the case for UI related disciplines but business software is the geography teacher of practices, boring and wearing tweed. Once all the requirements are down and how these are going to be met are sorted - that's it. Software is being driven out in a very prescriptive manner. The biggest debates that happen in my team are not religious wars about coding standards - we have static code analysis, not religious wars about performance - we have resources to burn, not religious wars about general designs issues we have SOLID (and other guiding design guidelines), but no - we have wars about how we are going to test something and where. ATDD brings this war right to the inception of a story/feature (sorry I often get these the wrong way around) and then we have a list of work todo and a checklist to say when its done. This is all very organised and delivers value, but I'd never realised how boring this actually is when you break it down. </span><br />
<br />
<h2>
<span style="font-size: large;">Summary </span></h2>
<span style="font-size: large;">This might sound negative, and boring is generally not considered good, but it is a matter or perspective. Boring is good, as I said before, for the business. Now, I enjoy going to work and doing a good job, but I am wondering what it is I actually like doing anymore, maybe I have become boring and that's what I now like. That's what my Mrs thinks anyway...</span><br />
<br />
<span style="font-size: large;"> </span><br />
<h2>
<span style="font-size: large;"><br /></span></h2>
<span style="font-size: large;"> </span>colhttp://www.blogger.com/profile/11580119772205731598noreply@blogger.com0tag:blogger.com,1999:blog-5375025131541210012.post-61009259084718712832016-12-06T14:00:00.000-08:002016-12-06T14:52:30.853-08:001..2…3…Freeze….Peak<!--[if gte mso 9]><xml>
<o:OfficeDocumentSettings>
<o:AllowPNG/>
</o:OfficeDocumentSettings>
</xml><![endif]--><br />
<!--[if gte mso 9]><xml>
<w:WordDocument>
<w:View>Normal</w:View>
<w:Zoom>0</w:Zoom>
<w:TrackMoves/>
<w:TrackFormatting/>
<w:PunctuationKerning/>
<w:ValidateAgainstSchemas/>
<w:SaveIfXMLInvalid>false</w:SaveIfXMLInvalid>
<w:IgnoreMixedContent>false</w:IgnoreMixedContent>
<w:AlwaysShowPlaceholderText>false</w:AlwaysShowPlaceholderText>
<w:DoNotPromoteQF/>
<w:LidThemeOther>EN-US</w:LidThemeOther>
<w:LidThemeAsian>X-NONE</w:LidThemeAsian>
<w:LidThemeComplexScript>X-NONE</w:LidThemeComplexScript>
<w:Compatibility>
<w:BreakWrappedTables/>
<w:SnapToGridInCell/>
<w:WrapTextWithPunct/>
<w:UseAsianBreakRules/>
<w:DontGrowAutofit/>
<w:SplitPgBreakAndParaMark/>
<w:EnableOpenTypeKerning/>
<w:DontFlipMirrorIndents/>
<w:OverrideTableStyleHps/>
</w:Compatibility>
<m:mathPr>
<m:mathFont m:val="Cambria Math"/>
<m:brkBin m:val="before"/>
<m:brkBinSub m:val="--"/>
<m:smallFrac m:val="off"/>
<m:dispDef/>
<m:lMargin m:val="0"/>
<m:rMargin m:val="0"/>
<m:defJc m:val="centerGroup"/>
<m:wrapIndent m:val="1440"/>
<m:intLim m:val="subSup"/>
<m:naryLim m:val="undOvr"/>
</m:mathPr></w:WordDocument>
</xml><![endif]--><!--[if gte mso 9]><xml>
<w:LatentStyles DefLockedState="false" DefUnhideWhenUsed="false"
DefSemiHidden="false" DefQFormat="false" DefPriority="99"
LatentStyleCount="374">
<w:LsdException Locked="false" Priority="0" QFormat="true" Name="Normal"/>
<w:LsdException Locked="false" Priority="9" QFormat="true" Name="heading 1"/>
<w:LsdException Locked="false" Priority="9" SemiHidden="true"
UnhideWhenUsed="true" QFormat="true" Name="heading 2"/>
<w:LsdException Locked="false" Priority="9" SemiHidden="true"
UnhideWhenUsed="true" QFormat="true" Name="heading 3"/>
<w:LsdException Locked="false" Priority="9" SemiHidden="true"
UnhideWhenUsed="true" QFormat="true" Name="heading 4"/>
<w:LsdException Locked="false" Priority="9" SemiHidden="true"
UnhideWhenUsed="true" QFormat="true" Name="heading 5"/>
<w:LsdException Locked="false" Priority="9" SemiHidden="true"
UnhideWhenUsed="true" QFormat="true" Name="heading 6"/>
<w:LsdException Locked="false" Priority="9" SemiHidden="true"
UnhideWhenUsed="true" QFormat="true" Name="heading 7"/>
<w:LsdException Locked="false" Priority="9" SemiHidden="true"
UnhideWhenUsed="true" QFormat="true" Name="heading 8"/>
<w:LsdException Locked="false" Priority="9" SemiHidden="true"
UnhideWhenUsed="true" QFormat="true" Name="heading 9"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="index 1"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="index 2"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="index 3"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="index 4"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="index 5"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="index 6"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="index 7"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="index 8"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="index 9"/>
<w:LsdException Locked="false" Priority="39" SemiHidden="true"
UnhideWhenUsed="true" Name="toc 1"/>
<w:LsdException Locked="false" Priority="39" SemiHidden="true"
UnhideWhenUsed="true" Name="toc 2"/>
<w:LsdException Locked="false" Priority="39" SemiHidden="true"
UnhideWhenUsed="true" Name="toc 3"/>
<w:LsdException Locked="false" Priority="39" SemiHidden="true"
UnhideWhenUsed="true" Name="toc 4"/>
<w:LsdException Locked="false" Priority="39" SemiHidden="true"
UnhideWhenUsed="true" Name="toc 5"/>
<w:LsdException Locked="false" Priority="39" SemiHidden="true"
UnhideWhenUsed="true" Name="toc 6"/>
<w:LsdException Locked="false" Priority="39" SemiHidden="true"
UnhideWhenUsed="true" Name="toc 7"/>
<w:LsdException Locked="false" Priority="39" SemiHidden="true"
UnhideWhenUsed="true" Name="toc 8"/>
<w:LsdException Locked="false" Priority="39" SemiHidden="true"
UnhideWhenUsed="true" Name="toc 9"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Normal Indent"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="footnote text"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="annotation text"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="header"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="footer"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="index heading"/>
<w:LsdException Locked="false" Priority="35" SemiHidden="true"
UnhideWhenUsed="true" QFormat="true" Name="caption"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="table of figures"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="envelope address"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="envelope return"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="footnote reference"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="annotation reference"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="line number"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="page number"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="endnote reference"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="endnote text"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="table of authorities"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="macro"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="toa heading"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List Bullet"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List Number"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List 2"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List 3"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List 4"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List 5"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List Bullet 2"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List Bullet 3"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List Bullet 4"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List Bullet 5"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List Number 2"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List Number 3"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List Number 4"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List Number 5"/>
<w:LsdException Locked="false" Priority="10" QFormat="true" Name="Title"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Closing"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Signature"/>
<w:LsdException Locked="false" Priority="1" SemiHidden="true"
UnhideWhenUsed="true" Name="Default Paragraph Font"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Body Text"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Body Text Indent"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List Continue"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List Continue 2"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List Continue 3"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List Continue 4"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List Continue 5"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Message Header"/>
<w:LsdException Locked="false" Priority="11" QFormat="true" Name="Subtitle"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Salutation"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Date"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Body Text First Indent"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Body Text First Indent 2"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Note Heading"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Body Text 2"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Body Text 3"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Body Text Indent 2"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Body Text Indent 3"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Block Text"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Hyperlink"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="FollowedHyperlink"/>
<w:LsdException Locked="false" Priority="22" QFormat="true" Name="Strong"/>
<w:LsdException Locked="false" Priority="20" QFormat="true" Name="Emphasis"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Document Map"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Plain Text"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="E-mail Signature"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="HTML Top of Form"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="HTML Bottom of Form"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Normal (Web)"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="HTML Acronym"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="HTML Address"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="HTML Cite"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="HTML Code"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="HTML Definition"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="HTML Keyboard"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="HTML Preformatted"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="HTML Sample"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="HTML Typewriter"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="HTML Variable"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Normal Table"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="annotation subject"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="No List"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Outline List 1"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Outline List 2"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Outline List 3"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Simple 1"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Simple 2"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Simple 3"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Classic 1"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Classic 2"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Classic 3"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Classic 4"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Colorful 1"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Colorful 2"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Colorful 3"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Columns 1"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Columns 2"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Columns 3"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Columns 4"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Columns 5"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Grid 1"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Grid 2"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Grid 3"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Grid 4"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Grid 5"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Grid 6"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Grid 7"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Grid 8"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table List 1"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table List 2"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table List 3"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table List 4"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table List 5"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table List 6"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table List 7"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table List 8"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table 3D effects 1"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table 3D effects 2"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table 3D effects 3"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Contemporary"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Elegant"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Professional"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Subtle 1"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Subtle 2"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Web 1"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Web 2"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Web 3"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Balloon Text"/>
<w:LsdException Locked="false" Priority="39" Name="Table Grid"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Theme"/>
<w:LsdException Locked="false" SemiHidden="true" Name="Placeholder Text"/>
<w:LsdException Locked="false" Priority="1" QFormat="true" Name="No Spacing"/>
<w:LsdException Locked="false" Priority="60" Name="Light Shading"/>
<w:LsdException Locked="false" Priority="61" Name="Light List"/>
<w:LsdException Locked="false" Priority="62" Name="Light Grid"/>
<w:LsdException Locked="false" Priority="63" Name="Medium Shading 1"/>
<w:LsdException Locked="false" Priority="64" Name="Medium Shading 2"/>
<w:LsdException Locked="false" Priority="65" Name="Medium List 1"/>
<w:LsdException Locked="false" Priority="66" Name="Medium List 2"/>
<w:LsdException Locked="false" Priority="67" Name="Medium Grid 1"/>
<w:LsdException Locked="false" Priority="68" Name="Medium Grid 2"/>
<w:LsdException Locked="false" Priority="69" Name="Medium Grid 3"/>
<w:LsdException Locked="false" Priority="70" Name="Dark List"/>
<w:LsdException Locked="false" Priority="71" Name="Colorful Shading"/>
<w:LsdException Locked="false" Priority="72" Name="Colorful List"/>
<w:LsdException Locked="false" Priority="73" Name="Colorful Grid"/>
<w:LsdException Locked="false" Priority="60" Name="Light Shading Accent 1"/>
<w:LsdException Locked="false" Priority="61" Name="Light List Accent 1"/>
<w:LsdException Locked="false" Priority="62" Name="Light Grid Accent 1"/>
<w:LsdException Locked="false" Priority="63" Name="Medium Shading 1 Accent 1"/>
<w:LsdException Locked="false" Priority="64" Name="Medium Shading 2 Accent 1"/>
<w:LsdException Locked="false" Priority="65" Name="Medium List 1 Accent 1"/>
<w:LsdException Locked="false" SemiHidden="true" Name="Revision"/>
<w:LsdException Locked="false" Priority="34" QFormat="true"
Name="List Paragraph"/>
<w:LsdException Locked="false" Priority="29" QFormat="true" Name="Quote"/>
<w:LsdException Locked="false" Priority="30" QFormat="true"
Name="Intense Quote"/>
<w:LsdException Locked="false" Priority="66" Name="Medium List 2 Accent 1"/>
<w:LsdException Locked="false" Priority="67" Name="Medium Grid 1 Accent 1"/>
<w:LsdException Locked="false" Priority="68" Name="Medium Grid 2 Accent 1"/>
<w:LsdException Locked="false" Priority="69" Name="Medium Grid 3 Accent 1"/>
<w:LsdException Locked="false" Priority="70" Name="Dark List Accent 1"/>
<w:LsdException Locked="false" Priority="71" Name="Colorful Shading Accent 1"/>
<w:LsdException Locked="false" Priority="72" Name="Colorful List Accent 1"/>
<w:LsdException Locked="false" Priority="73" Name="Colorful Grid Accent 1"/>
<w:LsdException Locked="false" Priority="60" Name="Light Shading Accent 2"/>
<w:LsdException Locked="false" Priority="61" Name="Light List Accent 2"/>
<w:LsdException Locked="false" Priority="62" Name="Light Grid Accent 2"/>
<w:LsdException Locked="false" Priority="63" Name="Medium Shading 1 Accent 2"/>
<w:LsdException Locked="false" Priority="64" Name="Medium Shading 2 Accent 2"/>
<w:LsdException Locked="false" Priority="65" Name="Medium List 1 Accent 2"/>
<w:LsdException Locked="false" Priority="66" Name="Medium List 2 Accent 2"/>
<w:LsdException Locked="false" Priority="67" Name="Medium Grid 1 Accent 2"/>
<w:LsdException Locked="false" Priority="68" Name="Medium Grid 2 Accent 2"/>
<w:LsdException Locked="false" Priority="69" Name="Medium Grid 3 Accent 2"/>
<w:LsdException Locked="false" Priority="70" Name="Dark List Accent 2"/>
<w:LsdException Locked="false" Priority="71" Name="Colorful Shading Accent 2"/>
<w:LsdException Locked="false" Priority="72" Name="Colorful List Accent 2"/>
<w:LsdException Locked="false" Priority="73" Name="Colorful Grid Accent 2"/>
<w:LsdException Locked="false" Priority="60" Name="Light Shading Accent 3"/>
<w:LsdException Locked="false" Priority="61" Name="Light List Accent 3"/>
<w:LsdException Locked="false" Priority="62" Name="Light Grid Accent 3"/>
<w:LsdException Locked="false" Priority="63" Name="Medium Shading 1 Accent 3"/>
<w:LsdException Locked="false" Priority="64" Name="Medium Shading 2 Accent 3"/>
<w:LsdException Locked="false" Priority="65" Name="Medium List 1 Accent 3"/>
<w:LsdException Locked="false" Priority="66" Name="Medium List 2 Accent 3"/>
<w:LsdException Locked="false" Priority="67" Name="Medium Grid 1 Accent 3"/>
<w:LsdException Locked="false" Priority="68" Name="Medium Grid 2 Accent 3"/>
<w:LsdException Locked="false" Priority="69" Name="Medium Grid 3 Accent 3"/>
<w:LsdException Locked="false" Priority="70" Name="Dark List Accent 3"/>
<w:LsdException Locked="false" Priority="71" Name="Colorful Shading Accent 3"/>
<w:LsdException Locked="false" Priority="72" Name="Colorful List Accent 3"/>
<w:LsdException Locked="false" Priority="73" Name="Colorful Grid Accent 3"/>
<w:LsdException Locked="false" Priority="60" Name="Light Shading Accent 4"/>
<w:LsdException Locked="false" Priority="61" Name="Light List Accent 4"/>
<w:LsdException Locked="false" Priority="62" Name="Light Grid Accent 4"/>
<w:LsdException Locked="false" Priority="63" Name="Medium Shading 1 Accent 4"/>
<w:LsdException Locked="false" Priority="64" Name="Medium Shading 2 Accent 4"/>
<w:LsdException Locked="false" Priority="65" Name="Medium List 1 Accent 4"/>
<w:LsdException Locked="false" Priority="66" Name="Medium List 2 Accent 4"/>
<w:LsdException Locked="false" Priority="67" Name="Medium Grid 1 Accent 4"/>
<w:LsdException Locked="false" Priority="68" Name="Medium Grid 2 Accent 4"/>
<w:LsdException Locked="false" Priority="69" Name="Medium Grid 3 Accent 4"/>
<w:LsdException Locked="false" Priority="70" Name="Dark List Accent 4"/>
<w:LsdException Locked="false" Priority="71" Name="Colorful Shading Accent 4"/>
<w:LsdException Locked="false" Priority="72" Name="Colorful List Accent 4"/>
<w:LsdException Locked="false" Priority="73" Name="Colorful Grid Accent 4"/>
<w:LsdException Locked="false" Priority="60" Name="Light Shading Accent 5"/>
<w:LsdException Locked="false" Priority="61" Name="Light List Accent 5"/>
<w:LsdException Locked="false" Priority="62" Name="Light Grid Accent 5"/>
<w:LsdException Locked="false" Priority="63" Name="Medium Shading 1 Accent 5"/>
<w:LsdException Locked="false" Priority="64" Name="Medium Shading 2 Accent 5"/>
<w:LsdException Locked="false" Priority="65" Name="Medium List 1 Accent 5"/>
<w:LsdException Locked="false" Priority="66" Name="Medium List 2 Accent 5"/>
<w:LsdException Locked="false" Priority="67" Name="Medium Grid 1 Accent 5"/>
<w:LsdException Locked="false" Priority="68" Name="Medium Grid 2 Accent 5"/>
<w:LsdException Locked="false" Priority="69" Name="Medium Grid 3 Accent 5"/>
<w:LsdException Locked="false" Priority="70" Name="Dark List Accent 5"/>
<w:LsdException Locked="false" Priority="71" Name="Colorful Shading Accent 5"/>
<w:LsdException Locked="false" Priority="72" Name="Colorful List Accent 5"/>
<w:LsdException Locked="false" Priority="73" Name="Colorful Grid Accent 5"/>
<w:LsdException Locked="false" Priority="60" Name="Light Shading Accent 6"/>
<w:LsdException Locked="false" Priority="61" Name="Light List Accent 6"/>
<w:LsdException Locked="false" Priority="62" Name="Light Grid Accent 6"/>
<w:LsdException Locked="false" Priority="63" Name="Medium Shading 1 Accent 6"/>
<w:LsdException Locked="false" Priority="64" Name="Medium Shading 2 Accent 6"/>
<w:LsdException Locked="false" Priority="65" Name="Medium List 1 Accent 6"/>
<w:LsdException Locked="false" Priority="66" Name="Medium List 2 Accent 6"/>
<w:LsdException Locked="false" Priority="67" Name="Medium Grid 1 Accent 6"/>
<w:LsdException Locked="false" Priority="68" Name="Medium Grid 2 Accent 6"/>
<w:LsdException Locked="false" Priority="69" Name="Medium Grid 3 Accent 6"/>
<w:LsdException Locked="false" Priority="70" Name="Dark List Accent 6"/>
<w:LsdException Locked="false" Priority="71" Name="Colorful Shading Accent 6"/>
<w:LsdException Locked="false" Priority="72" Name="Colorful List Accent 6"/>
<w:LsdException Locked="false" Priority="73" Name="Colorful Grid Accent 6"/>
<w:LsdException Locked="false" Priority="19" QFormat="true"
Name="Subtle Emphasis"/>
<w:LsdException Locked="false" Priority="21" QFormat="true"
Name="Intense Emphasis"/>
<w:LsdException Locked="false" Priority="31" QFormat="true"
Name="Subtle Reference"/>
<w:LsdException Locked="false" Priority="32" QFormat="true"
Name="Intense Reference"/>
<w:LsdException Locked="false" Priority="33" QFormat="true" Name="Book Title"/>
<w:LsdException Locked="false" Priority="37" SemiHidden="true"
UnhideWhenUsed="true" Name="Bibliography"/>
<w:LsdException Locked="false" Priority="39" SemiHidden="true"
UnhideWhenUsed="true" QFormat="true" Name="TOC Heading"/>
<w:LsdException Locked="false" Priority="41" Name="Plain Table 1"/>
<w:LsdException Locked="false" Priority="42" Name="Plain Table 2"/>
<w:LsdException Locked="false" Priority="43" Name="Plain Table 3"/>
<w:LsdException Locked="false" Priority="44" Name="Plain Table 4"/>
<w:LsdException Locked="false" Priority="45" Name="Plain Table 5"/>
<w:LsdException Locked="false" Priority="40" Name="Grid Table Light"/>
<w:LsdException Locked="false" Priority="46" Name="Grid Table 1 Light"/>
<w:LsdException Locked="false" Priority="47" Name="Grid Table 2"/>
<w:LsdException Locked="false" Priority="48" Name="Grid Table 3"/>
<w:LsdException Locked="false" Priority="49" Name="Grid Table 4"/>
<w:LsdException Locked="false" Priority="50" Name="Grid Table 5 Dark"/>
<w:LsdException Locked="false" Priority="51" Name="Grid Table 6 Colorful"/>
<w:LsdException Locked="false" Priority="52" Name="Grid Table 7 Colorful"/>
<w:LsdException Locked="false" Priority="46"
Name="Grid Table 1 Light Accent 1"/>
<w:LsdException Locked="false" Priority="47" Name="Grid Table 2 Accent 1"/>
<w:LsdException Locked="false" Priority="48" Name="Grid Table 3 Accent 1"/>
<w:LsdException Locked="false" Priority="49" Name="Grid Table 4 Accent 1"/>
<w:LsdException Locked="false" Priority="50" Name="Grid Table 5 Dark Accent 1"/>
<w:LsdException Locked="false" Priority="51"
Name="Grid Table 6 Colorful Accent 1"/>
<w:LsdException Locked="false" Priority="52"
Name="Grid Table 7 Colorful Accent 1"/>
<w:LsdException Locked="false" Priority="46"
Name="Grid Table 1 Light Accent 2"/>
<w:LsdException Locked="false" Priority="47" Name="Grid Table 2 Accent 2"/>
<w:LsdException Locked="false" Priority="48" Name="Grid Table 3 Accent 2"/>
<w:LsdException Locked="false" Priority="49" Name="Grid Table 4 Accent 2"/>
<w:LsdException Locked="false" Priority="50" Name="Grid Table 5 Dark Accent 2"/>
<w:LsdException Locked="false" Priority="51"
Name="Grid Table 6 Colorful Accent 2"/>
<w:LsdException Locked="false" Priority="52"
Name="Grid Table 7 Colorful Accent 2"/>
<w:LsdException Locked="false" Priority="46"
Name="Grid Table 1 Light Accent 3"/>
<w:LsdException Locked="false" Priority="47" Name="Grid Table 2 Accent 3"/>
<w:LsdException Locked="false" Priority="48" Name="Grid Table 3 Accent 3"/>
<w:LsdException Locked="false" Priority="49" Name="Grid Table 4 Accent 3"/>
<w:LsdException Locked="false" Priority="50" Name="Grid Table 5 Dark Accent 3"/>
<w:LsdException Locked="false" Priority="51"
Name="Grid Table 6 Colorful Accent 3"/>
<w:LsdException Locked="false" Priority="52"
Name="Grid Table 7 Colorful Accent 3"/>
<w:LsdException Locked="false" Priority="46"
Name="Grid Table 1 Light Accent 4"/>
<w:LsdException Locked="false" Priority="47" Name="Grid Table 2 Accent 4"/>
<w:LsdException Locked="false" Priority="48" Name="Grid Table 3 Accent 4"/>
<w:LsdException Locked="false" Priority="49" Name="Grid Table 4 Accent 4"/>
<w:LsdException Locked="false" Priority="50" Name="Grid Table 5 Dark Accent 4"/>
<w:LsdException Locked="false" Priority="51"
Name="Grid Table 6 Colorful Accent 4"/>
<w:LsdException Locked="false" Priority="52"
Name="Grid Table 7 Colorful Accent 4"/>
<w:LsdException Locked="false" Priority="46"
Name="Grid Table 1 Light Accent 5"/>
<w:LsdException Locked="false" Priority="47" Name="Grid Table 2 Accent 5"/>
<w:LsdException Locked="false" Priority="48" Name="Grid Table 3 Accent 5"/>
<w:LsdException Locked="false" Priority="49" Name="Grid Table 4 Accent 5"/>
<w:LsdException Locked="false" Priority="50" Name="Grid Table 5 Dark Accent 5"/>
<w:LsdException Locked="false" Priority="51"
Name="Grid Table 6 Colorful Accent 5"/>
<w:LsdException Locked="false" Priority="52"
Name="Grid Table 7 Colorful Accent 5"/>
<w:LsdException Locked="false" Priority="46"
Name="Grid Table 1 Light Accent 6"/>
<w:LsdException Locked="false" Priority="47" Name="Grid Table 2 Accent 6"/>
<w:LsdException Locked="false" Priority="48" Name="Grid Table 3 Accent 6"/>
<w:LsdException Locked="false" Priority="49" Name="Grid Table 4 Accent 6"/>
<w:LsdException Locked="false" Priority="50" Name="Grid Table 5 Dark Accent 6"/>
<w:LsdException Locked="false" Priority="51"
Name="Grid Table 6 Colorful Accent 6"/>
<w:LsdException Locked="false" Priority="52"
Name="Grid Table 7 Colorful Accent 6"/>
<w:LsdException Locked="false" Priority="46" Name="List Table 1 Light"/>
<w:LsdException Locked="false" Priority="47" Name="List Table 2"/>
<w:LsdException Locked="false" Priority="48" Name="List Table 3"/>
<w:LsdException Locked="false" Priority="49" Name="List Table 4"/>
<w:LsdException Locked="false" Priority="50" Name="List Table 5 Dark"/>
<w:LsdException Locked="false" Priority="51" Name="List Table 6 Colorful"/>
<w:LsdException Locked="false" Priority="52" Name="List Table 7 Colorful"/>
<w:LsdException Locked="false" Priority="46"
Name="List Table 1 Light Accent 1"/>
<w:LsdException Locked="false" Priority="47" Name="List Table 2 Accent 1"/>
<w:LsdException Locked="false" Priority="48" Name="List Table 3 Accent 1"/>
<w:LsdException Locked="false" Priority="49" Name="List Table 4 Accent 1"/>
<w:LsdException Locked="false" Priority="50" Name="List Table 5 Dark Accent 1"/>
<w:LsdException Locked="false" Priority="51"
Name="List Table 6 Colorful Accent 1"/>
<w:LsdException Locked="false" Priority="52"
Name="List Table 7 Colorful Accent 1"/>
<w:LsdException Locked="false" Priority="46"
Name="List Table 1 Light Accent 2"/>
<w:LsdException Locked="false" Priority="47" Name="List Table 2 Accent 2"/>
<w:LsdException Locked="false" Priority="48" Name="List Table 3 Accent 2"/>
<w:LsdException Locked="false" Priority="49" Name="List Table 4 Accent 2"/>
<w:LsdException Locked="false" Priority="50" Name="List Table 5 Dark Accent 2"/>
<w:LsdException Locked="false" Priority="51"
Name="List Table 6 Colorful Accent 2"/>
<w:LsdException Locked="false" Priority="52"
Name="List Table 7 Colorful Accent 2"/>
<w:LsdException Locked="false" Priority="46"
Name="List Table 1 Light Accent 3"/>
<w:LsdException Locked="false" Priority="47" Name="List Table 2 Accent 3"/>
<w:LsdException Locked="false" Priority="48" Name="List Table 3 Accent 3"/>
<w:LsdException Locked="false" Priority="49" Name="List Table 4 Accent 3"/>
<w:LsdException Locked="false" Priority="50" Name="List Table 5 Dark Accent 3"/>
<w:LsdException Locked="false" Priority="51"
Name="List Table 6 Colorful Accent 3"/>
<w:LsdException Locked="false" Priority="52"
Name="List Table 7 Colorful Accent 3"/>
<w:LsdException Locked="false" Priority="46"
Name="List Table 1 Light Accent 4"/>
<w:LsdException Locked="false" Priority="47" Name="List Table 2 Accent 4"/>
<w:LsdException Locked="false" Priority="48" Name="List Table 3 Accent 4"/>
<w:LsdException Locked="false" Priority="49" Name="List Table 4 Accent 4"/>
<w:LsdException Locked="false" Priority="50" Name="List Table 5 Dark Accent 4"/>
<w:LsdException Locked="false" Priority="51"
Name="List Table 6 Colorful Accent 4"/>
<w:LsdException Locked="false" Priority="52"
Name="List Table 7 Colorful Accent 4"/>
<w:LsdException Locked="false" Priority="46"
Name="List Table 1 Light Accent 5"/>
<w:LsdException Locked="false" Priority="47" Name="List Table 2 Accent 5"/>
<w:LsdException Locked="false" Priority="48" Name="List Table 3 Accent 5"/>
<w:LsdException Locked="false" Priority="49" Name="List Table 4 Accent 5"/>
<w:LsdException Locked="false" Priority="50" Name="List Table 5 Dark Accent 5"/>
<w:LsdException Locked="false" Priority="51"
Name="List Table 6 Colorful Accent 5"/>
<w:LsdException Locked="false" Priority="52"
Name="List Table 7 Colorful Accent 5"/>
<w:LsdException Locked="false" Priority="46"
Name="List Table 1 Light Accent 6"/>
<w:LsdException Locked="false" Priority="47" Name="List Table 2 Accent 6"/>
<w:LsdException Locked="false" Priority="48" Name="List Table 3 Accent 6"/>
<w:LsdException Locked="false" Priority="49" Name="List Table 4 Accent 6"/>
<w:LsdException Locked="false" Priority="50" Name="List Table 5 Dark Accent 6"/>
<w:LsdException Locked="false" Priority="51"
Name="List Table 6 Colorful Accent 6"/>
<w:LsdException Locked="false" Priority="52"
Name="List Table 7 Colorful Accent 6"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Mention"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Smart Hyperlink"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Hashtag"/>
</w:LatentStyles>
</xml><![endif]--><!--[if gte mso 10]>
<style>
/* Style Definitions */
table.MsoNormalTable
{mso-style-name:"Table Normal";
mso-tstyle-rowband-size:0;
mso-tstyle-colband-size:0;
mso-style-noshow:yes;
mso-style-priority:99;
mso-style-parent:"";
mso-padding-alt:0in 5.4pt 0in 5.4pt;
mso-para-margin-top:0in;
mso-para-margin-right:0in;
mso-para-margin-bottom:10.0pt;
mso-para-margin-left:0in;
line-height:115%;
mso-pagination:widow-orphan;
font-size:11.0pt;
font-family:"Calibri",sans-serif;
mso-ascii-font-family:Calibri;
mso-ascii-theme-font:minor-latin;
mso-hansi-font-family:Calibri;
mso-hansi-theme-font:minor-latin;
mso-bidi-font-family:"Times New Roman";
mso-bidi-theme-font:minor-bidi;}
</style>
<![endif]-->
<br />
<div class="MsoNormal">
<span style="font-size: large;"><b><span lang="EN-GB"></span></b></span></div>
<div class="MsoNormal">
<span style="font-size: large;"><span lang="EN-GB">I haven’t
written anything for ages as I’ve been
the busiest I have ever been at my current employer – a large E-Commerce Website - as a result of preparation (planned and unplanned)
for its largest ever “Peak” weekend, my company’s name for the <b>Black Friday</b> weekend. </span></span><br />
<br />
<span style="font-size: large;"><span lang="EN-GB">Sorry this is a long read so grab a coffee or skip to the Summary, or don't read it!</span></span></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<span style="font-size: large;"><span lang="EN-GB">I’ve started writing this on the train coming
back from London where I was required to provide extra-support for the Payments
Platform (Domain) - where I ply my trade
– a group of Microservices and Legacy components supporting Payment processing
for the Website and sitting very closely to the Orders Domain. </span></span></div>
<div class="MsoNormal">
<span style="font-size: large;"><span lang="EN-GB"><br />
I expected to be bleary eyed and wired to the hilt, full of coffee and
adrenaline as me and my colleagues worked like mad to pull out all the stops to
make our giant incredible machine work properly. But no, as our CIO put it,
everything just worked (well 99% of the time – more on this later). </span></span><br />
<br /></div>
<div class="MsoNormal">
<span style="font-size: large;"><span lang="EN-GB">The freeze
part of the title refers to the fact that, in the weeks up to Peak, as we tried
to ensure stability of our software, only the most critical software releases were
allowed out. I , like many of my peers,
do not think this is great. This is
costly in many ways; not least, all of the feature development still going on
actively within the company is shelved and starts to gather dust and not being
released means the software is accruing debt on many fronts. However, I can’t be too hasty in judgement,
this way of working comes from something along the way biting my company really
hard during this crucial trading period, this mentality is understandable if
not immediately completely excusable. Hopefully,
this will get better as the company gains more maturity, confidence and control
over its systems.</span></span></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<b><span style="font-size: large;"><span lang="EN-GB">Application Insights</span></span></b></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<span style="font-size: large;"><span lang="EN-GB">As part of
the preparations it became clear that our services did not have sufficient
monitoring capabilities. Services were
performance tested in quasi-production like environments and so we had some
idea about how they might perform, but only an idea. So we had some expectations about performance
but very little in the way of helping us see operational health . Measuring and keeping track both would have been impossible without telemetry
and logging of some sort. </span></span><br />
<br /></div>
<div class="MsoNormal">
<span style="font-size: large;"><span lang="EN-GB">We already
had some monitoring available via dashboards powered by <b>Grafana</b>. A number of dashboards exist showing Order flow, and, as a
result, part of an Orders path through Payments, but only at a high level. With various counters representing the number
of Orders at a particular status, e.g. arriving from the Website, currently
being billed, and shipped to the warehouse, this is used to help try to
generalise the performance and health of backend systems as Orders wing their
way through the Website, through Order and Payment Processing and much more and
then, eventually, out to the Warehouse(s). </span></span><br />
<br /></div>
<div class="MsoNormal">
<span style="font-size: large;"><span lang="EN-GB">If problems
start occurring, by some thresholds being exceeded or not met, represented by numbers and traffic light
boxes, focussed investigations into issues can take place in a particular
area. The problem with this is that in
between the different statuses there are a vast swathe of software services in
locations all over the world involved: including New-World Microservices,
Legacy systems, stuff in the sky, Messaging, RESTful APIs, document DBs, SQL DBs, No-Sql Dbs, Edge caching, and
hardware on premise. Application support
have lots of fun trying to diagnose errors and, where they are at a loss ,they
hand over to software development teams/developers who mostly run around in
blind panic at the same time.</span></span><br />
<br /></div>
<div class="MsoNormal">
<span style="font-size: large;"><span lang="EN-GB">More
recently with the release of a updated versions of the Website (which is
multi-platform/device) <b>New Relic</b> was
enlisted in helping see how the various APIs consumed by the Website were behaving
and this has helped to start to build a more fine grained picture of the operational
behaviour of APIs ( and by association the Websites) where
various HTTP statuses can be obtained and detailed analysis about response
times, payload sizes, customer locations, can be gleaned, but that’s about it in
terms of insight into what an API is doing. There is yet another monitoring
tool called <b>Kibana (sitting atop of the
ELK stack)</b> which is used for some of our Azure based services, largely this
gives transparency to telemetry just like <b>New
Relic. </b>My money’s on New Relic in this area though.</span></span><br />
<br /></div>
<div class="MsoNormal">
<span style="font-size: large;"><span lang="EN-GB">So, the
introduction of performance counters, custom and otherwise were conveniently
placed at the door of our APIs, on the 3<sup>rd</sup> party calls within them
and in and around the messaging infrastructure used in the backend processing
for our services. Relevant logging was
added too, to ensure that in the event of failure or exceptional behaviour we
had some trace of what was going on. So,
having all of this information, but <i>not</i>
at your fingertips is a bit of a nightmare. </span></span><br />
<br />
<span style="font-size: large;"><span lang="EN-GB">Consolidating this all into one single place is achieved with dramatic
effect using Application Insights hosted in Azure. Simply adding appropriate libraries in the
service and hooking it up to an App Insights Resource Group in Azure and boom
you are presented with a bewildering array of options to present you with your
telemetry. Coupled with this you can
also leverage Analytics to perform queries on all of the data collected and
herein lies a problem. Making sense of
all of this information is <i>hard</i>. Certainly, there are some headline acts
easily noticeable but more subtle problems like a semi dodgy 3<sup>rd</sup>
party call, or reasonably flaky database writes can be eked out but not after
some <i>considered</i> tweaking with Queries
etc. Also without reasonable SLAs from
the business in most places and the fact that a message queueing system is used,
what alerts should be used and when is a question we are still asking
ourselves.</span></span></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<span style="font-size: large;"><b><span lang="EN-GB">Long nights and Instability</span></b></span></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<span style="font-size: large;"><span lang="EN-GB">So getting
to a good place a lot of pain needs to be had.
And boy did we experience our fair share up to peak. With SAN migrations causing mayhem with our
messaging, Windows Cluster failovers misbehaving and deleted Orders in Azure there
was plenty of time to practice and gain experience in trying to solve
problems quickly and still meet “customer
promise” a term used to describe meeting
cutoff for various delivery options, which is not taken lightly. </span></span></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<span style="font-size: large;"><b><span lang="EN-GB">Graph Watching First Blood Part III<br />
</span></b><span lang="EN-GB"><br />
So the culmination in all of the Black Friday prep is a series of graph
watching sessions, with eyes trained on graphs rolling around in multi-tabbed
browsers and lots of suggestions and conjecture about what this trend is saying
and what that trend is saying. I was
personally involved in 4 of the 5 days of Peak support, remotely on Thursday, Black
Friday some of Saturday and on premise
(by demand) on Cyber Monday. Certainly,
at the beginning of the event focus was very high and everyone was apprehensive
about what might occur and what problems might need to be addressed. Previous year’s Peak had seen all manner of catastrophes
including site outages and various problems.</span></span><br />
<br />
<span style="font-size: large;"><span lang="EN-GB"> </span></span></div>
<div class="MsoNormal">
<span style="font-size: large;"><span lang="EN-GB">The biggest
problem this year, then, from a Payments
perspective, was that a Legacy component on dealing with timeouts with a
Payment Service Provider couldn’t reliably recover and compensate from the timeouts and so Orders would be left
in a state where we didn’t know whether they had been billed. Simply retrying the Orders meant risking a
cancellation occurring as the legacy component is pretty dumb, so a quick
support tool was knocked up and like a scene from Lost, a button was pushed
every so often to make sure these were processed properly. </span></span><br />
<br /></div>
<div class="MsoNormal">
<span style="font-size: large;"><span lang="EN-GB">The highlight
though is that everything, generally, worked.
On Cyber Monday, due to a failure with a voucher service Orders dropped
off a cliff. Fixing the problem and
spreading the word to our persistent and patient customers meant that between
9-10pm an average of 55 orders as second were being processed and all of this
would land at Payments door step. The system coped admirably and a backlog of
messages, of which there were up to about 40k at one point (spread across our
system) were slowly but surely dealt with. </span></span></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<span style="font-size: large;"><b><span lang="EN-GB">Summary</span></b></span></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<span style="font-size: large;"><span lang="EN-GB">The
Payments Platform have now got a lot a data regarding capabilities during spiky
load periods, a greater operational understanding and has breathed a big sigh of relief having
been able to support Black Friday. Thing
is, its going to be even bigger next year and we are just about to start preparations
in updating our Platform to be even more resilient, scalable and
available. Looking forward to it. We will have a lot more control of our
systems come this time next year and a maturity and experience to put what we
have learnt to good use. </span></span><br />
<br />
<span style="font-size: large;"><span lang="EN-GB">Up Next....Reactive Extensions or maybe even HAL (Hypermedia Application Language) </span></span></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<br /></div>
colhttp://www.blogger.com/profile/11580119772205731598noreply@blogger.com0tag:blogger.com,1999:blog-5375025131541210012.post-84424577607681875682016-06-05T14:16:00.002-07:002017-10-31T17:03:23.660-07:00Learning to TDD is like learning to drive a car<span style="font-size: x-large;">Its true (well sort of)</span><br />
<span style="font-size: x-large;"><br /></span><span style="font-size: large;">When I was learning to drive I was pre-occupied with where to put the gear stick, where it actually was, when to use the clutch and so on and so forth and being flustered all the time. </span><br />
<span style="font-size: large;"></span><br />
<span style="font-size: large;">But, quite quickly, with continued practice and with growing familiarity and confidence with a car, I could think about where I wanted to go instead of every detail of what I needed to do, to get there, and now I drive like Lewis Hamilton. </span><br />
<span style="font-size: large;"></span><br />
<span style="font-size: large;">Before I could start to TDD fairly well (In my .NET C# world) and be able to decide what works for me (for example choosing when to use classic and mockist TDD) I had to become comfortable with some pretty basic things including a good refactoring tool - <a href="https://www.jetbrains.com/resharper/">ReSharper</a>, a good test runner and Unit testing framework like <a href="http://nunit.org/">NUnit</a> and a good test isolation (mocking) library, like <a href="https://github.com/Moq/moq4">Moq</a> (all of these tools have their proponents and detractors) but this is what I use for the majority of the TDD I do at the moment, this could change, but the principles won't.</span><br />
<span style="font-size: large;"></span><br />
<span style="font-size: large;">I've been TDD'ing - or at least - trying hard to TDD over the last couple of years and it has been a grind. </span><br />
<span style="font-size: large;"></span><br />
<span style="font-size: large;">I've read <a href="http://www.amazon.com/Test-Driven-Development-Kent-Beck/dp/0321146530">TDD by example</a> by Kent Beck about 10 times (but never to the end) and watched plenty of videos and read blogs and - in spite of being told that practice makes perfect - have not practiced anywhere near enough - I just dont't have the opportunities in work time alone to do this (but that's another story - and that is in spite of being a TDD first organisation). </span><br />
<span style="font-size: large;"></span><br />
<span style="font-size: large;">Taking my lack of practice by the horns after reading Sandro Mancuso's book - <a href="https://www.amazon.co.uk/Software-Craftsman-Professionalism-Pragmatism-Robert/dp/0134052501">The Software Craftsman</a> this coupled with Roy Oshrove's <a href="http://osherove.com/tdd-kata-1/" rel="nofollow">string calculator</a> kata (and many others like it), <a href="http://www.growing-object-oriented-software.com/">Growing Object Oriented Software Guided by Tests </a> (GOOS) by Steve Freeman and Nat Pryce, <a href="http://martinfowler.com/articles/mocksArentStubs.html">Mocks aren't stubs</a> by Fowler and Sandi Metz's take on <a href="https://www.youtube.com/watch?v=URSWYvyc42M">what and what not to test </a></span><br />
<span style="font-size: large;"><span style="font-size: large;">a few things changed and things have, seemingly, <i>finally</i> clicked. </span></span><br />
<span style="font-size: large;">(Edit: 31/10/17 I am reviewin Sandi Metz's TDD eBook, 99 Bottles of OOP and the first impressions are really good) <br /> </span><br />
<br />
<span style="font-size: large;"></span><br />
<span style="font-size: x-large;">TDD Bigger than the sum of its parts</span><br />
<span style="font-size: x-large;"></span><br />
<span style="font-size: large;">Using the tools I mentioned above will get you writing unit test's pretty quickly but not TDD'ing. Using the tools and TDD'ing will get you TDD'ing but most likely in haphazard manner.</span><br />
<span style="font-size: large;"></span><br />
<span style="font-size: large;">Firstly, TDD'ing at all requires confidence in the tools, mentioned above, and not worrying or having to think about how to use them - its not that they are extremely complicated to use, but things can get bloody confusing. Using Moq, we can create Mock objects which are Stubs or we can use Mock objects to verify behavior with them. Of course, this can be done without Moq, but it <i>is great</i> at creating Mocks and Stubs, quickly.</span><br />
<span style="font-size: large;"><br />(Edit 31/10/17 Perhaps I have overstated having to use these tools. Using things like Resharper are really helpful, but not the be all and end all of course. And of course you could write your own Stubs and Mocks - I guess my point was that these things help speed up development somewhat and getting up to speed with them takes a while but also at my company I had to learn these at the same time. In retrospect perhaps it would be better to not use these types of tools AT ALL - when starting out - to <i>really </i>get some of the fundamentals down, especially the really intricate part of slowing down TDD and being very thorough and organised and applying a regimented approach, which I have come to appreciate more as time has gone on - and having read more about - even just recently.) </span><br />
<span style="font-size: large;"></span><br />
<span style="font-size: large;">(Edit 31/10/17 However,... ) Using a good refactoring tool really helps speed up things too, but - again - to use one fairly well there is a bit of a big curve learning useful shortcuts but then once this has been done learning when to pull objects into existence or when they should be Mocked or Stubbed needs to be learnt. </span><br />
<span style="font-size: large;"></span><br />
<span style="font-size: large;">So we have the tools and we need to learn how to use them and when to use them and why to use them. This is a lot to learn and then apply. However, when all of this starts to fall into place, <b>wonderful</b> things start to happen and the productive ebb and flow of TDD, which before, was likely staccato comes to the fore. </span><br />
<span style="font-size: large;"></span><br />
<span style="font-size: large;">With this, and something I forgot to mention, pairing with people when TDD'ing loses the scare factor, worrying about how to use the tools is a bit of a disadvantage and can disrupt the flow of pairing, it can make you feel pretty stupid if someone is already pretty comfortable, even if you are far from it. Nobody likes to appear like a novice - not knowing how to find a gear or the accelerator if comparing to driving. </span><br />
<span style="font-size: large;">(Edit: 31/10/17 Perhaps knowing the tools would be less of a worry if you knew the intent and understood the implications of what people you are pairing with are doing, a tool can be picked up and put down but understanding is a much slower gained but sought after quality) </span><br />
<br />
<span style="font-size: large;">So the upshot of having the requisites in place is that shaping applications, driving the design by pulling in collaborators (either by Mocking or Stubbing them) with a quick refactoring short cut can all be done under the TDD cycle with confidence and very nearly without a thought as to how you are using the tools to get to your end goal - you are just using them as a means to an end. This is a really fulfilling feeling. </span><br />
<span style="font-size: large;"></span><br />
<span style="font-size: x-large;">Summary</span><br />
<br />
<span style="font-size: large;">I am writing about my experiences with TDD as I've struggled with it, and watching people around me get it and not having that same Eureka moment (or moments) has added to my anxiety, stress and worry that I would never be able to TDD effectively. But I think I am just about there, I always like TDD'ing, I now think that I can do it pretty well. And on this note if other people are having trouble with it, stick with it, practice a bit more - read a bit more too, make sure you are reading the right things, some things may likely only make sense when you have some of the other fundamentals boxed off. </span><br />
<span style="font-size: large;"></span><br />
<span style="font-size: large;">Happy TDD'ing. </span><br />
<span style="font-size: x-large;"></span><br />
<span style="font-size: large;"><br /></span> <span style="font-size: large;"><br /></span>colhttp://www.blogger.com/profile/11580119772205731598noreply@blogger.com0tag:blogger.com,1999:blog-5375025131541210012.post-57374491524369122802016-04-25T16:00:00.001-07:002016-04-25T16:00:43.457-07:00Newline character is 1 character<span style="font-size: x-large;">I will never forget the above</span><br />
<span style="font-size: x-large;"><br /></span>
<span style="font-size: large;">When using substring and you can't work out why "\n" is not 2 characters refer back to this title. Does it make me look stupid, Yes, hopefully I will work this out sooner next time. To be fair to myself its like and and. </span><br />
<span style="font-size: large;"><br /></span>
<span style="font-size: large;">That is all. </span><br />
<span style="font-size: large;"><br /></span>
<span style="font-size: large;"><br /></span>colhttp://www.blogger.com/profile/11580119772205731598noreply@blogger.com0tag:blogger.com,1999:blog-5375025131541210012.post-18904071827489716532016-04-19T14:30:00.002-07:002016-04-25T06:36:31.116-07:00A Beautiful Solution?<span style="font-size: x-large;">^^Perhaps^^</span><br />
<br />
<span style="font-size: large;">For various reasons me and my team hadn't been spending a lot of time at the coal face, coding. From spikes, POCs and analysis to release activity - coding opportunities have been few and far between. Thankfully this changed over the last couple of weeks. </span><br />
<br />
<span style="font-size: medium;"><span style="font-size: large;">As part of the Payments Platform we have been given the opportunity to chip away at some of the Legacy processes to provide more flexibility in our Payments offering. Part of this process see's us hoping to consolidate some existing logic to manage the integration with a 3rd party Payment Solution Provider more flexibly. Specifically around handling particular types of payment which are handled differently based on certain criteria such as a customers Billing country, what currency they are paying with and the card type they are using. </span></span><br />
<br />
<span style="font-size: x-large;">Stuck</span><br />
<span style="font-size: large;"><br /></span>
<span style="font-size: large;">In essence the work we are carrying out is a redesign and re-factor of some logic which is spread out in a number of places. Bringing it all in, has been <i>both </i>interesting and frustrating.</span><br />
<br />
<span style="font-size: large;">Maybe its because of the lack of practice but I couldn't quite nail a solution (to a part of the problem) until about a week after starting the work. </span><br />
<span style="font-size: large;"></span><br />
<span style="font-size: large;">I'd started out test first, using TDD to drive out an initial solution, which I was sort of happy with, and then re-factored - went a bit mad - duplicated loads of tests and then had lost the essence of a simple solution. As I wasn't happy, I made a V2 of my work, left all of - what had become sociable tests - in place and started again. I ended up with real clarity, no over-engineering and a solution I am really happy with. </span><br />
<br />
<span style="font-size: large;"><span style="font-size: x-large;">The essence of the problem</span><br />
<strong></strong><br />When<strong> filtering </strong>data <strong>, </strong>there are a number of options available. </span><br />
<span style="font-size: large;"><br /></span><span style="font-size: large;">Typically if the data is stored in SQL (perhaps relationally - but not always) , querying and filtering can <i>easily </i>be performed in a stored procedure (or data layer). Certainly at the cost of having business logic in the DB (usually a pretty poor choice), amongst other trade-offs. </span><br />
<span style="font-size: large;"><br /></span>
<span style="font-size: large;">Conversely a denormalised bit of data stored in a document store or - again - in SQL can be retrieved and then worked on in memory and up in the vernacular of the UL of a domain using LINQ and , giving a nice opportunity to easily test behaviour and nail requirements around a business critical process.<br />
<br />
The existing solution, we are augmenting, used some pretty meaty LINQ queries. Now, arguably for some, this may suffice and do the job, but due to the number of things needed to filter on and various conditions, some pretty "all encompassing" LINQ 'WHERE' statements were being used <strong>with</strong> comments. Coupled with there were no tests for some of the behaviour, some of it seemed coincidental.<br /><br />The comments alone are enough to make you question the clarity of a design and are well documented as a marker of a code smell in a lot of situations (but not all <em>of course</em>).<br />
<br />
Me and my colleagues also noticed there were lots of <strong>null </strong>checks thrown in too. We all thought - we can do better than this! I lobbied for this task and was certain I'd knock something out quickly. I struggled!<br />
<br />
My first approach saw me using the decorator pattern or at least - a <i>bastardised </i>version of it. The basic concept is illustrated below. In fact the approach can be visualised as moving some data through a pipeline of filters, each filter being stateless and all working toward refining some data to a reduced set. I talked about this kind of approach when mentioning folding in an earlier post. This is a functional programming technique.</span><br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjnpaIzJBZWNqn-TCedewtyiJO1WG3V13LogUKSlzuuuKLI_3-mnQ117bWTEnJ2KNLOFub-MJPMQaHPLbDqzNywUxru4NCkEyH_Mo_kV8avDek8ffc29obmzI048g6oiVvzGLjaZSvjSpPK/s1600/StatelessTransform.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="480" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjnpaIzJBZWNqn-TCedewtyiJO1WG3V13LogUKSlzuuuKLI_3-mnQ117bWTEnJ2KNLOFub-MJPMQaHPLbDqzNywUxru4NCkEyH_Mo_kV8avDek8ffc29obmzI048g6oiVvzGLjaZSvjSpPK/s640/StatelessTransform.png" width="640" /></a></div>
<span style="font-size: large;"><br /></span>
<span style="font-size: large;"><br /></span>
<span style="font-size: large;"><br /></span>
<span style="font-size: large;"><br />
In code my first attempt was facilitated by the code below: </span><br />
<span style="font-size: large;"><script src="https://gist.github.com/colindooley11/0e300a6275eb48fbb2ee3cd6b14a618c.js"></script>In the snippet above I have an interface that takes the seed collection (the original query) in the first argument - only the first filter in the chain would work on this.</span><br />
<span style="font-size: large;"><br /></span>
<span style="font-size: large;">The other arguments are values which are needed to drive the filtering, all of this information has to go down so that each filter can use what it needs. This could be simplified by applying <b><a href="http://refactoring.com/catalog/introduceParameterObject.html" rel="nofollow">Extract/Introduce Parameter Object</a></b> to the parameters. </span><br />
<span style="font-size: large;"><br /></span>
<span style="font-size: large;">Subsequent filters just work on the results from previous filters who have their Execute method invoked within a filters Execute method. The other filter being a dependency of course.<br /><br />Only one filter implementation is shown here, the currency filter. Its Execute method dictated by the interface, applies any filtering and returns the result for the next filter to do its thing (if there is one configured).</span><br />
<span style="font-size: large;"><br /></span>
<span style="font-size: large;">Each filter is stateless, in the end after applying a number of filters, an input set is reduced to just one matching value based on various criteria. </span><br />
<br />
<span style="font-size: large;">Setting up a "chain" of filters would look like:</span><br />
<span style="font-size: large;"><br /></span>
<script src="https://gist.github.com/colindooley11/6f79a4fd3609fe120ec158788e7febf2.js"></script>
<br />
<span style="font-size: large;">Now, this worked but after review by my colleague, the fact I'd gone out of my way to encapsulate the filters as separate classes, kind of indicated that the filters should be usable by him as standalone autonomous bits of logic and this is exactly what he wanted to do as he had some similar filtering which only required some of the filters I'd made. Fair enough.</span><br />
<span style="font-size: large;"><br /></span>
<span style="font-size: large;">With a few tweaks this would work. The problem was that with the dependency of the filter up in the constructor without some kind of null object pattern or specific handling to check for nulls or handle setting up the chain of filters, things would break if composed in the wrong way. That is, for example, if he wanted to use a filter but didn't want to chain a specific child filter.</span><span style="font-size: large;"></span><br />
<span style="font-size: large;"></span><br />
<span style="font-size: large;">As well as a set of scenario driven tests for the domain service, I'd written tests for each filter and in so doing duplicated lots of test. I'd not quite implemented what I desired.</span><br />
<span style="font-size: large;"></span><br />
<span style="font-size: large;">I started looking at Haskell list comprehensions and thinking about a functional way to tackle the problem and came across a post on S/O which was exactly what I was looking for. In essence I could use a <strong><a href="http://learnyouahaskell.com/higher-order-functions#folds" rel="nofollow">left-fold</a></strong> and have my filters called one after another with a LINQ extension method called Aggregate which - like my solution - takes a seed but - unlike my solution - handles 'accumulating' or keeping track of the result of applying filters without the need to hand-crank any other classes.
Composition looks like below. A list of methods are used to impose filtering with the heavy lifting done by a bit of LINQ as we can see below in the next snippet.</span><br />
<span style="font-size: large;"></span><script src="https://gist.github.com/colindooley11/a6ed717f2f3cbd5ad6ad067639256b3b.js"></script><span style="font-size: large;"><br />Below, the Aggregate function is called which takes a seed (our initial data to be filtered - number 1 in the diagram above), a lambda (or anonymous function) representing an accumulator method, which matches the signature of the methods in the filters list (numbers 2 and 3 in the diagram above).</span><br />
<br />
<script src="https://gist.github.com/colindooley11/0a52834f42920a205f4a30e3ea2a3400.js"></script>
<span style="font-size: large;"><br /></span>
<span style="font-size: large;">It then applies each of the methods - composed above - in turn. Running each filter and passing its result, via the accumulator to the next filter, eventually pushing out a much more <i>refined </i>result adhering to the nip and tuck that the filters imposed along the way. </span><br />
<span style="font-size: large;"><br /></span>
<span style="font-size: x-large;"><b>Summary</b></span><br />
<span style="font-size: x-large;"><br /></span>
<span style="font-size: large;">I really like the elegance of this solution. Yes it is simple (you may wonder why so many lines were devoted to something which may not have even been given a second glance on what you are working on) and there are yet simpler ways to tackle our problem, the filters could just be looped over for example (or even revert to clunkier LINQ queries). It just struck me as, erm, nice. You may well disagree. If so, please let me know in the meantime I might do a V3 and just filter down in the persistence layer :)</span><br />
<span style="font-size: large;">
<br />
</span><br />
<span style="font-size: large;"></span><br />
<span style="font-size: large;"><br /></span>
<br />
<br />
<br />
<br />
<br />colhttp://www.blogger.com/profile/11580119772205731598noreply@blogger.com1tag:blogger.com,1999:blog-5375025131541210012.post-4217868104773308622016-03-31T08:07:00.001-07:002016-04-12T13:04:22.035-07:00Book Review: The Software Craftsman: Professionalism, Pragmatism, Pride <span style="font-size: large;"><b>Spur</b></span><br />
<span style="font-size: large;"><b><br /></b></span>
<span style="font-size: large;">There were a few things that made me read this: </span><br />
<ul>
<li><span style="font-size: large;">I saw it lying on my dev managers desk</span></li>
<li><span style="font-size: large;">We have a coach from the company that the author co-founded in our place at the moment</span></li>
<li><span style="font-size: large;">I was cynical about the software craftsman idea and thought it was about self-serving, elitist and pumped up self publicists</span></li>
</ul>
<div>
<span style="font-size: large;"><b>Inspirational</b></span><br />
<span style="font-size: large;"><b> </b></span><span style="font-size: large;"><b><br /></b></span></div>
<div>
<span style="font-size: large;">I'm not a soppy or effusive person, but I have to say this book really resonated with me and, like a really good rally call, made me feel positive about my current position and how to sustain and develop my career further and just how I might achieve this. <i>The thing is</i>, I knew most of this already but just hadn't bundled the virtues, thoughts and values that are described, under a particular banner. </span><span style="font-size: large;"><b><br /></b></span><br />
<span style="font-size: large;"><br /></span></div>
<div>
<span style="font-size: large;"><b>The Software Craftsman Manifesto</b></span><span style="font-size: large;"><b><br /></b></span><br />
<span style="font-size: large;"><b><br /></b></span></div>
<div>
<span style="font-size: large;"><a href="http://manifesto.softwarecraftsmanship.org/#/en">It actually exists!</a> A set of values to which a software craftsman can hang their hat. Now, like the author says, there are people who don't like to call themselves craftsman and I am one of them, but the ideas of professionalism, adding value and trying to do a good job are things I already embody and value in my peers. Its good to see that in the UK there is a large community of people who are interested in the same things, the <a href="http://www.meetup.com/london-software-craftsmanship/">LSCC</a> being one of these groups. I need to get to one of these groups locally, any of these in Brum? </span><span style="font-size: large;"><br /></span><br />
<span style="font-size: large;"><br /></span></div>
<div>
<span style="font-size: large;"><b>Not about TDD or XP</b></span></div>
<div>
<br /></div>
<div>
<span style="font-size: large;">The book isnt about TDD or XP necessarily but they are advocated as, currently, the best ways of helping produce quality and valuable software. In fact it is mentioned an awful lot - but you will need to buy a different book for advice on specific technical practices. </span></div>
<div>
<span style="font-size: large;"><br /></span><span style="font-size: large;"><b>What to do</b></span></div>
<strong><span style="font-size: large;"></span></strong><br />
<div>
<span style="font-size: large;">I, like the author, often hear that people say that the company they are working at is not investing in them and propelling them forward in their career. Sandro pulls no punches in saying that this responsibility is left squarely at your own feet, which I completely agree with. Professionals in other industries have to stay current and continue a life long journey of mastery why should devs be any different. </span><br />
<span style="font-size: large;"></span><br />
<span style="font-size: large;">From reading books to attending meetups to pet projects to blogging, a number of strategies are discussed to help you <i>help yourself. </i></span><br />
<span style="font-size: large;"></span><br />
<span style="font-size: large;">This is probably obvious to most people, but doing it is different from talking or knowing about what you need to do. In the first part of my career - after leaving Uni - I didn't invest enough time in learning new stuff and keeping up to date with what is current - its a lot easier now and I'm lazy - but I'd be further ahead if I'd maintained my out of work learning after Uni. Don't get me wrong, I was still working hard and wasn't a 9 to 5'er but my working life may have been a bit easier if I'd supplemented it with stuff I'd learnt so that I could lead and not just <em>be lead</em>. </span></div>
<div>
<span style="font-size: large;"><br /></span><span style="font-size: large;"><b>Identifying your peers</b></span><br />
<span style="font-size: large;"><b><br /></b></span>
</div>
<div>
<span style="font-size: large;">There's a great section on the different types of personality traits of people you will work with from the "fanboy" to the "irrational" to the "cynic". Discussing how to deal with different people and knowing that other people have identified them, certainly makes you realise that you weren't the only frustrated person. There is some great advice about being humble and open to ideas, which is a quality some peers lack and hopefully when they read his hallowed words they will reflect and be a bit more open. Also "fanyboys'" should take stock and just "chill the fuck out". Yes, I'm sure the tech you learnt about last night is great I just don't want it rammed down my throat - again....</span><span style="font-size: large;"><b><br /></b></span><br />
<span style="font-size: large;"><br /></span></div>
<div>
<span style="font-size: large;"><b>Interviewing/Being interviewed/Career choice</b></span><span style="font-size: large;"><b><br /></b></span><br />
<span style="font-size: large;"><b><br /></b></span></div>
<div>
<span style="font-size: large;">Yeah , tips on interviewing may seem a bit out of place but good advice on interviewing your prospective employers is indispensable. Also, advice on owning your career and knowing what you want and when you want it vindicates some of the choices you have had to make where more money was necessary or opting for a slightly less hectic pace when building a family. </span><span style="font-size: large;"><br /></span></div>
<div>
<span style="font-size: large;"><b><br /></b></span>
<span style="font-size: large;"><b>Summary</b></span></div>
<div>
<br /></div>
<div>
<span style="font-size: large;">I'm certain I've missed some pivotal parts of the book out of the review, my short term memory is terrible (and I promise to pay them their dues next time I pick up the book) but I am left in no doubt that this is an excellent book, if you are disillusioned it may reignite you hunger, if you are looking at what you need to do to help you progress, it has sage advice. I'd almost label it a developers self-help book, I certainly wouldn't have bought it, if it had a self-help title but it makes me think I might read a few more of these if it gives you a spring in your step. Or is that just because its Spring? Enjoy. </span></div>
<div>
<span style="font-size: large;"><br /></span></div>
<div>
<br /></div>
<div>
<br /></div>
colhttp://www.blogger.com/profile/11580119772205731598noreply@blogger.com0tag:blogger.com,1999:blog-5375025131541210012.post-68410557380145053442016-03-30T07:05:00.002-07:002016-03-31T12:42:21.728-07:00Book Study: Patterns-Principles and Practices of Domain Driven Design<h2>
<span style="font-size: x-large;">Book Study: As title</span></h2>
<h2>
Summary</h2>
<span style="font-size: large;">In summary this is an <b>excellent </b>book. It is really accessible <i>but </i>some of the more advanced areas will need further reading elsewhere. For example, things like Aggregate design </span><br />
<br />
<span style="font-size: large;">If you are involved in Enterprise application development and you haven't yet been able to take advantage of the Domain Driven Design (DDD) philosophy, this book will show you the way. I know this because it is almost like a narrative to my experiences of the philosophy at a large e-commerce website in terms of some of the problems experienced, practices adopted, conversations with the business and stakeholders, patterns used, misused, abused and not used and every conversation about everything in DDD in-between. </span><br />
<h2>
Go Compare</h2>
<div>
<span style="font-size: large;">I previously read the fist 3 chapters of Vaughn Vernon's Implementing DDD book and I ran out of steam for various reasons (the birth of my second son primary amongst them). Its not that it isn't a good book (I refer to it later on and why I think this is the case) but it felt a little bit formal and dry on occasion and I'm sorry to say the cowboy logic comic strips both confused me and grated with me, maybe you have to be an American to understand the jokes. </span><br />
<span style="font-size: large;"><br /></span>
<span style="font-size: large;">Keen to still learn further about DDD and, on the recommendation of a friend, I'd picked up Patterns Principles and Practices of Domain Driven Design (PPPDDD - I know!!!) after seeing an example of using Atom Feeds and HTTP to provide a reactive messaging solution as an alternative to using a message bus. This was a great antidote to having a particular service bus, which will rename nameless, rammed down my throat for the last 2 years as <strong>the </strong>way to "do asynchronous" messaging. </span><br />
<br />
<span style="font-size: large;">The language and concrete examples in the section seemed a lot more accessible than some of the stuff in Vaughn's book and so I decided to study it from cover to cover - doing as much of the code examples from scratch reading all of the stuff about strategic patterns and drawing out designs, before moving on to the less subjective tactical patterns and other patterns this book has to offer an opinion on. </span><br />
<br />
<span style="font-size: large;">What followed was an epic 6 month journey where I ended up cross referencing the Vaughn's book, blogs and everything in between to try and get a better handle on DDD. </span><br />
<br />
<span style="font-size: large;">After all this, I am only <strong><em>bit </em></strong>wiser about DDD, although I can now hold my own in a conversation on the subject and suggest a few patterns, decomposition techniques and identify domains, sub-domains and bounded contexts, it is clear that it will take me many years and experience of certain situations to be truly conversant and comfortable with DDD, and this book is only going to get you some of the way, actual real world experience is going to have to back up what is learnt in this book, otherwise you will lose what you have learnt. </span><br />
<span style="font-size: large;"></span><br />
<span style="font-size: large;">This might be ok, you might never need to do context-mapping at your current job with all of the seams and domains and sub-domains identified already, but having a deeper insight into what affects this kind of arrangement, as well as many other aspects, are key to having ammunition to engage with the business and help drive solutions to new problems that emerge in your business's domain. </span></div>
<h2>
The Book</h2>
<span style="font-size: large;">There is so much stuff in this book I started writing notes and actually ended up taking about 100 pages of notes as well as creating about 5 solutions and posting <strong>gists</strong> of useful code examples to pinch for future use, I started this post the same day I wrote my post about Twitter in October last year. It's taken me an age. </span><br />
<br />
<span style="font-size: large;">There are 4 main parts covering:</span><br />
<span style="font-size: x-large;"></span><br />
<ul><span style="font-size: x-large;">
<li><span style="font-size: large;">Part 1 - The Principles and Practices of Domain Driven Design</span></li>
<li><span style="font-size: large;">Part 2 - Strategic Patterns: Communicating between bounded contexts</span></li>
<li><span style="font-size: large;">Part 3 - Tactical Patterns: Creating effective domain models</span></li>
<li><span style="font-size: large;">Part 4 - Design patterns for effective applications</span></li>
</span></ul>
<span style="font-size: x-large;">
<span style="font-size: large;">I will discuss a few of the highlights of the parts, I will not go into too much detail, but because of this I will be doing some of the book a disservice, so if reading this leaves you asking questions (which it probably should) then buy the book! </span></span><br />
<h2>
Part 1 - The Principles and Practices of Domain Driven Design</h2>
<span style="font-size: x-large;"><span style="font-size: large;">The main thrust of this part is about DDD as a philosophy, what it is and isn't and that it isn't just about tactical patterns and when and where DDD should and shouldn't be applied. </span><br />
<span style="font-size: large;"><br />This part <b>clearly spells </b>out what things like the Ubiquitous language is, how to decompose complex problems into more manageable ones, getting buy-in from the business, collaboration, context-mapping and what a bounded context is, to name but a few things. The very essence of what DDD is, is covered and you wont be left scratching your head after. </span></span><br />
<span style="font-size: large;"></span><br />
<span style="font-size: x-large;">
<span style="font-size: large;">This part of the book is very insightful and as a developer I was initially tempted to skip past large parts of this part, but doing this - it quickly became clear - would mean I would be missing the point of DDD. I would recommend you do the same if you want to start to grasp it properly. </span><br />
<span style="font-size: large;">There is so much which resonates in this chapter in terms of what engaging with stakeholders is like, advice on managing the expectations and of stakeholders and advice on stopping stakeholders jumping to solutions and keeping them focused on the problems in the domain and in the language of the domain and things like mapping a problem to a model of the problem. The break down of what a core domain, domain, sub domain and a bounded context are, is <b>indispensable </b>too. I personally found decomposition hard to understand before I read this, and the chapters guide you through decomposition in a very clear manner.</span></span><br />
<span style="font-size: x-large;"><span style="font-size: large;">Suffice it's to say, I don't agree with everything here though, some of the bounded context modeling is not how I would model bounded contexts necessarily, as some of the naming relies on the domain messages they receive, but the decomposition aspect of this exercise is the main point here. </span><br />
<br />
<span style="font-size: large;">There is lots of ground to cover and, if I am being honest, I would have to refer back to my notes (or the book) to help me remember large parts of this section in detail and here in lies a problem with a book of this magnitude, its really not possible to retain all of its contents to a high level of detail and I'd recommend referring to it as a reference depending on your needs in a project.</span></span><br />
<h2>
Part 2 - Strategic Patterns: Communicating between bounded contexts </h2>
<span style="font-size: x-large;"><span style="font-size: large;">As a developer the real fun starts here, after a lot of scene setting and ideological imparting in the first part, this part focuses on some examples of getting bits (bounded contexts) of a domain (or other sub-domains) to talk to each other. Using NServiceBus, EventStore and Atom Feeds for various types of communication between bounded contexts, the examples are clear and really help bring to life what and how reactive messaging and RPC can be used to create solutions that communicate effectively. Reading this you may wonder why RPC might even be a consideration these days, but it was an eye-opener to see that this should still be a worthy consideration. If anything, this book helps you keep an open mind and you can practice your "it depends" come back when talking to colleagues about potential solutions to problems.</span> </span><br />
<br />
<span style="font-size: large;">A small gripe with this section is the fact that HAL (Hypermedia Application Language) is introduced as a basis for the messaging with atom feed event examples without a real good reason as to why it is introduced or used. Don't get me wrong it seems like a useful implementation of HATEOS but probably wasn't necessary for the examples. </span><span style="font-size: x-large;"><br /></span><br />
<h2>
Part 3 - Tactical Patterns: Creating Effective Domain Models</h2>
<div>
<span style="font-size: large;">This was my favourite part of the book, as a developer, it has stuff that I can use in my day to day and indeed some of the stuff I learnt here is out in the real world shortly to be used by millions of customers (albeit a small thing, but nevertheless) . The examples around value types are great and although I had an understanding of what they were - again - I was left with no ambiguity in my understanding. Likewise the clear distinction between domain services and application services is excellent. These two concepts were a blur to me before but now stand distinguished - side by side. Event sourcing is described in detail and is a good as guide as any I have seen. The bit I was looking forward to the most was the chapters on Aggregates. These are one of the fundamental tactical tenets of DDD and they are given thorough treatment here. From what an invariant is to determining aggregate boundaries and considerations on when and where to use immediate and eventual consistency. I must admit though that I cross referenced Vaughn's book here because he does provide more insight and consideration. It is worth saying that when I come to design aggregates I will refer back to both books to give me as much chance of modeling them correctly as possible. </span><span style="font-size: large;"><br /></span></div>
<div>
<h2>
Part 4 : Design Patterns for Effective Applications</h2>
</div>
<div>
<span style="font-size: large;">The stand out chapters of this section is the treatment of CQRS. Again, an area I am pretty au-fait with but this left me with no doubt about its applicability in certain situations and it left me crying out loud as to where we could have adopted it in some of the work we applied DDD to at work recently. We didn't go the whole hog in generating a simple read model and got a little confused between our write and read concerns </span><span style="font-size: large;"><br /></span></div>
<div>
<span style="font-size: large;">Application and domain services are worked through well with good examples. However, one of my major gripes with some of the application</span><span style="font-size: large;"> service examples is the seemingly poor choice of aggregate design to show application service co-ordination, which was bit of a shame. The example involved a customer referral example and saw an application service updating 2 separate aggregates in the same transaction which is generally advised against (but not always) in chapters in the previous part of the book. I can see that the authors chose to really highlight co-ordination here but I think a more expansive example of eventual consistency and immediate consistency would have really helped and not meant I had to go looking in Vaughn's book. </span><span style="font-size: large;"><br /></span></div>
<div>
<span style="font-size: large;">There are also chapters on reporting, by this time I was ready to put down the book but managed to see it through. </span></div>
<div>
<br /></div>
<div>
<span style="font-size: large;"><b>Summary</b></span><span style="font-size: large;"><b><br /></b></span></div>
<div>
<span style="font-size: large;">If you want an accessible book and want to have a good grounding with DDD as a philosophy I would highly recommend this book. </span><span style="font-size: large;"><br /></span></div>
<div>
<span style="font-size: large;">As far as I know all major angles of DDD are covered from both a design and implementation point of view, with a nice collection of current and relevant technologies covered. The book comes more from a practitioner point of view and is less formal in it description of the various aspects of DDD as compared to say the blue or red book (Evans and Vaughn's books respectively) the authors don't use the <b>this/that pattern(54) </b>calling out of patterns etc and some may consider it a little bit less high brow than the other books. On this note some of the more advanced things like Aggregates could probably do with more examples, as with all of these books real world examples are thin on the ground. The contrived example of SaaSOvation in Vaughn's book although obviously contrived doesn't easily map well to some of the problems found in more e-commerce web applications, but there is enough there as food for thought. If this post seems a bit weary, it is, I could barely muster the energy to regurgitate what it is I learnt after a long haul. I think I'll have to revisit this post when I revise some of the material to give the book the justice it deserves. </span><span style="font-size: large;"><br /></span></div>
<div>
<span style="font-size: large;">In contrast to this marathon study, I read the Software Craftsman by Sandro Mancuso in 2 days after this, and felt inspired and pumped this will be the subject of my next post. As one of my famous colleagues would say "<b>I better go and do some work</b>" </span><span style="font-size: x-large;">
</span></div>
colhttp://www.blogger.com/profile/11580119772205731598noreply@blogger.com0tag:blogger.com,1999:blog-5375025131541210012.post-78346852557292523562015-10-24T12:54:00.003-07:002015-10-24T12:58:20.556-07:00Twilosophy\Twitosophy<h2>
<span style="font-size: x-large;">Observations of the space at present. </span></h2>
<div>
<span style="font-size: x-small;">Tongue in cheek of course</span></div>
<h2>
<span style="font-size: x-large;">Twilosophy\Twitosophy.</span></h2>
<span style="font-size: large;">I don't usually offer much opinion in what I do. I sit on the fence alot, maybe I shouldn't . However, when I do, I try to base what I know on what I really know, not on what other people tell me I need to know. I'm sure what I am observing has been commented on before, and maybe other people agree or not, but reading a post on Twitter and then using it as your own keen insight or shallowly cited insight is more of an indication of your lack of ability instead of just keeping schtum or just saying "I dont know", where more respect would probably be gained. </span><br />
<span style="font-size: large;"><br /></span>
<span style="font-size: large;">The above practice seems to be rife in software development at the moment. I'd be a hypocrite if I said I hadn't fallen into the trap, but - like a recovering alcoholic - I have realised the errors of my ways and I am standing up and saying my name is Colin and I am a twitosopher (at times) and shallow citer (at times). </span><br />
<br />
<h2>
<span style="font-size: x-large;">Shallow Citing. (Shiting?) </span></h2>
<div>
<span style="font-size: large;">The practice of citing something when you don't really have the full picture of what it is you are citing and/or name dropping at the same time . I have only really noticed this recently, maybe its a larger organisation based thing which has pervaded the rest of the industry or just that people are more aspirational or deluded in the places I have worked recently. </span></div>
<div>
<span style="font-size: large;"><br /></span></div>
<div>
<span style="font-size: large;">I work in enterprise development and the most cited things are the testing triangle and Fowler (as in Martin Fowler). The Shiter often uses a cite to gain the upper hand in an argument. I mean just what is there be said if someone who is the Oracle on a subject has said it, what can I - as mere mortal - offer in the way of comeback or alternative. It can be the argument/debate ender - I'd encourage it not to be and challenge the citation for all its worth. People have with me and have seen me coming up short. </span></div>
<div>
<span style="font-size: large;"><br /></span></div>
<div>
<span style="font-size: large;">Is this the same where everyone works? Don't get me wrong, things are often correctly and relevantly cited but, dig a little deeper , and asking the person just what it is they are referring to can often prompt some hand-wringing awkwardness.</span></div>
<div>
<span style="font-size: large;"><br /></span></div>
<div>
<span style="font-size: large;">Again, I have been guilty of this and have been caught out red-handed one more than one occasion. I felt embarrassed and felt that there's an element of charlatanism at play, and I came to the conclusion that unless I can confidently back up what I am citing, then don't cite it - unless a disclaimer of some sort is provided. </span></div>
<div>
<span style="font-size: large;"><br /></span></div>
<div>
<span style="font-size: large;">So for all the pseudo intellectuals out there, most people have got your number, but <i>it is</i> entertaining to watch you in all of your pumped-up, self-important glory, so carry on shiting. </span></div>
<div>
<br />
<br />
<br />
<br />
<br />
<span style="font-size: large;"><br /></span><span style="font-size: large;"><br /></span><span style="font-size: large;"><br /></span><span style="font-size: large;"><br /></span><span style="font-size: large;"><br /></span></div>
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<div style="-webkit-text-stroke-width: 0px; color: black; font-family: 'Times New Roman'; font-size: medium; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; margin: 0px; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px;">
</div>
<br />
<div style="-webkit-text-stroke-width: 0px; color: black; font-family: 'Times New Roman'; font-size: medium; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; margin: 0px; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 1; word-spacing: 0px;">
<br /></div>
<h2 style="-webkit-text-stroke-width: 0px; color: black; font-family: 'Times New Roman'; font-style: normal; font-variant: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 1; word-spacing: 0px;">
</h2>
colhttp://www.blogger.com/profile/11580119772205731598noreply@blogger.com0tag:blogger.com,1999:blog-5375025131541210012.post-55303049427014556042015-08-31T15:12:00.003-07:002015-08-31T15:22:53.188-07:00Sql Static Code Analysis (revisited) <h2>
<span style="font-size: x-large;">Static Code Analysis.</span></h2>
<span style="font-size: large;">You may remember my opening gambit in a post about SSDT code analysis talking about coding standards back in May 2014 (but most probably not) It went something like....</span><br />
<span style="font-size: large;">Coding standards (and more latterly styles) in any organisation of any size are liable to incite religious wars and makes zealots out of even the most considered or apathetic of us. Inconsistency and subjectivity, lack of enforcement and heavy handedness all contribute to the contempt often levelled at them... blah blah</span><br />
<br />
<h2>
<span style="font-size: x-large;">Contributing to solving the problem.</span></h2>
<div>
<span style="font-size: large;">As part of one of our projects we had some funding to invest some time and dev into SQL static code analysis, after researching what was available we had the green light to incorporate analysis written to leverage the SSDT code analysis framework into our own CI/CD ALM framework. The appetite for this waned somewhat with the DB community having their own take on what they wanted to use, but the code lives on and some small bits of work continue to be done on it, when possible. </span><br />
<span style="font-size: large;"><br /></span>
<span style="font-size: large;">Anyway, permission was given to publish our efforts. I wasn't the sole contributor but continued/continue to run with what we had, looking to match , at least, the capabilities of other offerings in the arena but also crucially trying to ensure quality- by providing what amounts to integration testing - and some time-saving features to developing code analysis rules. The source code for all of this work is now published on GIT (with permission),but can be used as a basis for your own SQL static code analysis needs. </span><br />
<span style="font-size: large;"><br /></span>
<span style="font-size: large;">It incorporates the work we did as well as the adoption and adaptation of the testing framework suggested by the SSDT code analysis team themselves. </span><br />
<span style="font-size: large;"><br /></span>
<span style="font-size: large;">The code lives in a Visual Studio solution (initially written in VS2012) and is written in C#, it was written against SSDT March 2014 however there are subsequent updates to SSDT currently at <a href="http://blogs.msdn.com/b/ssdt/archive/2015/06/24/ssdt-june-2015-preview.aspx">SSDT June 2015</a></span><br />
<span style="font-size: large;"><br /></span>
<span style="font-size: large;">You can find it here: </span><a href="https://github.com/colindooley11/SSDT-Code-Analysis" style="font-size: x-large;">Git repo</a><br />
<br />
<span style="font-size: large;">Hopefully it might provide you with a start to your own custom analysis needs. </span><br />
<br />
<br /><br /><br />
<br />
<br />
<br />
<br />
<br />
<span style="font-size: large;"><br /></span>
<span style="font-size: large;"><br /></span>
<span style="font-size: large;"><br /></span>
<span style="font-size: large;"><br /></span>
<span style="font-size: large;"><br /></span></div>
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<h2>
</h2>
<br />
<div style="-webkit-text-stroke-width: 0px; color: black; font-family: 'Times New Roman'; font-size: medium; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; margin: 0px; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px;">
<br /></div>
colhttp://www.blogger.com/profile/11580119772205731598noreply@blogger.com0tag:blogger.com,1999:blog-5375025131541210012.post-35069969149762275202015-03-11T08:30:00.002-07:002016-12-06T14:30:08.902-08:00Async/Await Revisited <h2>
</h2>
<span style="font-size: large;">I put up an article back in November '14 about using async/await for a toy project I had created. I have now had a chance to use it for "real work" and, as a result, have had a chance to research the language feature in more depth and have come up with a list of considerations, resources, gotchas and tid-bits. </span><span style="font-size: large;"> </span><br />
<br />
<h2>
<span style="font-size: x-large;">Must Reads</span></h2>
<br />
<span style="font-size: large;">There is a lot of literature available for this feature and plenty of MSDN articles and Stack Overflow related help, however, I'd highly recommend the following books for greater insight into </span><br />
<span style="font-size: large;">TAP, TPL and async/await: </span><br />
<ul>
<li><a href="http://www.amazon.co.uk/CLR-via-C-Developer-Reference/dp/0735667454" target="_blank"><span style="font-size: large;">CLR via C#</span></a><span style="font-size: large;"> (Fourth edition for async/await content), Jeffrey Richter</span></li>
<li><a href="http://www.amazon.co.uk/Concurrency-C-Cookbook-Stephen-Cleary/dp/1449367569/ref=sr_1_1?s=books&ie=UTF8&qid=1426083535&sr=1-1&keywords=concurrency+in+c+cookbook" target="_blank"><span style="font-size: large;">Concurrency in C# Cookbook,</span></a><span style="font-size: large;"> Stephen Cleary</span></li>
<li><a href="http://www.amazon.co.uk/Async-C-5-0-Alex-Davies/dp/1449337163/ref=sr_1_1?s=books&ie=UTF8&qid=1426083569&sr=1-1&keywords=async+in+c+5.0" target="_blank"><span style="font-size: large;">Async in C# 5.0</span></a><span style="font-size: large;">, Alex Davies</span></li>
</ul>
<span style="font-size: large;">Blogs by the latter two also cover this feature in great detail. </span><br />
<span style="font-size: large;"></span><br />
<span style="font-size: large;">They all overlap in their coverage on async/await. As an aside, the first book I would recommend to any serious C# developer anyway, I'd say its on a par with the "C Programming language by Kernighan and Ritchie" which is a book I should probably buy again for posterity (something to show the grandchildren, when they are programming using some super tools)</span><span style="font-size: large;"> </span><br />
<br />
<h2>
<span style="font-size: x-large;">Considerations in an ASP.NET application</span> </h2>
<span style="font-size: large;"></span><br />
<span style="font-size: large;">Using async/await in an ASP.NET application (WebAPI, MVC and Webforms) is slightly different to using it in a WinForms or WPF application. </span><br />
<br />
<span style="font-size: large;">As I am sure you are aware, the UI thread is the main thread in desktop GUI applications. If async/await is used and the UI needs to be updated in a continuation after awaiting, then its is necessary to ensure that the UI thread is the one which is posted back to, in order to ensure updates are made on the UI thread and there are no cross threading issues. </span><br />
<br />
<span style="font-size: large;">Thankfully, this is handled by default because of the use of a SynchornisationContext. This has the UI thread associated with it and allows work, from worker threads, to be posted back to the UI thread. This incurs performance costs, both with the context switching of threads and the posting of work to the UI thread, but this is inescapable, as the UI thread is the only one which can update the UI! </span><br />
<br />
<span style="font-size: large;">In an ASP.NET the application threading model is rather different. </span><br />
<span style="font-size: large;"><br /></span>
<span style="font-size: large;">The thread used for an initial request to an ASP.NET application will - likely - not be the same thread that is used for doing the work in a continuation after an await. </span><br />
<span style="font-size: large;"></span><br />
<span style="font-size: large;">Because of this there is no requirement to use the SynchronisationContext, unless contextual information such as HttpContext or other information passed down the way on the initial request, is needed. </span><br />
<span style="font-size: large;"><br /></span>
<span style="font-size: large;">The recommendation, therefore, is to use a method called ConfigureAwait - passing false - whenever it is deemed unnecessary to have any of the context available from the initial request. (other contexts such as the execution context are still flowed - which has security settings etc). </span><br />
<br />
<script src="https://gist.github.com/colindooley11/4bd0176b86cc92c0090d.js" type="text/javascript"></script>
<span style="font-size: large;"></span>
<span style="font-size: large;">In the snippet above we have a controller operation set up to work in an async manner. The awaitable is configured not to use the SynchornisationContext. This has smaller performance benefits in an ASP.NET application but is still a consideration to be aware of. </span><br />
<span style="font-size: large;"></span><br />
<span style="font-size: large;">There is a gotcha I encountered in upgrading a WebAPI application from .NET 4 to .NET 4.5 (as opposed to creating a .NET 4.5 app from the off) concerning the use of the LegacyAspNetSynchronisationContext. Ensure that if upgrading an ASP.NET application the following entry is added to the web.config (it is added by default if you create a new .NET 4.5 application) </span><br />
<br />
<script src="https://gist.github.com/colindooley11/49eac4cdb35a0363cfe1.js" type="text/javascript"></script>
<span style="font-size: large;">or </span><br />
<br />
<script src="https://gist.github.com/colindooley11/fc91614ec555bb457de4.js" type="text/javascript"></script>
<span style="font-size: large;"></span>
<span style="font-size: large;">Without it I had to use ConfigureAwait(false) everywhere to stop use of the aforementioned context, otherwise the application would fall over when trying to use it, I suspect because the thread initially associated with the context was no longer available.</span><br />
<br />
<h2>
<span style="font-size: x-large;">Cost of async/await</span></h2>
<br />
<span style="font-size: large;">Given all of the machinery produced as part of providing the async/await capability - the state machine - there are costs to bear in mind. Some of the profiling detailed in "Async in C# 5.0", showed that a call to an async method can be up to 10 times slower, even when no asynchronous work is performed. </span><br />
<span style="font-size: large;"><br /></span>
<span style="font-size: large;">So, if asynchronous work is not needed or is only needed in edge cases, paying this price is worth more than cursory consideration. </span><br />
<br />
<h2>
<span style="font-size: x-large;">Interoperability with legacy asynchronous models.</span> </h2>
<br />
<span style="font-size: large;">Interacting with older asynchronous programming models is possible using a number of approaches.</span><br />
<br />
<span style="font-size: large;"></span>
<span style="font-size: large;">A helper method is available to create Tasks (and then await them) from the older APM model.(asynchronous programming model - BeginXXX, EndXXX methods and IAsyncResult) </span><br />
<span style="font-size: large;"><br /></span>
<span style="font-size: large;">The method is called </span><span style="font-size: large;"><a href="https://msdn.microsoft.com/en-us/library/system.threading.tasks.taskfactory.fromasync%28v=vs.110%29.aspx" target="_blank">"FromAsync"</a>.</span><br />
<span style="font-size: large;"><br /></span>
<span style="font-size: large;">"FromAsync" takes the BeginXXX and EndXXX methods of the older APM style model and returns a Task/Task<T>, with the Task containing the result when EndXXX completes. </span><br />
<br />
<span style="font-size: large;"></span>
<span style="font-size: large;">Alternately the EAP model can be interacted with by providing a "puppet task" by using a </span><a href="https://msdn.microsoft.com/en-us/library/gg145045(v=vs.110).aspx" target="_blank"><span style="font-size: large;">TaskCompletionSource</span></a><span style="font-size: large;"> and hooking up the original async EAP method and callback event to properties on this class. The TaskCompletionSource also supports co-operative cancellation allowing a cancellation token to be passed in to control the execution of the operation being wrapped.</span><br />
<span style="font-size: large;"><br /></span>
<span style="font-size: large;">The TaskCompletionSource provides a wrapper to the older EAP methods and, when wired up correctly, will respond to the callback event when async work is completed and transition the state of the wrapped Task<T> and provide the result, if any. If there are any exceptions, or cancellations, the state and result can be set for these too. </span><br />
<span style="font-size: large;"></span><br />
<span style="font-size: large;">Once created, the Task<T> on a TaskCompletionSource can be awaited using async/await, thereby hiding consumers from the EAP or APM completely, which helps consistency and readability of code upstream. </span><br />
<br />
<h2>
<span style="font-size: x-large;">Unit Testing</span> </h2>
<span style="font-size: large;">Unit testing async/await is pretty straightforward. As far as I am aware the latest versions of NUnit, MsTest and XUnit all support the ability to execute an async bit of code without problem. (more on NUnit in a bit). As well as this the latest version of Moq has some nice extension methods to help out here too. In my limited reading on unit testing support for async/await it is generally facilitated by having a some kind of context and related SynchronisationContext in order to ensure that methods can be awaited correctly.</span><br />
<br />
<span style="font-size: large;">Stephen Cleary says that it might be worthwhile testing synchronously and asynchronously as well as testing that exceptions thrown in an async method are appropriately handled. </span><br />
<span style="font-size: large;"></span><br />
<span style="font-size: large;">In the test below, we utilise the ReturnsAsync extension that Moq provides instead of writing </span><br />
<span style="font-size: large;">".Returns(Task.FromResult(expected))"</span><br />
<br />
<script src="https://gist.github.com/colindooley11/5ba039db051376e6d6ce.js" type="text/javascript"></script>
<span style="font-size: large;">Worthy of note here and a potential future gotcha lies within the signature of the method above which is, "public async void when...."</span><br />
<span style="font-size: large;"></span><br />
<span style="font-size: large;">Apparently in NUnit 3.0 (although I am using the 3.x Alpha above without problem) support for asnyc void will die and the method above would continue executing synchronously after the await (instead of waiting if we had introduced a delay) and potentially appear to have passed with an exception or failure rearing its head on another thread at some point in the future, causing unpredictable results. So instead the async method should have return type of Task or Task<T> just as you would write normally in production code.</span> <br />
<br />
<span style="font-size: large;">Anyway truly testing code asynchronously we can introduce a Task.Delay(x) or Task.Yield() to relinquish the thread momentarily, as below:</span> <br />
<br />
<script src="https://gist.github.com/colindooley11/628cc850d38a07dbd170.js"></script>
<span style="font-size: large;">Here Task.Yield() allows the currently executing thread to be returned to the thread pool and the call to service1.GetResult() executes truly asynchronously, awaiting Task.Delay(x) would work here too. This kind of testing would be useful if wanting to test a timeout set on a cancellation token, for example</span>. <br />
<br />
<br />
<span style="font-size: large;">Lastly, exceptions can be tested asynchronously too. </span><br />
<span style="font-size: large;">Unlike normal exceptions, exceptions which occur in an async method are placed on the Task and the await operation essentially unwraps the exception and rethrows it in the calling code (preserving - mostly - the stack trace) </span><br />
<span style="font-size: large;">The accepted answer in this </span><a href="http://stackoverflow.com/questions/12837128/unit-testing-async-method-for-specific-exception" target="_blank"><span style="font-size: large;">S/O</span></a><span style="font-size: large;"> post and a helper method in Stephen Cleary's book provide a way to test for async exceptions. Apparently NUnit will support this at some point. Moq allows exceptions to be thrown asynchronously ( using the ThrowAsync method). </span><br />
<br />
<h2>
<span style="font-size: x-large;">Mixing async and blocking code</span> </h2>
<br />
<span style="font-size: large;">Don't. There is the potential to cause deadlocks if mixing blocking and async code. </span><br />
<span style="font-size: large;"><br /></span>
<span style="font-size: large;">Consider the following example: </span><br />
<span style="font-size: large;"><br /></span>
<span style="font-size: large;">If a thread is executing in a method which awaits some Task - which is not yet complete - upon returning the same Task to the calling method, if the same thread is then blocked by calling a blocking method such as Task.Wait(), a deadlock will occur when the awaited Task completes. </span><br />
<span style="font-size: large;"><br /></span>
<span style="font-size: large;">This is because, when the awaited Task completes, by default, and owing to the SynchornisationContext captured when the await was invoked (which has the initial thread associated with it) the thread associated with the SynchronisationContext would be used to schedule the continuation after the await, but as this is blocking due to the call to Task.Wait(), which is waiting to transition to unblock, with the transition only able to occur when the continuation occurs there is no way forward. </span><br />
<span style="font-size: large;"><br /></span>
<span style="font-size: large;">This can be addressed by calling ConfigureAwait(false) on the Task being awaited, with the cost of another thread being created or, even better, by ensuring that all methods are marked with async and awaited appropriately(as well as dropping the call to Task.Wait()) and relying on the "waiting" provided by awaiting as opposed to explicitly blocking. </span><br />
<span style="font-size: large;"><br /></span>
<span style="font-size: large;">As a rule of thumb, in order to facilitate scaling, blocking code should probably be avoided if at all possible.</span><br />
<span style="font-size: large;"><br /></span>
<span style="font-size: x-large;">Scaling </span><br />
<span style="font-size: x-large;"><br /></span>
<span style="font-size: large;">The primary benefit of using async/await is the ability to use less resources, notably less memory but also the cost of context switching between threads. </span><br />
<span style="font-size: large;"><br /></span>
<span style="font-size: large;">This benefit is primarily gained when using async/await with I/O (network or disk) as the I/O work is performed by hardware and I/O completion threads independently of worker threads and the thread pool. </span><br />
<span style="font-size: large;"><br /></span>
<span style="font-size: large;">The magnitude of scaling can be quantified by load testing a synchronous and asynchronous version of an application (which my team did for a WebAPI component - which give expected results). Keeping an eye on user requests/sec, failures and requests queued, the benefits really are tangible. The ASP.NET and IIS performance counters can be used to gather information to ascertain the numbers. </span><br />
<span style="font-size: large;"></span><br />
<span style="font-size: large;">To get a good feel for the differences in resources used between sync and async, is to lower the I/O completion port threads and available thread pool threads to mimic contention more quickly and keeping any eye on the threads in use, you would definitely expect to see, as we did in the load testing we performed that the async version of the application used less resources than its synchronous counterpart.</span><br />
<span style="font-size: large;"></span><br />
<span style="font-size: large;">This </span><a href="https://robinsedlaczek.wordpress.com/2014/05/20/improve-server-performance-with-asynchronous-webapi/" target="_blank"><span style="font-size: large;">blog</span></a><span style="font-size: large;"> sums up getting a handle on the differences between sync vs async/await in terms of resources used.</span><br />
<br />
<h2>
<span style="font-size: x-large;">Summary</span> </h2>
<span style="font-size: large;">Async programming has been made almost trivial by the introduction of the async/await keywords in C# 5.0.</span><br />
<span style="font-size: large;"><br /></span>
<span style="font-size: large;">Bearing in mind a few of the gotchas I identified, there shouldn't be too many problems putting it in to practice in production and in unit testing. </span><br />
<span style="font-size: large;"><br /></span>
<span style="font-size: large;">However, I wouldn't necessarily go async-ho and start adding it everywhere, I think that there should be careful consideration on whether or not it is going to provide tangible benefits before peppering code with the language feature, especially as it is pervasive and has some performance costs (and other costs such as learning curve, .NET upgrade impact, unit test updates, and adapting older async APIs - my point being its not free, cheaper in a greenfield project yes, but most certainly not cheaper in a legacy code base). </span><br />
<span style="font-size: large;"><br /></span>
<span style="font-size: large;">To this end some load and performance profiling should be considered as well as not ruling out configuring IIS, the .NET runtime, database and network resources to remove the potential for bottlenecks, by increasing the number of threads available or adding memory or whatever else may be deemed necessary. (this is probably more applicable to retro-fitting this in a legacy codebase)</span><br />
<span style="font-size: large;"><br /></span>
<span style="font-size: large;">But anyway..... </span><br />
<span style="font-size: large;"><br /></span>
<span style="font-size: large;">Happy asyncing. </span>colhttp://www.blogger.com/profile/11580119772205731598noreply@blogger.com0tag:blogger.com,1999:blog-5375025131541210012.post-62345124818875012352014-12-10T14:05:00.002-08:002016-12-06T14:17:41.549-08:00A little bit of powershell in my life<span style="font-size: x-large;">Things were so simple then. </span><br />
<br />
<span style="font-size: large;">Having used 'nix at University and a bit beyond I came to love scripting, using things like Bash and tcl, prior to that I had just written a few simple batch scripts in DOS. Excluding JavaScript from the fray which doesn't, arguably, fit in the same category as lower level scripting languages I really haven't done much apart from a bit of Perl and although its called a scripting language it <i>is </i>a high level programming language. </span><br />
<br />
<span style="font-size: large;">Years later, and after having been indoctrinated into the Microsoft Stack, I have hardly written any scripts or have had any need to, but that is changing. </span><br />
<br />
<h2>
<span style="font-size: x-large;">ALM</span></h2>
<span style="font-size: large;"><br /></span>
<span style="font-size: large;">My current place has a few bits of software that use Powershell and, some, lots of it. Its the tool/language of choice for the ALM framework developed in house, used to push software out from local environments to production environments (in some cases). The framework consists of 100s of scripts and XML configuration files, to map to the myriad of upstream test/integration environments and settings required therein, and it is plugged into TFS to allow CI and gated/manual deployments (no CD here yet!)</span><br />
<span style="font-size: large;"><br /></span>
<span style="font-size: large;">The ALM framework is in the process of being wound down and eventually something like Octopus deploy is going to be used in its place, with the ability to provide extensibility - ironically - using Powershell.</span><br />
<span style="font-size: large;"><br /></span>
<span style="font-size: large;">There are lots of arguments against the bespoke framework including things like why a company whose competitive advantage is not ALM software didn't use an off-the-shelf product, for one.</span><br />
<span style="font-size: large;">Apparently there weren't many (ALM frameworks/tools) about when a consolidated and consistent release framework to production was originally sought. There are aspects of it I like, but as it stands it is far too fragile and "indie" to be a serious candidate for continued use, but that's mostly a moot point.</span><br />
<span style="font-size: large;"><br /></span>
<span style="font-size: large;">Regardless, it has given me the chance to use Powershell and can be used really productively for the jobs you hate. </span><br />
<br />
<h2>
<span style="font-size: large;">Turning on our distributed computer</span></h2>
<br />
<span style="font-size: large;">As part of a re-platforming exercise, we are currently undertaking, of a large e-commerce application, often referred to as the old legacy ball of mud. We are ripping it to shreds (into a new world glass of sand?) and refactoring large parts of it into a distributed messaging based system. </span><br />
<br />
<span style="font-size: large;">NServiceBus is the technology of choice to provide pub/sub choreography and all of the benefits that come with it (resiliency, durability, scaling, error retries and SLAs to name but a few) , With this however, comes a new set of problems over an in-process monolith, not least administering the many smaller parts deployed out to n-number of bits of tin. Which is where this rambling is going toward making my point (sorry!) </span><br />
<span style="font-size: large;"><br /></span>
<span style="font-size: large;">Due to licensing issues, and with a consideration for allowing easy scale out, there is a need to host and administer lots of windows services (hosting NServiceBus endpoints) on lots of different servers. The ALM framework does this most of the time, but not always, leaving little choice of going through the rigmarole of waiting another 15 minutes to push out deployment, or turning on the bits needed out on the boxes. </span><br />
<span style="font-size: large;"><br /></span>
<span style="font-size: large;">NServiceBus provides "endpoints" which are effectively message handlers listening out for specific messages from other endpoints they have subscribed to or are being routed messages from (i.e. events and commands). Once they receive a message they will do something as a result, like persist some data or start off a process. </span><br />
<span style="font-size: large;"><br /></span>
<span style="font-size: large;">Considering that a typical arrangement of NServiceBus endpoints involves something called a distributor (dishing out work) on 1 box and 2+ workers (actually performing the work) on other boxes, its easy to see that if there are 40odd services in total, then in total there are potentially 120 services (housing the endpoints) which are going to have to be administered. As you can imagine, administering start/stop gets very tedious, very quickly in the services manager and this is where Powershell comes into its own.</span><br />
<span style="font-size: large;"><br /></span>
<span style="font-size: large;">For example: </span><br />
<br />
<script src="https://gist.github.com/colindooley11/4cf930a2f70e62157470" type="text/javascript"></script>
<span style="font-size: large;"><br /></span>
<span style="font-size: large;">The simple script above s</span><span style="font-size: large;">tarts all the services on 3 the boxes named in the $computers array, which have the "OrganisationRootNamespace" in the name. This saves about 15 minutes of clicking and lots of sanity.</span><br />
<span style="font-size: large;"><br /></span>
<span style="font-size: large;">The stock library of Powershell has lots of commands providing easy programmatic access to common OS tasks such as registry editing and copying files. Before settling in for a big point and click-a-thon consider using scripting, for the jobs you hate. </span><br />
<span style="font-size: large;"><br /></span>
<span style="font-size: large;">Another thing to consider Powershell for is if you are automating deployment of software components, there are often requisite tasks that are needed, adding users to user accounts, copying files and perhaps pulling code from source control for example.</span><br />
<br />
<span style="font-size: large;">Powershell comes in handy again here. </span><span style="font-size: large;">If you want to do more with Powershell you can load in managed .NET DLLs and have access to powerful APIs for things like TFS and Active Directory. </span><br />
<br />
<span style="font-size: large;">It can be argued that this can all be done in any .NET language, but </span><br />
<span style="font-size: large;">scripting is appealing for getting stuff up quickly, easier adoption/learning curve for noobs, and no need to use a *clunky* IDE. </span><br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />colhttp://www.blogger.com/profile/11580119772205731598noreply@blogger.com0tag:blogger.com,1999:blog-5375025131541210012.post-9855755896735839652014-11-18T15:46:00.002-08:002016-03-31T08:38:13.930-07:00Messaging and transient issues.<h2>
<span style="font-size: x-large;">Messaging and transient issues.</span></h2>
<br />
<span style="font-size: large;">A small post but something worth considering. </span><br />
<span style="font-size: large;"><br /></span>
<span style="font-size: large;">Recently when discussing a situation where a MongoDB replica set was in the process of failing over, concern was raised about writing data - whilst a new primary was being elected. </span><br />
<br />
<span style="font-size: large;">This is going to be a transient issue and issues similar to it - such as temporary server outages and routing are too . </span><br />
<span style="font-size: large;"><br /></span>
<span style="font-size: large;">They are going to take a little time to resolve but should resolve fairly quickly. In the meantime, there a</span><span style="font-size: large;">re a few solutions available whilst this transient issue sorts itself out: </span><br />
<br />
<ol>
<li><span style="font-size: large;">Do nothing. In this case give up and find another job you lazy hacker.</span></li>
<li><span style="font-size: large;">Let the process fall over, report the failure to users and let them try again via a button click. A users experience might be sullied - in the opinion of some - but it still could be a reasonable way to recover (this depends on what stakeholders/business think really). This might not be reasonable for important information which must stored and can not be optionally retried. </span></li>
<li><span style="font-size: large;">Employ a retry mechanism, simply loop a few (reasonable times) until we get success, or employ an exponential back off to give reasonable time for recovery. I have done the former using, as someone I know put it, some funky "AOP" shit. However, I wouldn't recommend doing the AOP stuff for transaction management or bounded retries, because eventual consistency still may never occur and AOP, certainly in a lot of the frameworks I have used, is complete magic. I did have trouble explaining the concepts to some of the less experienced and even seasoned members in my team what that "Transaction" attribute was above my DAL method, and how this was setup on the IOC container and how code was executed pre and post the method in an execution pipeline . </span></li>
<li><span style="font-size: large;">Use a durable messaging framework like NServiceBus. The command or event (message) which was sent will fail and if second level retries are enabled will be retried a specified number of times. If unsuccessful within the specified number of retries (which have exponential back off) The message will be placed on an error queue, relevant administrators will be notified or at least be reporting on this stuff. The exception and or problem will be noted and hopefully fixed and then the message in the error queue will be replayed bringing everything back into a consistent state. And all of this with the user completely unaware that four data centers where nuked. </span></li>
</ol>
<br />
<span style="font-size: large;">That is all. </span><br />
<br />colhttp://www.blogger.com/profile/11580119772205731598noreply@blogger.com3tag:blogger.com,1999:blog-5375025131541210012.post-28968410858112423632014-11-18T15:08:00.002-08:002016-12-06T14:45:42.658-08:00Async/Await<h2>
<span style="font-size: x-large;">Async/Await</span></h2>
<span style="font-size: large;">As I am sure you are aware, this is not a new feature in .NET - having been around for over a year as part of .NET 4.5. There have been plenty of posts about it, and so I'm not going to go into a great deal of depth, as I am not an expert and there are people who have gone into it in more depth than I could. I just want to get a couple of the key concepts up and provide a laymans tilt on it. </span><br />
<span style="font-size: large;"><br /></span>
<span style="font-size: large;">I've been using the language feature as part of a site I am developing as a platform to testbed a few technologies including:</span><br />
<br />
<ul>
<li><span style="font-size: large;">Knockout.Js, </span></li>
<li><span style="font-size: large;">The continuous integration and continuous deployment capabilities of Visual Studio Online (formerly Team Foundation Service)</span></li>
<li><span style="font-size: large;">WebAPI and integrating with 3rd party </span><span style="font-size: large;">web services</span></li>
<li><span style="font-size: large;">Some of the newer C# language features </span></li>
<li><span style="font-size: large;">Bootstrap templates </span></li>
<li><span style="font-size: large;">Stanford Core NLP for .NET </span></li>
</ul>
<div>
<span style="font-size: large;">The site is work in progress and is available @</span><br />
<span style="font-size: large;"><a href="http://wearedev.azurewebsites.net/">http://wearedev.azurewebsites.net/</a></span></div>
<div>
<br /></div>
<br />
<span style="font-size: large;">The main purpose of the site is as a front end for a simple aggregation service for searching for development related articles, plumbing in information from a number of sources such as YouTube, MSDN, PluralSight and Stackoverflow. I've got a few ideas about potentially leveraging the </span><a href="http://sergey-tihon.github.io/Stanford.NLP.NET/StanfordCoreNLP.html" target="_blank">Stanford Core NLP fpr .NET</a><span style="font-size: large;"> ported by Sergey Tihon so that I can, perhaps, perform slightly more accurate content matching, but its a bit of a mess at the moment. </span><br />
<br />
<h3>
<span style="font-size: large;">What is Async/Await? </span></h3>
<h3>
<span style="font-size: large;"> </span></h3>
<span style="font-size: large;">A C# language feature making it almost trivial to perform asynchronous work. There are a number of different ways to already do this in C#/.NET including the Asynchrounous Programming Model (APM), think IAsyncResult, and Event based Asynchrounous Pattern (EAP) think Background Worker. I'm not going to discuss these much here, but suffice it to say async/await is probably a better and easier way to do async work and is also now the preferred way to do async work (I believe there are some low level optimisations to take advantage of here provided for by the TPL (Task based parallel library) which async/await sits atop of. </span><br />
<br />
<span style="font-size: large;">In order to identify a method as something which will initiate asynchronous work, we let the compiler know by adding the async keyword to the signature like so:</span><br />
<br />
<br />
<script src="https://gist.github.com/5f0c2dadc9b74e76ba70.js" type="text/javascript"></script>
<br />
<span style="font-size: large;">Here the Evaluate method is going to initiate some asynchronous process. It also makes this method awaitable. That is to say, a call to this method will not block and can work in an asynchronous manner too. </span><br />
<br />
<span style="font-size: large;">The Evaluate method will return a Task<ResultSet>. The return type here is important. We are not returning something that will necessarily have the value immediately but rather something that signifies that the result of some work will appear at some point in the future. I think I've heard this same kind of concept being called a promise in other languages. Another thing to note is that the actual return statement inside the method body will wrap the return type in Task<T>. So here ResultSet is wrapped in Task<ResultSet> impicitly.</span><br />
<br />
<span style="font-size: large;">Inside the method a call to _dataFeedRetriever.GetDataFeedResults(searchTerm) is prefixed with await. This means that when the aforementioned method does work which does not get a result immediately, the Evaluate method, itself, will be returned from immediately, but crucially the position, in the Evaluate method, where execution got to, will be retained/saved so that when there <i>are </i>results available, normal synchronous execution will resume inside the Evaluate method. </span><br />
<br />
<script src="https://gist.github.com/0e83eaa3c1828097192d.js" type="text/javascript"></script>
<span style="font-size: large;"> </span><br />
<span style="font-size: large;">Interestingly prefixing a method call with await, has the effect of unwrapping the Task<T> type the method returns and pulling out the results when they are available. Conversely, prefixing the return type in the signature means that the return type of the method is wrapped in a Task<T> implicitly. </span><br />
<span style="font-size: large;"><br /></span>
<span style="font-size: large;">If awaitable methods are nested, when encountering the first await where a delay is encountered, execution returns immediately on the first await and then where each await is encountered, back up the call stack, so we back out upstream if you like. When a result is available at the inner most await, execution resumes here and then will continue back up the call stack, perhaps without any further delay at the outer awaits. </span><br />
<span style="font-size: large;"><br /></span>
<span style="font-size: large;">A misconception of async is that utilises additional threads to do its bidding, but this is not always the case. A single thread can be used to call out to an async operation, such as file IO (i.e. I/O bound work). Whilst IO work is being done and a response/result is being waited upon, the thread, used to make the request, can be freed and returned to the thread pool to be used to perform other work, such as incoming web requests. Once the IO is completed, the thread, upon which the operation was started, can be scheduled to be used to continue execution where it left off. </span><br />
<br />
<span style="font-size: large;">This kind of pattern allows for more efficient use of system resources and threads and potentially reduces the need to create new threads helping save on all of the associated costs with this (TEB, TKO, User mode stack and context switching).</span><br />
<span style="font-size: large;"><br /></span>
<span style="font-size: large;">Facilitating the above is a state machine which the compiler generates where the async/await keywords are encountered. It <i>is</i> framework magic but I think it safe to say this is no fly by night 3rd party library but a C# language feature, which although not infallible, is pretty consistent. </span><br />
<br />
<span style="font-size: large;">Coincidentally, the state machine which is generated for async/await is not too dissimilar to the state machine used for the yield keyword, where the position in a sequence is maintained whilst enumerating a sequence. Jeffrey Richter leveraged the yield feature to provide an async library before it became part of the C# language. If you are interested about Jon Skeet has both an excellent tutorial where he dissects async/await <a href="http://codeblog.jonskeet.uk/category/eduasync/" target="_blank">EduaAsync</a> and also a Pluralsight video (which is probably a little easier to follow) going into the innards of the state machine used in async/await.</span><br />
<br />
<h3>
<span style="font-size: large;">Putting it all together</span></h3>
<h3>
<span style="font-size: large;"> </span></h3>
<span style="font-size: large;">Using async/await in the Website mentioned earlier allows me to query the 4 feeds I mentioned at the same time, and take advantage of only having to wait as long as the longest call to a feed, as opposed to the sum of the calls to all the feeds. At the same time making use of only one thread to do all this work and in a nice clean easy to understand manner, is a great argument for using the feature. I'll add the rest of the code to this post or the next. </span>colhttp://www.blogger.com/profile/11580119772205731598noreply@blogger.com0tag:blogger.com,1999:blog-5375025131541210012.post-72329026417292405922014-11-16T12:15:00.001-08:002016-12-06T14:36:19.294-08:00MongoDB<h2>
<span style="font-size: x-large;">Mongolicious</span></h2>
<div>
<br /></div>
<div>
<span style="font-size: large;">MongoDB training, provided by MongoDB themselves, was up for grabs recently, so I put my name in the hat to get a keener insight into MongoDB and NoSQL databases, having used them only a few occasions. The training was attended by a near 50/50 split of devs and DBAs and this led to some interesting debates and reactions. </span><br />
<br />
<span style="font-size: large;">Like lots of the stuff I've been introduced to/have started looking at recently (I am late to nearly every technology party there is) they (Mongo and NoSQL) have been around for a while and have become established as a viable alternative persistence solution to the likes of MSSQL. </span></div>
<div>
<span style="font-size: large;"><br /></span></div>
<div>
<span style="font-size: large;">The main features:</span></div>
<div>
<ul><span style="font-size: large;">
</span>
<li><span style="font-size: large;">Its schema less</span></li>
<span style="font-size: large;">
</span>
<li><span style="font-size: large;">Document oriented/No support for JOINs </span></li>
<span style="font-size: large;">
</span>
<li><span style="font-size: large;">Querying performed using JS/JSON</span></li>
<li><span style="font-size: large;">Indexing</span></li>
<span style="font-size: large;">
</span>
<li><span style="font-size: large;">Replication/Redundancy/High availability</span></li>
<span style="font-size: large;">
</span>
<li><span style="font-size: large;">Scaling out via Sharding </span></li>
<li><span style="font-size: large;">Aggregation (unfortunately we ran out of time so this is not covered here)</span></li>
<li><span style="font-size: large;">Authorisation (again ran out of time on this one) </span></li>
</ul>
<div>
<h3>
<span style="font-size: x-large;">Its schema less</span></h3>
</div>
</div>
<div>
<br /></div>
<div>
<span style="font-size: large;">You'll know doubt be aware that relational databases use schema to determine the shape and type of data that is going to be stored in them. MongoDB doesnt, quite simply. You take an object from some application (most likely) and it is serialised (typically) from your application into JSON and then into BSON and then stored on disk. </span><br />
<br />
<span style="font-size: large;">MongoDB does not care for the type of data used or its shape and when a document is added, a field with a particular datatype, used for one commit, may be of a different type on another (I'm not sure how this would impact indexing) But the look of sheer horror and gasps from the DBAs was priceless (more on this in a minute). </span><br />
<span style="font-size: large;">In practice data type changes like this would probably be as rare as they are in an application as they are in the database, how often (apart from during development) do you go and change the fundamental data-type (that is,to say not its precision or max size) in production code/databases? </span><br />
<br />
<h3>
<span style="font-size: x-large;">Document oriented</span></h3>
<span style="font-size: large;"></span><br />
<span style="font-size: large;">As there is no schema, things we think of as objects in applications (e.g. a Customer object) can be serialised into JSON and then as a MongoDB document added to a collection of customers. A collection is roughly analogous to a table and therefore a collection can have a number of documents like a table can have a number of rows. </span><br />
<span style="font-size: large;"><br /></span>
<span style="font-size: large;">Documents can be entire object graphs with other embedded documents (e.g. a customer with a collection of addresses). Access to documents can be expressed through dot notation when querying and projecting. </span><br />
<span style="font-size: large;"><br /></span>
<span style="font-size: large;">With this arrangement the notion of relationships expressed through queries over many sets of tables (like in SQL) dissapears, relationships are instead expressed by virtue of parent child relationships you see in a typical object graph. Joins are killed off at the DB layer and would generally then be performed in the application (if at all). However, there is the ability to define foreign key relationships with other documents in the database. There is a cost to this though as referential integrity will come at some cost to query performance. </span><br />
<br />
<span style="font-size: large;">With document orientation though I think it seems like a good opportunity to embrace and leverage small transactional boundaries perhaps closely aligned to an aggregate root/entities like in DDD. Each document commit is atomic and so getting the aggregate root root right, I'm sure, would certainly improve concurrency and reduce the size of working sets. </span><br />
<br /></div>
<div>
<h3>
<span style="font-size: x-large;">Querying performed using JS/JSON</span></h3>
<br />
<span style="font-size: large;">One of the particularly great features of MongoDB is the simplicity of its querying. </span><br />
<span style="font-size: large;">For example finding all of the documents with a customer whose name is bob on a reports database (db context is reports below) within the customers collection looks like:</span><br />
<span style="font-size: large;"><br /></span>
<br />
<script src="https://gist.github.com/colindooley11/e87f09876bf2c8424bfc.js" type="text/javascript"></script>
<br />
<span style="font-size: large;">The find method is simple JavaScript the first argument is JSON - straightforward. The predicate is expressed by a key and value pair, with the key being the field in a document with values matching the specified value. Just like in SQL, complex relational expressions can be built up using in built functions such as $lt, $gt, $or and $in.</span><br />
<br />
<br />
<span style="font-size: large;">Taking it a little further we can determine which part of the document we would like to view</span><br />
<br />
<script src="https://gist.github.com/colindooley11/83f308360cdb5750d188.js" type="text/javascript"></script>
<br />
<span style="font-size: large;">The second argument to the find method {customer_name : 1} is a filter to say which fields in the document to return in any results found, analogous to the select statement in SQL. Here customer_name will be returned. We could inverse the statement passing 0 to customer_name and all of the fields EXCEPT customer_name would be returned. </span><br />
<br />
<span style="font-size: large;">As queries get more complicated there can be a tendency to unbalance the brackets in the expression, so be mindful of this. A good tip was to add in all of the sub-expressions first and expand them out. Alternately, use a text editor which has highlighting for brackets/curly braces etc. </span><br />
<br />
<h3>
<span style="font-size: x-large;">Indexing</span></h3>
<div>
<span style="font-size: large;">Indexing is supported via single or compound indexes on document fields. They can be configured differently on primary and secondary nodes dependent on needs, with a bit of jiggery-pokery (taking nodes offline and configuring them differently). </span></div>
<div>
<span style="font-size: large;">I can't speak much for the absolute speed and performance of the indexes, but relative to not having them, they are much quicker. </span></div>
<div>
<span style="font-size: large;">Under the hood they are implemented using B-Trees, whether this is optimal would be for some performance analysis and comparisons to answer. </span></div>
<br />
<h3>
<span style="font-size: x-large;">Replication/Redundancy/High availability </span></h3>
<br />
<span style="font-size: large;">Apart from when development sandboxing and running in standalone mode, the recommendation is to run MongoDB databases in a replica set which must contain at least 3 nodes to be effective and always an odd number for larger replica sets (explained in a moment). A replica set consists of a minimum of 3 nodes in total. A primary and 2 secondaries. </span><br />
<br />
<span style="font-size: large;">All writes and reads are, by default performed, at the primary and then the instructions and data are replicated to the secondaries. </span><br />
<br />
<span style="font-size: large;">Primary node failure and subsequent failover is handled by an election process to determine which secondary should become the new primary. The odd number of nodes in a replica set is crucial here as this ensures that a majority can be obtained by participating nodes and a new primary elected. If an even number of nodes was configured in a replica set, there could be a shared vote for leader and all of the nodes would go into a read only state, as there would be no primary. I'm not entirely sure if MongoDB enforces the odd number rule we didn't try it out in the examples. </span><br />
<br />
<span style="font-size: large;">In the 3 node replica set arrangement, if the primary node where to fail, an election would take place between the secondaries and which ever node obtained the majority vote first would be come primary. Obviously, nodes can not vote for themselves. In this instance there is 50/50 chance of either one of the secondaries becoming the new primary, as long as there are no priorities configured. </span><br />
<span style="font-size: large;"><br /></span>
<span style="font-size: large;">Using secondaries here as foils for the primary would see this arrangement more attuned with Mirroring in MSSQL where exact copies of the primary data a farmed out to slaves (in essence) ensuring redundancy. And so replication here is not necessarily the same as replication used in MSSQL. </span><br />
<br />
<span style="font-size: large;">However, a slight tweak to configuration of the secondaries means that all reading need not occur through the primary, and specific secondaries can be read from directly. This provides an element of scaling out, where writes and reads are performed through different servers, Indeed, indexing can be tuned separately on any of the nodes to help with optimal query execution </span><br />
<span style="font-size: large;"><br /></span>
<span style="font-size: large;">The configuration of replica sets is an absolute doddle and this was the general consensus amongst both devs and DBAs</span>.<br />
<br />
<br />
<h3>
</h3>
<h3>
<span style="font-size: x-large;">Scaling out via Sharding</span></h3>
<br />
<span style="font-size: large;">This got the trainer hot under the collar and is one of the more prominent features, particularly for the larger software projects you may find in an enterprise. Sharding only requires a tad more work than is required for adding a replica set to get going. </span><br />
<br />
<span style="font-size: large;">Analogous to partitioning in MSSQL, sharding is, in essence, a way of making use of a distributed index. It also helps distribute storage and retrieval across MongoDB instances, thereby performing a kind of load balancing. </span><br />
<br />
<span style="font-size: large;">Sharding occurs at a collection level and an appropriate shard key is used to shard upon. So, for example, if we were to take a customer collection we could shard it on the email address of customers.</span><br />
<span style="font-size: large;"><br /></span>
<span style="font-size: large;">Consequently documents with email addresses starting A through S and S through Z could end up being placed into 2 different shards with all read and writes for data with email addresses falling within the respective ranges being routed appropriately to write or retrieve related data (a mongos server and configuration servers are required for this) </span><br />
<span style="font-size: large;"> </span><br />
<span style="font-size: large;">Shards contain chunks of a pre-defined size. If data, in a shard, exceeds this threshold then splitting occurs and data is moved between shards in attempt to maintain a close to equal split of data amongst the shards. This process occurs in the background, it will most likely impact performance but its not something the user has to worry about (that is to say this would be a non-functional requirement). This allows auto-scaling to a degree, until you realise or plan for more shards as your data needs grow. </span><br />
<br />
<br />
<h3>
<span style="font-size: x-large;">DBA Horror</span></h3>
<span style="font-size: large;">I think the developers found using and configuring MongoDB a more edifying experience than the DBAs. I would hope the DBAs involved would agree I was in fair in saying they seemed to find the use of the shell, command prompt and JSON a little challenging. Higher level concepts similar in MongoDB and SQL worlds were easily digested by attendees from both sides. The consensus though, generally, was that MongoDB was a low friction route into administering and working with a highly scalable and available data persistence mechanism for developers! Having only been a SQL hacker and not being involved enterprise persistence choices I'm not best placed to say whether MongoDB would be a better or replacement solution for SQL, but it does seem bloody easy to use and administer.</span><br />
<span style="font-size: large;"><br /></span>
<span style="font-size: large;">To this end I would recommend attending the MongoDB course if it wings its way into your organisation or take the MongoDB intro courses on Pluralsight or MongoDB University to help make an informed decision before defaulting to SQL.</span><br />
<span style="font-size: large;"><br /></span>
<span style="font-size: large;"><br /></span>
<br />
<h3>
<span style="font-size: x-large;"> </span></h3>
</div>
<div>
<br /></div>colhttp://www.blogger.com/profile/11580119772205731598noreply@blogger.com5tag:blogger.com,1999:blog-5375025131541210012.post-49463236432987045292014-09-01T16:05:00.001-07:002014-09-03T14:58:11.499-07:00TFS and integration test builds from Shelvesets.<h2>
</h2>
<span style="font-size: large;">Our team have been involved in a fair bit of release activity, as of late. We are gearing up for our wagile (our previous CTO's term not mine) project's big bang release to production.</span><br />
<br />
<span style="font-size: large;">Due to the release pipeline we have been forced to release our changes with another team. </span><br />
<br />
<span style="font-size: large;">The code resides in 2 different branches, both of which are branched off of a main branch. However, in TFS terms, these are unrelated until at least one merge has taken place (a baseless merge). </span><br />
<br />
<span style="font-size: large;">Quite simply we have the following arrangement: </span><br />
<br />
<span style="font-size: large;">a (main)</span><br />
<span style="font-size: large;">b (us) c (them)</span><br />
<br />
<span style="font-size: large;">It was decided that we should merge our changes in (b) into their changes in (c) and then use the build resulting from the merge to push out to an integration environment, where our testers would run any remaining manual tests (or kick off any semi automated service/integration tests) </span><br />
<br />
<span style="font-size: large;">Forgetting the fact this would actually be a baseless merge in the first instance, actually performing the merge and getting it out, would mean that we would either have to push out locally from where the merge was performed and not check-in, or - if we were prepared to rollback if anything was to fail - check in. </span><br />
<span style="font-size: large;">We took the latter option in the and it was a bit painful, as there were failures and we had to rollback.</span><br />
<span style="font-size: large;"><br /></span>
<span style="font-size: large;">To save this pain we took advantage of the fact that builds can be pushed out from shelvesets as well as normal changesets and latest sources. </span><br />
<span style="font-size: large;"><br /></span>
<span style="font-size: large;">Bearing in mind a merge still has to be performed , the merge can be checked-in, as a shelveset, and then this shelveset can be referred to in the options on the build definition like so:</span><br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhn5mEArSOafagPxC_a4aMoEI5wDjjpOAf6Zn8pQL6ayq-5hImSJFuVVBt1j8POPrjt7TDNL-ATZZJjrjoNcek4wRiS2-KMF5yVr0KQM9Ij0fsf8z0RKVAUuphyphenhyphen47p1zQ1PsFTG1p7xd78a/s1600/Build+with+shelveset..PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhn5mEArSOafagPxC_a4aMoEI5wDjjpOAf6Zn8pQL6ayq-5hImSJFuVVBt1j8POPrjt7TDNL-ATZZJjrjoNcek4wRiS2-KMF5yVr0KQM9Ij0fsf8z0RKVAUuphyphenhyphen47p1zQ1PsFTG1p7xd78a/s1600/Build+with+shelveset..PNG" height="640" width="603" /></a></div>
<br />
<span style="font-size: large;"><br />Simply choose "latest sources with shelveset", in the drop down, and then the relevant shelveset and voila you have branch (c) + branch (b), without any of the baggage of a changeset (I'm sure internally shelvesets are stored in a similar manner but are surfaced differently.) </span><br />
<span style="font-size: large;"><br /></span>
<span style="font-size: large;">Hopefully this, "soft merging" saves someone a bit of pain and allows integration testing before fully merging 2 branches. </span>colhttp://www.blogger.com/profile/11580119772205731598noreply@blogger.com0tag:blogger.com,1999:blog-5375025131541210012.post-79405642716909816452014-09-01T15:30:00.002-07:002014-12-10T14:35:57.146-08:00DDD or not to DDD? That may or may not be the question.<h2>
<span style="font-size: x-large;">DDD </span></h2>
<h2>
<span style="font-size: x-large;"></span></h2>
<span style="font-size: large;">I've had/am involved with my first commercial experience of DDD, or at the very least, something heading that way. My team are developing their first DDD software component. I didn't realise I was at least 10 years behind the leading edge ( the year which Eric Evans seminal work DDD was published was the same year I started my career as an embedded software developer, which is as close to enterprise development as Pluto is to the Sun) </span><br />
<br />
<h3>
<span style="font-size: large;">DDD Quickly</span></h3>
<span style="font-size: large;">After being in a few larger organizations and having dealt with a fair few types of design approaches (ranging from monolithic to typical n-tier), it seems that DDD has finally achieved mainstream adoption in the industry and the shit kickers - I include myself here - are just about ready to consume it, and murder it potentially :) </span><br />
<br />
<span style="font-size: large;">I have read "DDD quickly" on InfoQ, which I would highly recommend to get a flavour for DDD in general, and have recently started reading "Implementing DDD" by Vaughn Vernon (or Vince Vaughan - as I keep saying) . I chose this over Eric Evans' book as it is more current and seems to have lots of practical examples. </span><span style="font-size: large;"><span style="font-size: large;">The book is well written and has so far (3 chapters in) kept me engaged and interested. Pretty good going for quite a dry subject like DDD.</span> </span><br />
<span style="font-size: large;"> </span><br />
<span style="font-size: large;">My first impressions of the subject are that its complex, nuanced and subjective. When things aren't black and white and there are lots of possible ways to skin a cat, extra attention and leaps of faith are required. Also because of this there are lots of terms to learn and use which in turn refer to almost every other aspect of important literature in the field of enterprise </span><br />
<span style="font-size: large;">development (P of EAA, GoF, DDD) . I will never be able to grok it, I have too many distractions, like life. However, I will try and learn it enough to practice it and contribute to discussion about it. </span><br />
<br />
<h3>
<span style="font-size: large;">DDD Lite</span></h3>
<span style="font-size: large;">For our team's problem at hand it appears, on the surface, we have had some of the hard work done for us and have a bounded context with a ubiquitous language that we can play around with. Additionally the bounded context is conveniently mapped 1-1 to a sub-domain (good practice according to Vaughn).</span><br />
<br />
<span style="font-size: large;">The lack of any analysis about what the bounded context and sub-domain are here, by our team, and also our use of just a few tactical patterns (aggregate/entity identification and the use of a repository) owe to the "DDD lite" subheading, which Vaughn says is symptomatic of a lack of strategic patterns being employed. </span><br />
<br />
<span style="font-size: large;">That is to say our team is taking a technical approach to DDD and is not fully involved in bounded context or ubiquitous language identification at this point. (or other strategic practices). This is fortunate and unfortunate. Fortunate that we can kick on and use tactical patterns but unfortunate in that we can't apply the strategic patterns which we will have to apply at a future point in time and thereby miss out on crucial learning in this area. </span><br />
<br />
<span style="font-size: large;">The software component was initially going to be developed using an anaemic domain model/data model and transaction script. However, our architect saw it as a candidate for DDD, with the idea that it will prepare us for future DDD projects.</span><br />
<span style="font-size: large;"> <br /> </span><br />
<span style="font-size: large;">I don't believe there was anything particularly wrong with the familiar and fairly straight-forward approach (anaemic/transaction script), we were going to use, and going on some of Vince's guidelines (or Greg young agile-esque T-shirt size - Small Small estimation), the component wouldn't necessarily be a priority for DDD love, as it is not in a core domain or a particularly complex sub-domain, but I have to agree with my architect, it does seem that it is useful as a stepping stone as our team gear up for more DDD development. </span><br />
<br />
<span style="font-size: large;">The bounded context is a little lacking though, there are no invariants, and consistency is a fairly straightforward affair. There is no usage of domain events, either, for example. However, the structure of the component means it aligns to a ports and adapters type architecture, where we can plug in back ends and front ends in with ease (for example we plug in a resilient/distributed messaging framework to communicate with other bounded contexts). </span><br />
<br />
<h3>
<span style="font-size: large;">Back to OO</span></h3>
<span style="font-size: large;">Martin Fowler describes the anaemic model in less than favourable terms. I have seen it used in my enterprise and elsewhere. I now know where I have seen it correctly used, in simple RESTful CRUD components for example, and "wrongly" used if what you end up doing is pulling behavior into a myriad of "domain?" services - which is arguably an easier way to get to work done on some data - and transposing/pivoting behavior out of domain objects and splaying/fanning it into "services", relying on DI to pull everything together. If your constructor has 15 dependencies maybe something is a bit smelly.</span><br />
<br />
<span style="font-size: large;">So with DDD what happens regarding anaemia?. We feed the domain model up by falling back on encapsulation, a fundamental tenet of OO, where state and behaviour - on that state - are packaged together, heping form what is known as a rich domain model. </span><br />
<br />
<span style="font-size: large;">We are told it is a big problem that this kind of stuff has been abandoned for the skinny latte way of design, but I do wonder how bad it actually is. </span><br />
<span style="font-size: large;"><br /></span>
<br />
<h3>
<span style="font-size: large;">The Dilemma</span> </h3>
<span style="font-size: large;">So now we are developing using DDD (or "DDD lite") and this goes back to the question in this post's title. Should we or shouldn't we? It seems that this decision shouldn't just be based on technical considerations or that the decision should just be outright rejection because of hearsay about complexity, over engineering or the absence of a highly-skilled hero team. Vaughn states that DDD is a complex process but out of it <i>can </i>drop a clear and simple manifestation of a solution to a set of business problems. </span><br />
<br />
<span style="font-size: large;">There are a series of questions that should be asked before embarking down the DDD route which people like Greg Young and Vaughn Vernon both advocate asking. One of those shared by them goes something like: "By embarking on DDD for your software, is this adding value to the competitive advantage of the business the software is being developed for?". </span><br />
<br />
<span style="font-size: large;">In this instance we <i>are </i>adding value to the competitive advantage of the business - even if it is marginal. We are also providing ourselves with a firmer footing for DDD work in the future, which will see further value added. So maybe that's OK even if we are just doing "DDD Lite" for now. </span><br />
<br />
<span style="font-size: large;">If adopting only some of the tactical patterns give us marginal improvements this is still worthwhile and not hurting the business, and if an opportunity like this on a project you are working on presents itself - to get a leg up with DDD without going the whole hog - then it should be considered. </span><br />
<span style="font-size: large;"><br /></span>
<span style="font-size: large;"><br /></span>
<br />
<br />
<br />
<br />
<br />
<span style="font-size: large;"><br /></span>
<br />
<h2>
<span style="font-size: large;"><br /></span></h2>
colhttp://www.blogger.com/profile/11580119772205731598noreply@blogger.com3tag:blogger.com,1999:blog-5375025131541210012.post-1672594928119120422014-07-04T15:02:00.001-07:002017-09-19T12:59:46.359-07:00Currying and Closures<h2>
<span style="font-size: x-large;">Just a quick one</span></h2>
<span style="font-size: large;">Closures might be well known amongst many C# practitioners - apart from me it seems. I read about them ages ago and then haven't knowingly applied them. Of course, I have been using them (or at least a variation on the theme) often in C# by virtue of using 'lambdas in lambdas' when using LINQ for example, and lambdas which refer to variables in the method in which they are created. </span><br />
<br />
<span style="font-size: large;">I mentioned functional programming including Currying in the last post, and this is still my current focus, its satisfying my ADHD for the time being. </span><br />
<br />
<span style="font-size: large;">Anyway, It turns out that if you want a bit of a better understanding of how Currying works, then understanding closures can help you take a leap of faith. </span><br />
<br />
<span style="font-size: large;"></span>
<br />
<h2>
<span style="font-size: x-large;">Currying in Haskell</span></h2>
<span style="font-size: large;">The following example is taken from <a href="http://learnyouahaskell.com/" rel="nofollow" target="_blank">Learn you a Haskell</a></span><br />
<br />
<script src="https://www.gistfy.com/github/gist/2634d412b736e6b38c8b" type="text/javascript"></script>
<span style="font-size: large;">The function multiplies 3 numbers together and returns the result.</span><br />
<br />
<span style="font-size: large;">The first line defines a function, multThree, which can be read as taking 3 arguments and returning a value - all of the same type - signified by a. </span><br />
<br />
<span style="font-size: large;">The second line maps the first 3 arguments to x,y and z which are then used in a function body where x * y * z occurs. When the function is called, e.g. multThree 2 5 5, it will yield the result of 50. There is no need for an explicit return , like in C#, the result is just returned. </span><br />
<br />
<span style="font-size: large;">In Haskell, every function (well, nearly - I think - functions can actually be uncurried) is curried automatically which means that functions with multiple arguments are broken down so that each argument has an accompanying return value. This opens up the door to Partial Application (to be covered in a future post), which means that not all of the arguments need to be supplied immediately. </span><br />
<br />
<span style="font-size: large;">Taking the above function definition again but showing where Currying occurs with the use of parentheses: </span><br />
<br />
<script src="https://www.gistfy.com/github/gist/33e3289ba68011e7ba93?access_token=29a633f0a757668bf3a8b67f4062c1e6995e1cd8" type="text/javascript"></script>
<span style="font-size: large;">We have a function which takes an a and returns</span><br />
<span style="font-size: large;">a function which takes an a and returns a function which </span><br />
<span style="font-size: large;">takes an a and returns an a. Confusing, yes. Bloody confusing. </span><br />
<br />
<span style="font-size: large;">This is how I have read Currying definitions in lots of places which hasn't been a great deal of help. What I couldn't work out, particularly, was where the actual multiplication would take place, if state was being maintained somewhere and/or if the result was being worked out cumulatively (via some kind callback mechanism maybe).</span><br />
<br />
<h2>
<span style="font-size: x-large;">Reading</span></h2>
<span style="font-size: large;">After reading a few chapters of a CLR via C#, a mixture of blogs and stack overflow posts, a simple rearrangement of the function makes things alot more clear. (as well as knowing closures are at work) </span><br />
<br />
<span style="font-size: large;">When the multThree function definition is encountered by the Haskell compiler something like this occurs:</span><br />
<br />
<script src="https://www.gistfy.com/github/gist/2bc276b1f42c242938f1" type="text/javascript"></script>
<span style="font-size: large;">Three lambda statements are created. </span><br />
<span style="font-size: large;">The first takes x and </span><br />
<span style="font-size: large;"> returns a lambda which takes y and </span><br />
<span style="font-size: large;"> returns a lambda which takes z and has the function body which returns the result</span><br />
<br />
<span style="font-size: large;">(Note the "/x -> ..." notation, this signifies a lambda that takes one argument in Haskell, in C# this is very similar " x => ...." would mean the same thing.) </span><br />
<span style="font-size: large;"></span><br />
<span style="font-size: large;"></span>
<span style="font-size: large;">And with values: </span><br />
<br />
<script src="https://www.gistfy.com/github/gist/53033e678155cb073926?access_token=29a633f0a757668bf3a8b67f4062c1e6995e1cd8" type="text/javascript"></script>
<span style="font-size: large;">With all of the values known, the function body can multiply all of the values together, to get a result. And this is the important bit, only when ALL of the values are supplied can the function body be executed. </span><br />
<br />
<span style="font-size: large;">The key thing here is that this is all facilitated by closures which was the missing bridge to working out how the values were getting passed around. Basically each nested lambda or child lambda closes over its calling environment and any of the values present in its calling environment are available to it. Essentially the values, once supplied, are in scope all the way back up the calling chain. </span><br />
<br />
<h2>
<span style="font-size: x-large;">Back to C#</span></h2>
<span style="font-size: large;">In trying to work all of this out I cross referenced an example by Jon Skeet on Currying and Partial application and a chapter in CLR via C# by Jeffrey Richter about lambdas, which showed how lambdas referencing variables in the method, in which they are defined, close over that method. That is, have access to values in the method. </span><br />
<span style="font-size: large;"><br /></span>
<span style="font-size: large;">There is a curious side effect in C# (at least) in that a value which is closed over, has its scope widened beyond the method to the class in which the method is defined. This is because a helper class is generated for the lambda method - containing the implementation of the method itself and values which that method will need to use which are copied into the class. This class is then added to the original class containing the anonymous method as a field. </span><br />
<br />
<span style="font-size: large;">I then cross referenced that this occurred in Haskell (well on a few S/O posts - so still anecdotal - but certainly reasonably explained) and this has cemented my understanding. </span><br />
<span style="font-size: large;"></span><br />
<h2>
<span style="font-size: large;"><span style="font-size: x-large;">Off to the Balti House</span></span></h2>
<span style="font-size: large;">Currying may be straightforward but without knowing some of the facilitators it can be hard to know or accept, even, what is supposed to happen when Currying is occurring. A lot of the posts I have read regurgitate, almost, the Wiki definition of Currying which isn't that helpful if you want to know a bit more of the "how". I might have simply needed to read somewhere about lambda calculus which explains this simple requisite. But in case you land here, before there, then I hope it might help a little. </span><br />
<br />colhttp://www.blogger.com/profile/11580119772205731598noreply@blogger.com1tag:blogger.com,1999:blog-5375025131541210012.post-4575506976623002422014-06-23T06:11:00.001-07:002014-06-23T15:12:10.687-07:00Functional programming and blind adoption of 3rd party software.<h2>
<span style="font-size: x-large;">Training</span></h2>
<span style="font-size: large;">I was lucky enough to have had the pleasure of attending an introduction to Domain Driven Design, CQRS (coined - in part at least - by the presenter himself) and Event Sourcing by Greg Young at Skills Matters in London last week. It was snappy, contentious - at times - and very informative and definitely gave me a lot of food for thought, I now have a reading list as long as my arm. As an added bonus, the people who attended, as well as Greg himself, were approachable and easy going, as the beers after will testify to. </span><br />
<span style="font-size: large;"><br /></span>
<span style="font-size: large;">The background to me getting on this course, is that we are currently going through a massive set of architectural, infrastructural and cultural change at my current employer, and this course has set me up to be asking more searching questions and provide input on some of the stuff we are currently doing and are planning to be doing soon.</span><br />
<span style="font-size: large;"><br /></span>
<span style="font-size: large;">I'm not going to write about any of the above though, a quick search will elicit a million articles, and will be a lot more informative about them than I could be, at the moment. </span> <br />
<h2>
<span style="font-size: x-large;">Piqued Interest</span></h2>
<br />
<span style="font-size: large;">However, the thing that interested me most was a small nod to functional programming. I'm aware of functional programming but didn't study it at Uni and didn't really know the differences between imperative and functional programming until I actively searched it out.</span><br />
<br />
<span style="font-size: large;">Greg mentioned that he was using Erlang which has been developed by a phone company for future use internally and this had given him ideas about how to approach programming in the imperative world, from different angles. I was hooked, anything that can make me think about problems differently is worth knowing about.</span><span style="font-size: large;"> </span><span style="font-size: large;"><span style="font-size: x-large;"> </span></span><br />
<span style="font-size: large;"><span style="font-size: x-large;"><br /></span></span>
<br />
<h2>
<span style="font-size: large;"><span style="font-size: x-large;">Containerless</span></span></h2>
<span style="font-size: large;">Everyone knows, by now, that the best (and maybe only) way to arrange your n-tier architecture is to introduce your favourite IOC container after separating out your dependencies and introducing myriad mapping between the layers. You are now at the pinnacle of software architecture. Or not, perhaps. </span><br />
<br />
<span style="font-size: large;">The problem with introducing a container, or any 3rd party library for that matter, is reliance on that bit of software being stable or reasonably straightforward to leverage. When they are neither or worse, life becomes painful. Greg raised this point and it really resonated, as I have been there before. No matter though, he suggested a way around it!</span><br />
<span style="font-size: large;"><br /></span>
<br />
<h2>
<span style="font-size: x-large;">Container Bowels</span></h2>
<span style="font-size: large;">I have been in the bowels of Unity (of Enterprise Library fame) trying to work out why I'm</span><span style="font-size: large;"> </span><span style="font-size: large;"> getting a Stack Overflow exception after cycles in my composition root (where the dependencies are registered) have been inadvertently configured (obviously only knowing that bit retrospectively). Whilst at the same time the job I'm actually supposed to be doing is on the back burner. </span><br />
<br />
<span style="font-size: large;">The solution. Ditch the container. I had searched for alternatives registering dependencies without a container, in the past, but gave up after about 10 minutes of Googling and didn't think to attempt it any other way. I know it's not a great leap to do all the wiring up in one place but its easy to become blinkered when something, in the main, seems to work even if it is a painful from time to time. </span><br />
<span style="font-size: large;"><br /></span>
<br />
<h2>
<span style="font-size: x-large;">Partial Application</span></h2>
<span style="font-size: large;">Partial application is a technique used in functional programming which allows a function to have some of its arguments to be made invariant or hard-coded in effect. This function can then be used as an argument to another function giving the same result each time it is invoked. This allows composition of dependency trees . For example, a function which took a connection string, which returned a data access component could be used by multiple functions which required data access. (Code example to follow) </span><br />
<span style="font-size: large;"><br /></span>
<span style="font-size: large;">This approach can be used as a way of performing dependency injection and allow you to get rid of the container and see all of your dependencies up front in one place without 10k lines of code acting as an intermediary. You will inevitably lose some flexibility and features but on the other hand you may gain simplicity, performance increases and less headaches, so make a judgement call; you might just have defaulted on this before. It will certainly make me think twice. </span><br />
<span style="font-size: large;"><br /></span>
<br />
<h2>
<span style="font-size: x-large;">Haskell</span></h2>
<span style="font-size: large;">After learning about this I wanted to know a bit more about functional programming and came across Haskell and an absolutely fantastic tutorial website <a href="http://learnyouahaskell.com/" rel="nofollow" target="_blank">Learnyouahaskellforgreatgood</a>. The first few chapters are fairly easy going and I am now up to the section about Currying (related to Partial Application) and it was only at this point that I had to refer to my mother tongue of my day to day programming (C#) in order to understand exactly what was going on, as I couldn't quite visualise or translate it. </span><br />
<br />
<span style="font-size: large;">Step forward Jon Skeet and his distillation of the concepts in C# <a href="http://msmvps.com/blogs/jon_skeet/archive/2012/01/30/currying-vs-partial-function-application.aspx" rel="nofollow" target="_blank">Jon Skeet currying in C#</a>. This helped to highlight a potential issue or advantage, perhaps, I end up trying to transpose one language into another. If I was starting programming from a clean sheet I wouldn't suffer from this turmoil. </span><br />
<span style="font-size: large;"><br /></span>
<span style="font-size: large;">Haskell is based on lambda calculus and its easy to see it was used as an inspiration for C#'s LINQ - which has seen "lambdas" being awarded first class citizenry by the <a href="http://stackoverflow.com/questions/4683506/are-there-any-connections-between-haskell-and-linq" target="_blank">language designers</a>. </span><br />
<br />
<span style="font-size: large;">And so it dawns on me that by using LINQ and lambdas I have been using (at least in essence) functional programming without even knowing it. </span><br />
<br />
<span style="font-size: large;">I particularly like the heavy use of recursion (not always optimal but definitely elegant) the quick sort example (although noted "cheesy" by the Haskell tutorial website author) shows just how elegant Haskell is and, shown side by side with a C example, is about 10 times smaller! </span><br />
<span style="font-size: large;"><br /></span>
<span style="font-size: large;">Collections and their processing are a fundamental tenet of Haskell and simply splitting lists into heads and tails makes you reason about where that approach might have been useful before. I am yet to get to folding but I'm hoping this and other features, gives me some new ideas about how I can approach problems in the imperative world in the future.. I've still got lots to get through and it may</span><span style="font-size: large;"> just turn out to be an academic exercise but its better than Coronation Street. </span><br />
<br />
<h2>
<span style="font-size: large;">So...</span></h2>
<span style="font-size: large;">I intended this post to be far more technical, with code examples, but I can't find my notes from the course and so the presentation of some examples is a bit lacking. The main thrust of the post though, I suppose, is to consider the adoption of 3rd party libraries a little more carefully before folding them into your architectural cake mix, I've been burnt on more than 1 occasion and so I won't just blindly follow implied industry practices or being a technology magpie hankering after the latest and greatest.</span><br />
<span style="font-size: large;"></span><br />
<span style="font-size: large;"></span><br />
<span style="font-size: large;"></span><br />
<span style="font-size: large;"></span><br />
<span style="font-size: large;"></span><br />
<span style="font-size: large;"></span><br />
<span style="font-size: large;"></span><br />
<span style="font-size: large;"></span><br />
<span style="font-size: large;"><br /></span>
<span style="font-size: large;"><br /></span>
<span style="font-size: large;"><br /></span>
<br />
<span style="font-size: large;"><br /></span>
<span style="font-size: large;"><br /></span>colhttp://www.blogger.com/profile/11580119772205731598noreply@blogger.com0tag:blogger.com,1999:blog-5375025131541210012.post-36343483730756447222014-05-23T14:53:00.003-07:002014-05-27T16:17:41.337-07:00SSDT March 2014 Static Code Analysis Extensibility<h2>
<span style="font-size: x-large;">Static Code Paralysis.</span></h2>
<span style="font-size: large;">Coding standards (and more latterly styles) in any organisation of any size are liable to incite religious wars and makes zealots out of even the most considered or apathetic of us. Inconsistency and subjectivity, lack of enforcement and heavy handedness all contribute to the contempt often leveled at them. </span><br />
<br />
<h2>
<span style="font-size: x-large;">Sod it.</span></h2>
<span style="font-size: large;">I say sod all that, if you or your company aren't using static analysis tools then you are missing out on improved code quality and productivity (borne out from a consistent approach) and reducing the most banal and mind numbing conversations and sometimes pissing contests associated with discussing the minutiae of standards. </span><br />
<br />
<span style="font-size: large;">As an aside, having a few years under my belt in the industry I've found that people who "really" moan most about this are usually rabble-rousing morale sappers who like to moan about everything, so it stops them doing this and maybe might lead to them getting on with work instead! </span><br />
<br />
<h2>
<span style="font-size: x-large;">Makes things easier for everyone involved - really.</span></h2>
<br />
<span style="font-size: large;">Barring any major clangers or anything truly detrimental to program performance and readability if there is a consistent view of coding standards and styles which are automatically enforced, and can - within reason - be suppressed and challenged from time to time to aid with best practice, then the pain of the standards largely goes away and it becomes something that just happens is accepted and then more important things which should be reasoned about - are.</span><br />
<br />
<span style="font-size: large;">Currently, in our organisation we use a combination of StyleCop, Resharper, and FxCop for our .NET static code analysis needs and its generally a pretty edifying experience, even though there were a few integration issues which were quickly resolved. </span><br />
<br />
<h2>
<span style="font-size: x-large;">The SQL parallel universe.</span></h2>
<br />
<span style="font-size: large;">There is one big place (at least) where all of the pain and fundamentalism, mentioned above, still exists, the database Universe. Our team have taken the initiative to get code analysis into the enterprise for new SQL development in order that our data team has less reasons to pull releases, saving many thousands of pounds and delay in the process, for missing a few spaces and tabs (yes this stuff really does still happen). </span><br />
<br />
<span style="font-size: large;">Moving the identification of this type of "defect" back to the development life cycle, where it belongs, in itself saves cost, egg on peoples faces and confidence from the exec that the IT department aren't a gang of idiots. </span><br />
<br />
<h2>
<span style="font-size: large;">SSDT Code Analysis Extensibility</span></h2>
<span style="font-size: large;">With the SSDT March 2014 release, comes the ability to extend (once again - its been missing from the last few releases) the code analysis rules for Visual Studio 2012 and 2013, that you get out of the box. </span><br />
<br />
<span style="font-size: large;">The MSDN example application for extending the rules is good (</span><span style="font-size: large;"><span style="font-size: large;"><a href="http://msdn.microsoft.com/en-us/library/dn632175%28v=vs.103%29.aspx" rel="nofollow" target="_blank">Authoring an SSDT Rule</a>) </span> the accompanying documentation for the some of the types used in the API could be better, which meant having to debug to find out what some of the TSqlFragments were, which was a little time consuming. </span><br />
<span style="font-size: large;"> </span><br />
<span style="font-size: large;">A very important thing to note is that you will need to install the SSDT 2014 March release in order to debug and execute the code analysis sets you develop. Also the folder which is probed for the extensions is in the SQL extensions folder </span><br />
<ul>
<li><span style="font-size: large;"><span style="font-size: small;"><Microsoft SQL Server Install Dir>\120\DAC\Bin\Extensions</span></span></li>
</ul>
<span style="font-size: large;"><u>NOT</u> the Visual Studio extensions folder </span><br />
<ul>
<li><span style="font-size: large;"><span style="font-size: small;"><Visual Studio Install Dir>\Common7\IDE\Extensions\Microsoft\SQLDB\DAC\120\Extensions</span></span></li>
</ul>
<span style="font-size: large;"><span style="font-size: small;"><span style="font-size: large;">I couldnt get the latter location to work after a number of attempts.</span> </span> </span><br />
<br />
<span style="font-size: large;">That being said, once the sample is up and running getting yourself going is a breeze. </span><br />
<br />
<span style="font-size: large;">The analysis is facilitated by dog-fooding SQL scripts under scrutiny, into a set of strongly type TSqlFragments which are themselves plopped into an AST. The </span><span style="font-size: large;"><span style="font-size: large;">TSqlFragments </span>allow for extensibility via use of the visitor pattern, accepting TSqlFragmentVisitors which works really well. </span><br />
<br />
<span style="font-size: large;">This arrangement also allows for fine grained control and the ability to have scope, such that a BooleanExpression on a JOIN can be treated differently to one in a WHERE clause, by nesting visitors. </span><br />
<br />
<span style="font-size: large;">The visitors are used by rules derived from SqlCodeAnalysisRule which can look at the fragments and allow developers to codify their organisation's take on what they should look like or whether or not they should be present. For example, ensuring SELECT * isn't used or that column numbers aren't used in ORDER BY clauses. </span><br />
<br />
<span style="font-size: large;">Errors and warnings are presented to users by building up a SqlRuleProblem giving appropriate context and severity. </span><br />
<span style="font-size: large;"><br /></span>
<span style="font-size: large;">The TSqlFragmentVisitor has over 800 overloads which, incidentally, rips ReSharper intellisense to shreds. It allows developers to visit on nearly every conceivable element type of a SQL script, and do something with it. </span><br />
<span style="font-size: large;"><br /></span>
<span style="font-size: large;">Developing rules can be time consuming especially when considering specific formatting rules. </span><br />
<br />
<span style="font-size: large;">Its quite conceivable that a tool could be written atop of this API to code gen the rules, especially the formatting ones. This would be a great toy project. </span><br />
<br />
<span style="font-size: large;">As I write this I have just come across TSQL Smells by Dave Ballatyne</span><br />
<span style="font-size: large;"><a href="http://dataidol.com/davebally/ssdt-tsql-smells-add-in/" rel="nofollow" target="_blank">TSQL code smells in SSDT</a> </span><br />
<span style="font-size: large;">There are 43 rules (compared to the stock 15 or so in Visual Studio) you can utilise, it is open sourced and can be extended or used as a leg up for your own stuff. </span><br />
<br />
<span style="font-size: large;">One last advantage and compelling reason to use SSDT is the fact it will work with msbuild, with little or no extra config other than ticking a tick box and ensuring the library is accessible on your build agents, a large consideration when assessing your needs. This will ensure CI is subject to the same static code analysis as local development envs.</span><br />
<br />
<h2>
<span style="font-size: large;">In Summary. </span></h2>
<span style="font-size: large;">SSDT isn't the last word on SQL static code analysis, there are other tools available such as a Red Gate offering, SQL Enlight and SQL Code Guard to name but a few (and there really are only a few) but all of these have been judged (in relation to our needs), after recent research by my colleagues to have various shortcomings (from price, incomplete features, to lack of integration in Visual Studio) in comparison to SSDT. </span><br />
<br />
<span style="font-size: large;">Given all of this it seems that SQL static code analysis is still an immature area and hopefully one which will see more attention in the not too distant future, and hopefully I don't have to author too many rules....</span><br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<h2>
</h2>
<br />colhttp://www.blogger.com/profile/11580119772205731598noreply@blogger.com4