cumulativehypotheses

mostly professional blather

On TDD and Double Entry Bookkeeping

leave a comment »

“Uncle” Bob Martin offers here a number of claimed parallels between TDD and Double-entry Bookkeeping (DeB). I think he’s rather missed important aspects of—surprisingly—both of those things. They aren’t primarily about correctness, although they do help with that, they are both more about knowing the state of your world.

That TDD is somehow like DeB is not an especially new observation, people have been making that comparison for a decade or more, but in the recent post Bob says that:

  • Both [TDD and DeB] are disciplines used by experts who carefully manipulate complex documents full of arcane symbols that must, under pain of terrible consequences, be absolutely correct in both type and position.
  • Both involve representing a long sequence of granular gestures in two different forms on two different documents.
  • Both techniques update their documents one granular gesture at a time, and each such update concludes with a check to be sure that the two documents remain in balance with each other.

This is…highly questionable. Bob uses the parallel to try to make programmers who don’t TDD with much enthusiasm (or even at all) feel bad, asking:

[…] why are accountants able to maintain their discipline so assiduously and so completely? Can you imagine a group of accountants saying to each other: “Guys, we’re really under the gun. We’ve got to finish these accounts by Friday. So, do all the Assets and Equities. We’ll come back and do the Liabilities later.”

No. You can’t imagine that. Or, at least, you shouldn’t. Accountants can go to jail for doing things like that.

With a suitable analogy to sloppy programmers. Also…highly questionable.

Suppose that your business chooses to use DeB, then…oh, but wait, that is a choice. DeB is not required of all businesses.

There are two gold-standard benchmark type things for financial reporting: GAAP (from the USA) and IFRS (rest of the world, ie most of it). The main point of these standards is to have common approach to reporting between businesses, to allow for comparisons, and a clear and accurate approach to accounting within a business; thereby to afford good management and avoid fraud. These standards relate largely to how the business is described in its published financial records. They do not, so far as I know, explicitly mandate DeB. However, for a business that’s much more than a lone sole trader it is hard to meet the requirements for financial reporting any other way. Not that a small scale Sole Trader would be reporting much, anyway, but that’s another story. I can be done, there is such a thing as Single Entry Bookkeeping and it is perfectly respectable, but its hard to extract from such records the kind of reporting that auditors, and even more so again markets, if your business is publicly traded, want to see. And that’s the important bit. Both GAAP and IFRS relate to what, and how, a business publishes about its finances.

People tend to view accountancy as a rather arid discipline, and tend to get hung up on the vast quantities of arithmetic required. But this is to miss the point. The accounts of a firm are just that: they give an account of what went on during a trading period. They tell a story. These invoices were settled, this capital expenditure was incurred, that provision for whatever it it was was made, and so on. The accounts tell the story of where the value of the company came from. The art of accountancy, and where it starts to overlap with management and finance, is how that story is told.

  • Do bookkeepers and accountants make mistakes? Yes. Do they go to jail for them? No. If they aid and abet deliberate fraud by the company, then they may do, of course. And if they get a reputation for making lots of errors then they will find it hard to secure work, and may eventually lose professional accreditations. But jail for honest mistakes? No.
  • Are a company’s accounts free of error at all times? No. At the close of an accounting period the accountant will prepare a “Trial Balance”, which is exactly what is sounds like—a first attempt at constructing a valid balance sheet. A balance sheet shows a net position for each account at the end of the period, taking all the transactions during that period into account. Is the trial balance bound to be correct because of DeB? No. There can be errors that do not break the accounting identity. Does anyone go to jail if the Trial Balance does not…balance? No. They just have to do a lot more work and hunt down the error. And recall that it is the Directors of a business that sign off and publish the financial reports of a business, not the accountants. And recall that in most jurisdictions you need two accountants: the every-day ones that help operate the business and the independent auditors who validate that this was done correctly. But it is still the Directors who sign off and publish the accounts.

Bob emphasises that DeB contains some built-in checks against error, he emphasises correctness. And at one level, DeB does help with correctness. If the fundamental accounting equation (the invariant: assets = liabilities + equity) is broken then an error has occurred somewhere. It could be a simple arithmetic error, and back in the day that was a very useful thing to be able to spot, or it could be a data entry error, and again, good to be able to spot that. But since the 1950s, starting with larger and extending now down to all but the very smallest companies, the arithmetic is fully automated and the data capture of receivables and payables and what-not is very largely so and the kinds of error that DeB flags all by itself are now very rare. The mistakes that DeB doesn’t catch are modelling errors.

A large business may maintain many, many accounts for different purposes. The “Double” in DeB can be misleading. A transaction is not necessarily recorded in in two accounts, but in at least two. Suppose that Consolidated Amalgamated wants to buy a new Turbo-Encabulator—those are pricey, so in addition to the (perfectly respectable) slight-of-hand in depreciating that capital expenditure they also sell a bond to raise additional cash. As a result of that one transaction a bunch of accounts will be updated: cash on hand, debt, perhaps a couple of kinds of short and long term asset, maybe others depending on how tricky sophisticated they want to be. Here are many opportunities for putting the wrong values in the “wrong”—that is, badly disadvantageous—places, even if the whole thing still balances out. Other kinds of check are required over and above the balancing.

DeB is more complicated than single entry and does need trained operators. If the correctness guarantees offered by DeB are relatively weak, why do it? One answer, and a very important one, is that with DeB the value of a company can be known exactly at any time. The invariant assets = liabilities + equity is true, in principle, all the time. Which means that the managers of a company, and its shareholders (if it has them) or other investors, can see the state of the company all the time. And, even better, can quickly assess the effect of a management decision or action on the finances of the company.

TDD does the equivalent for a codebase. It lets you find out its status at any time, and assess very quickly the impact of a change.

If TDD is like DeB, then we can ask: in TDD, who is it that fill roles comparable to those the auditors and the Directors? What is the equivalent of signing off the audited accounts?

Advertisements

Written by keithb

December 21, 2017 at 10:54 am

Posted in Uncategorized

Call for Volunteers: Programming Paradigm Competition

with 4 comments

[There are updates, and also useful discussion in the comments]

Join the Slack team for this competition where the baseline requirement is on the backlog. Get an email address to me somehow and I will invite you. PLEASE NOTE that the use of Slack does not imply any sort of real-time interactive collaboration between participants, or between participants and myself. Slack enables but does not require “chat”. If you choose to participate you will be added to the Slack team and, I believe, be able to see what conversations have gone before. 

Let’s say that you think it’s really important that people who work as programmers write their code in a certain way.

Whether this is for aesthetic reasons, or economic ones, or…actually the reason why you think this is not important. Only that you do. And, for my purposes, how you think they should isn’t important either. Maybe you think that they should only write assembler using pen and paper, maybe you think they should always use the very latest IDE to write dynamically typed OO code using TDD, maybe you think they should write using only pure functions and immutable data, maybe you think they should have rigorous proofs of correctness of every line of code, maybe…no, I don’t care. Only that you do.

Here’s the thing: advocates of one programming language, paradigm, toolset, what-have-you, always have really good explanations for why their way ought to be the best—for whatever figure of merit they say they care about—but we don’t really know. This should really be a topic for academic research but unfortunately no one is prepared to pay for the sort of study that would really settle the question, which would be astonishingly expensive. For practical reasons, what studies there are tend to be based on small cohorts of students who work on toy problems (code katas and such like), for a short time, once, having received a few lectures on whatever the techniques in question are. There is no reason to think that the results of such experiments should carry over particularly well to the work of seasoned practitioners.

But, maybe we can take advantage of the enthusiasm of advocates to do a semi-controlled, informal study using seasoned practitioners.

This is an idea that I’ve been kicking around on twitter with John De Goes. He likes a certain kind of functional programming. I don’t, much. I like a certain kind of OO programming and John doesn’t, much. What if, we wondered, some people who really knew what they were doing were to use their preferred approach to address a realistic—but small, we are talking about volunteers—problem under something like industrial conditions? What might we, they and the programming world at large learn about what the various programming paradigms do and don’t lead to?

“Industrial conditions” here means things like: that the scope of the problem grows and changes over time, that you—therefore—don’t know and can’t even in principle find out what the entire scope is, that the very goal of the exercises changes under you. If such volunteer expert practitioners were to work on such a system then we could look at what they have to do as those things evolve and develop and we could try to get a handle on how different programming paradigms—in the hands of seasoned practitioners—perform against a figure of merit of great interest in industry: how hard is it to adapt the code to new or changing requirements.

And this would be a back-office-y, line-of-business-y problem, because that’s so many programmers’ bread and butter.

I have such a problem in hand.

The Exercise

There’s a business domain that I know reasonably well: the scheduling of advertising copy into commercial breaks between TV programs on a linear-broadcast station. You can, of course, buy COTS software to manage this problem. Such products are vast in scope and mind-boggling in expense. But the central problem is—to begin with—simple. I think that this is a good proxy for a broad class of business problems.

Data stores can be file based, or use an integrated in-memory database of some sort, but no separate database products, please. Processing can be mostly batch, with a very minimal interactive command-line interface if required.

Product Owner

Here’s how it will work: I will be the Product Owner.

There will be is a Slack channel where stories will be published and volunteers will ask me questions from time to time about stories and I will answer, from time to time—so everyone can see both questions and answers, asynchronously. I do not expect to hold any synchronous conversations about this on Slack. In fact, to provide an equitable environment for all participants, I will not hold any synchronous conversations. Such answers will be—for the purposes of the exercise—normative.

There will be a backlog of stories, quite a long one, recorded as a series of additions to a text file in a private github repo. So that I can’t prejudice the exercise stories will be written far in advance and published from time to time along with the hash of the checkin which added them to the backlog. At the close of the exercise the repo will be made public and everyone will be able to see that the backlog wasn’t manipulated along the way to favour or disfavour any one implementation.

I won’t intensively test anyone’s code, but I will review the means they choose to use to demonstrate that their code implements the story and take a view on whether or not I’m convinced by it. But that’s just, like, my opinion. I would like to be able to (build and) run all the solutions using the most vanilla setup—ie, what the download of a recent version give me out-of-the-boxfor whatever the target language is. I’d much prefer to do that in a FreeBSD VM. Linux if I must. If you would like to use a .Net language, perhaps you could make sure your solution works with Mono? Maybe I can borrow a Windows machine from somewhere if not.

Practitioners

People will volunteer to be Practitioners will implement stories. Their code will be in a public github repo—sadly this does limit us to languages that have a sensible representation in text files.

Practitioners can work on the problem for as long or as little as they like—but should apply their nominated language, tools and approach with maximum sincerity. I’m more interested in seeing solutions done “right”—whatever that means in a Practitioner’s chosen paradigm—than done quickly. I can generate new stories basically forever—it’s a huge problem space—so we’ll run the exercise until no one is interested in doing any more of it, and then stop.

Solutions should include whatever the equivalent of a Makefile is so that I can build and run them locally.

Assessment

The figure of merit will be not how quickly, in real time, stories are implemented, this is not a race, but how small, proportionately, the impact of each new or changed requirements is on the already existing code.

Volunteer

If you’d like to volunteer to be a Practitioner, please reply to this posting and drop me and John a note on twitter.

There’s no reason why we couldn’t usefully have more than one Practitioner for a given paradigm, but it would be good to have at least one for each of:

  • Procedural language (eg C)
  • OO in a dynamically typed language (eg ruby, python) with TDD [this is the one I’d pick]
  • OO in a statically typed language (eg Java, C#) with TDD
  • OO of some sort without TDD
  • strict (and eager) dynamically typed functional-ish language (eg CommonLISP, Erlang)
  • non-strict (and lazy) statically typed functional language (eg Haskell) in a less polymorphic, more concrete style
  • non-strict (and lazy) statically typed functional language (eg Haskell) in a maximally polymorphic, most highly abstracted style [John likes this one]
  • Other (eg Prolog, Forth, what the hell: assembler)

Prizes

There will be one prize for the paradigm most clearly superior in the consensus opinion of the Practitioners as interpreted by the Judge (me). And that prize is GLORY! 

The Judge’s decision (ie, mine) is final, normative, and nugatory.

FAQ:

  • Am I biased? Yes.
  • Do I stand by to be pleasantly surprised? Yes.
  • Do you have to take my word for that? Yes.

 

If this does come off, I’ll attempt to find an academic interested in doing something with the repos.

Written by keithb

October 27, 2016 at 5:03 pm

Posted in Uncategorized

Special–ism

leave a comment »

If there is a common theme to this “Agile” journey that you’re almost certainly on if you’re reading this—and there might not be—then a good candidate for that could be a certain series of unpleasant, ego–challenging realisations.

It began with programmers, back in the late 90s, as eXtreme Programming took hold and the unpleasant, ego–challenging realisations dawned that: the common old–school programmers’ preferred mode of engagement with users—pretending that they don’t exist—and their preferred mode of engagement with their employers—surly passive–aggression leavened with withering cynicism—and their preferred mode of engagement with each other—isolation interspersed with bouts of one–up–manship—and their preferred mode of engagement with the subject matter at hand—wrestling it into submission via the effortful application of sheer individual intellectual brilliance—weren’t going to cut it any more.

Over time, the various other specialisms that go into most industrial–scale software development—and IT systems work more generally—have had also to come to terms with the idea that they aren’t that special. With the idea that locking themselves into their caves with a sign outside saying “Do Not Disturb, Genius at Work” and each finding a way to make their particular specialism into the one true bastion of intelligence—and the true single point of failure, and that not seen as a bad thing—is not going to cut it any more.

A cross–functional, self–organising team will of course contain, must contain, various individuals who have, though education, experience or inclination, a comparative advantage over their colleagues in some particular skill, domain, tool, or whatever. It would be perverse indeed for a cross–functional, self–organising team not have the person who knows the most about databases look first at a data storage problem. And it would be foolish indeed to let that person do so by themselves—at least pair, maybe mob—so that the knowledge and experience is spread around. And it would be perverse indeed for a cross–functional, self–organising team to make a run preventing any one member of the team having a go at any particular problem merely because they aren’t the one with the strongest comparative advantage in that topic. And it would be foolish indeed for such a team to not create a physical, technical and psychological environment where that could be done safely. And so on.

Different disciplines have embraced their not–special–ism at different times and with different levels of enthusiasm. “Lean UX” represents the current batch, as designers get to grips with the idea that they—in common with ever other discipline—turn out not to be the special and unique snowflakes uniquely crucial to the success of any development endeavours. Where is your discipline on this journey?

Written by keithb

May 18, 2016 at 2:47 pm

Posted in Uncategorized

TDD as if You Meant It at Agile Mancs 2016

leave a comment »

Please share your experience in comments to this blog post. Would be most grateful if you also shared your code. Thanks.

Written by keithb

May 12, 2016 at 9:22 am

Posted in Uncategorized

Corrigendum to Lightning Talk at Agile Mancs 2016

leave a comment »

Wrong Theorem

We’ll see on the video, I guess, but I think I said “Mean Value” when I should have said “Central Limit”. My mistake. It had been a long day, and they were handing out beer.

On the β-distribution

If you are going to try that sort of Monte Carlo estimation for yourself you don’t need to wrestle with the full-blown β-distribution, which is a bit of a beast. You can approximate it well enough using a triangular distribution. The base of the triangle is the closed interval—time, money, whatever, I’ll talk about time—between, as Dan North suggested, that time which if the work took less than that you’d be very, very surprised and that time which if the work took much longer you’d be very, very embarrassed. Somewhere in that interval—and very rarely in the middle—is the time that you think it’s most likely for the work to take* and the apex of the triangle lies above that point. The altitude of the apex must be such that the area of the triangle is 1.

Have fun!

Getting to not having to do any of that

Being asked for a estimate is, it’s true, a sign that your interrogator doesn’t trust you.

Usually this is not because they are monsters who relax every evening with a well-thumbed copy of The 48 Laws of Power and a small glass of chilled kitten’s tears. Usually it’s because, firstly: they have suffered a long history of paying people to develop software for them and not getting much back in return, and secondly: they don’t know you from Adam or Eve.

In a better world, they would trust you, because your heart is pure and your intentions good and wouldn’t if be great if we could all just get along? And it would.

Until that world arrives we will have to deal with suspicious, hard-bitten CFOs who have a keen understanding of their fiduciary duty. They—or their subordinates—will ask you for an estimate. Some of them will want then to interpret your estimate as a commitment, perhaps with contractual remedies for failure to meet same, and so hold your feet to the fire to deliver exactly against those estimates. I strongly recommend not doing business with the ones that cleave tightly to such a position. They are setting up themselves, and you, for failure.

But many will be amenable to understanding that, as McConnell says, the point of estimation is not to predict outcomes but to see if you are in with a chance of managing your way to success.

How to break the cycle?

Deliver value. Early, often, reliably. Very early, very often.

With contemporary engineering practices and tooling this can now often be done very quickly. Even in those “heavily regulated” environments. When you have—and maintain—a reputation for delivering value then your conversations with the people who’re paying you to do that change dramatically. From: “how much will X cost?” To: “how much do I need to pay to have you keep on doing J, P, R, S, Q…?” That’s much more agreeable for all concerned.

Stepping back

YMMV, but I’ve found, while doing and managing software development work in and for VC funded local startups, publicly traded global blue-chips, privately owned product companies, government departments and many other scenarios, on  four continents, that when something about the scenario says “new”, whether it’s that you are a new supplier, or they are a new CFO, or whatever, then requests for estimates—maybe with some story about “just for budgetary purposes” attached—will come; but once a reputation for reliable delivery of value is established they fade away, until something changes that takes us back to “new”.

It would be nice if we didn’t have to erect this infrastructure of estimation every now and again, and you can if you wish refuse to work under such terms—you may then find work unpleasantly hard to come by—or you can invest in re-educating people not to ask those questions—but the first rule of consultancy is not to solve a problem you aren’t being paid to solve—or you can roll with it, demonstrate the simple lack of necessity of estimation in an effective organization and wait for the next cycle which often will be less severe than the previous one. And so on.

Good luck!


* Whenever you are asked for an estimate, for anything, you should reply with at least these three numbers—or their equivalent—and if that shape, never mind the content, of answer is not acceptable then you have discovered that you were not being asked for an estimate.

Written by keithb

May 12, 2016 at 9:19 am

TDDaiYMI XPManchester

with 3 comments

Please report on your experience with the session in comments to this post. And, if you’re happy to, please post a link to your code.

Written by keithb

April 14, 2016 at 4:36 pm

Posted in Uncategorized

Some Mass Transport Metaphors

leave a comment »

One day the software development industry will be gown-up enough to talk about what we do in its own terms, but for now we have, for some reason, to use metaphors. And we love, for some reason†, transport related ones. A recent entry is the (Agile) Release Train. It’s a terrible metaphor. Here are some other mass–transportation metaphors for how you might organise the release schedule for your software development activities, ordered from less to more meritorious in respect of the figures of merit batch size, and the cost of delay if you miss one.

In these metaphors, people stand for features.

  1. Ocean liner—several thousands of people move slowly at great expense once in a while. Historically, this has been the status quo for scheduling releases.
  2. Wide–body airliner—several hundreds of people move quickly, a couple of times a day
  3. Train—several hundreds of people move at moderately high speeds several times a day

Some failure modes of the train metaphor…

Although the intent of a “release train” is that it leaves on time no matter what, and you can get on or off at any time until then, and if you miss this one, another will be a long in a little while, in practice we see attempts to either:

  • cram more and more people onto the train in a desperate attempt to avoid having to wait for the next one, à la rush-hour in many large cities, or
  • add more and more rolling stock to the train to avoid having to run another one

More generally, what a metaphor based on trains will mean to you may depend a lot on your personal experience of trains. For my colleagues in Zürich trains are frequent, swift, punctual, reliable, capacious & cheap. For my colleagues in London, not so much any of those things…

  1. Tram—several dozens of people move significantly faster than walking pace every few minutes

† Actually, that might be because for quite a lot of the the industrial period, say from about 1829 to about 1969, various forms of transport were the really hot technological stuff of the day.

Written by keithb

March 29, 2016 at 11:22 am

Posted in Uncategorized