Wednesday, December 30, 2015

Polarities of test data

I've recently been testing a complete redo of our reporting functionalities, and all in all surprised in how is it possible for a pair of developers to think it works when it does not. Even when there is a clear oracle in form of an existing previous implementation you can test against.

Testing this redo as if it is new functionality but with a simpler oracle has lead me to create simpler test data. I first handle just individual pieces and then move on further into combinations. The main principle guiding me is control: me in control of the data, understanding the basic before going into the complex and complicated.

My testing of the reporting functionalities was interrupted by a release with just a few little updates. All the updates were related to upgrading components, both 3rd party and our apps shared components. These usually cause specific types of problems, so I run a set of explorations around basic scenarios but this time, did not pay attention to data much. I used the data I had created for the reports, simple and controllable.

And a bug escaped us: there's a grid component that we for purposes of one view overloaded for height calculation, ending up with a problem that in other places, scrolling would fail. A classical (for us) mistake of one developer working with the component in one place tweaking it to be for that purpose, and it having trouble then when used elsewhere.

For me the interesting lesson was on data. If I had been on my typical data, I could not have avoided to see the bug. But since I was in a narrow and limited, controllable set of data, it hid the problem.

With continuous delivery though, the problem was shortlived. But it lead me to create two specific sets of data to reuse as part of my checklists. There's never one, but I can try doing smart selections of what I keep available. 

Monday, December 14, 2015

Value in contributing or learning - when to automate tests

I listened to a podcast with Llewellyn Falco today. There's many things of interest, but one worth an inspiration for this blog post.

In one point of the discussion, there is a discussion about failures. The first story Llewellyn tells about failure is having five developers work for a week on programming when the end result of that task is equivalent to 6 hours of manual one-time work. The story is about asking of value in advance. Good lesson.

I have had the privilege of many discussions about unit testing and test automation with Llewellyn. The story reminded me of my perception: with test automation, I often bring forth the discussion about real need of repeating and value of spending hours and hours in automating something I don't care to repeat but could do in seconds. But while the value seems like a relevant driver for me, the point that I feel we end up with is doing things anyway just to learn if (and how) it can be done. Value of learning (and solving a challenge) goes over the value of using time on testing.

There's a rule for unit testing Llewellyn cites a lot: Test until bored. As long as you are contributing to the overall value that testing could provide immediately or in long term, you are not bored. As long as you are learning, you are not bored.

I'm puzzled with this: I can always name many things I intentionally don't test, as there is never enough time for testing it all. I'm painfully aware of the opportunity cost of writing documentation or creating test automation over testing just a little more from the endless list of ideas.

I see my teams developers approach unit testing avoidance with similar arguments that I use on selecting what is worthy of automation on system or feature level. It makes me appreciate the approach test-oriented programmers like Llewellyn take: the challenge of learning is worth the effort.

But I still can't help but wonder: why the same rule of overall value wouldn't hold in the domain of testing as for other problems we solve with programming?

Perhaps because: 
  • adding the first test enables us to add more tests like that with a cheaper cost - repeatability is the solving a particular problem in a particular way in addition to repeating the same test by means of automation
  • if we don't automate because of cost-value discussion (with unknown cost), it's just another trick in the bag of excuses for us
  • we don't really know to estimate the cost before we have a solution, until then we can only discuss if the problem is worth solving at all 
  • the discussion about the unknown cost can take more effort than just doing it
The core difference in the story in the podcast and our common experiences of automating tests are in knowing how the problem can be solved. Perhaps the rule of "contributing or learning" applies in these examples. 

Thursday, December 10, 2015

Birds and statues - or phrases to keep to yourself

There's a conflict emerging - on merge conflicts and how to deal with them.

Sometimes I wonder why I care - I'm a tester after all. But I do care, since the quality of the code matters to what kind of results we can achieve as a team with help of testing. I care enough to nurture an atmosphere where refactoring is not only allowed, it is encouraged. We're not trying to stop change, we're trying to flow with change. And we've been relatively good at that, doing continuous delivery without test automation. Clean code is easier to work with. And in most cases, we have pretty similar idea what is clean code.

It seems there's an exception to every rule. My exception comes in the form of different beliefs. There's one (with 20+ years of experience) who believes that the things I've taught to believe about team work and clean code are incorrect. That we should find ways of working with code by one individual only to avoid merge conflicts - and human conflicts that result in changing the status quo.

When I suggest we wouldn't have a conflict if we paired or mobbed, I'm not being helpful. So I try not to mention that while I still think of that - a lot.

When I suggest we would have less conflict if we had smaller methods and that refactoring would be a good idea, I'm told we should stop refactoring all in all and always just write new code. We can just add fresh branches, leaving the old that worked there as is, still working. And that code style and cleanliness is just an opinion anyway.

When I suggest doing smaller increments that can be released to contain the conflicts, I get a shrug. And I get a bunch of others saying how good a strategy that is, but also remarks on how this area is just different.

When I ask what we could do, I hear we could just work on different areas completely, in isolation. To avoid merge conflicts - and human conflicts. It's worked for decades, what would be different now?

There's a phrase that I managed to keep to myself that I've mentioned before this all became urgent and pressing: be the bird, not the statue. I heard this at Agile 2015 from Arlo Belshee and Llewellyn Falco. The one who stays put is the one who gets hurt in merge conflicts in the modern software development. But saying that right now might be again not helpful.  But I think of that - a lot. And I admire the conflict-averse other developers, who increase their birdlike features in all three dimensions of how they deal with the shared code, leaving one statue there to realize implications later.