Testing Adventures

Some years ago, I inherited an arcane testbed. It was based on a test tool that was already end-of-life, plus no other testers in the company knew anything about it,  and it used a dedicated configuration that could not be regenerated, only backed up and restored.. Also it was not practical to use with  the local automated testing framework. For these excellent reasons, my manager wanted to junk that testbed. But, my Devs really liked the testing I did with it, it was a primo tool for uncovering a certain class of bugs in an important subsystem.

Solution! There’s a another test tool which can provide the same functionality as the end-of-life one, the company owns several units of it and maintains paid support for them, and it is an accepted tool for use with the ATS (multiple testers have written test modules that use it). I should re-implement that testbed that no one else can use, as an ATS-compliant module based on that other tool, and then we can junk the old one.

OK, I did that … and discovered that although the tests appeared to be doing the same thing, the 2 testbeds uncovered some different bugs. The implementation details were just different enough to matter – different testbed configuration, differences in how the calls were established, and especially differences in how the 2 tools implemented “detect an audio tone at a specified frequency”. The arcane testbed remained in service along with the new one.

So before you discard a test that seems to be a duplicate, to a little checking to see if it actually is adding value.

Recently I had a plumber in to do a minor toilet repair, the sort of thing that YouTube videos had convinced me that even klutzy me could fix myself – if only the shutoff valve for the water supply had not seized up. I tried the suggestions for unsticking the value, no joy. So my options were (1) try to apply more leverage myself, with some sort of wrench (2) get my husband the Strong Guy to do that or (3) call in a trained professional, who knew how much force could be safely applied, and had the training and truck full of equipment to cope if something went wrong. I went for (3) because though I’m pretty sure Tom could have made that handle turn, I was not willing to risk him (or me!) breaking the pipe and flooding the house. (Long ago, exasperated with him for some reason, I wailed “what planet are you from?”, and without missing a beat, he replied “Krypton”. Right.) Plus, I had no sense at all of the actual probabilities for valve getting unstuck vs broken pipe, so I arbitrarily assigned “high” to the broken pipe scenario.

When I report a bug, often it is pretty obvious how urgently it needs to get fixed, but not always, and that question is harder to answer if the bug in question is intermittent. What’s the worst consequence if a live customer hits the bug, and how likely is that to happen?

The problem with the “how likely is it to hit” issue is that it can be hard to judge, customer usage patterns can vary a lot. And even if the probability is once per umpteen gazillion, well, when dealing with modern high volume transaction processing, that might mean you hit the problem twice a day.

Worst consequences should either be obvious, or understandable after a little thought. If the immediate reaction is that the worst consequences aren’t all that bad, consider thinking about it a bit more, especially in regards to combining with other possible unfortunate events or conditions. (Like, bug leaves a valuable resource exposed, but only on the internal network … but if the internal network has been cracked, is that resource a likely target?)

Most postings for tester jobs these days call for test automation as part of the job description. Test automation is code. (Duh!)  It is code that does not ship, and might not need to be particularly efficient, and its development process may be lightweight, but still, it is code that is expected to perform its intended purpose, will be checked into source control, may need maintenance, etc.

That’s the same sort of thing the Devs do. So if a company does coding interviews for Devs, expect them to also do coding interviews for their testers, who write code.

All the advice that I’ve seen and heard about doing coding interviews emphasizes that you’re supposed to ask questions about the problem you’ve been given, and talk about what assumptions you are making and what you’re thinking about as you go through the problem. You might be forgiven syntax mistakes if you make it clear that you have a reasonable implementation in mind.

To that, I will add – offer test cases. After all, testing is what we do. In fact – try to write the test cases first. If you’re given a problem with sample output, spare a moment’s thought for trying to come up with another case or 2, preferably demonstrating correct handling of an error or an edge case. And do pay attention to that sample output – does it actually show the data asked for by the problem statement? (I got a coding problem where that wasn’t true … clever!)

The lure of Jenkins was too much to resist …

So now I have a toy REST application, made by copying a tutorial and making a few modifications, and a few tests for it – a couple that are simple pytest, and a few more that are Tavern tests for the REST API. The tests are just barely more than the equivalent of “hello world”, but they and their app are in GitHub. I can execute them by running py.test from the command line, or by invoking py.test on the test directory from Pycharm. I can execute them by launching my AWS instance, pulling the whole package from GitHub, and running the application and tests there. Or – I can tell Jenkins to scan my repository, and it will pull the latest code, run the application, and then run my tests.

Enough infrastructure. Must practice writing tests with Python.

Jerry Wiesner’s famous “Getting an education from MIT is like taking a drink from a fire hose” quote has been on my mind recently. (And yes, it certainly was like that.) I’ve set myself the task of learning Python, and how I as a tester might use it.

The language itself? Straightforward enough, I have a couple of hardcopy books, and the internet is full of Python tutorials and interview-type coding problems that can be worked in Python. That’s plenty of material for the practice I need.

On to how I might use it, and that’s where the fire hose got turned on. In Python, I made a simplified version of functionality that a previous employer’s automated testing framework had implemented in Perl, and wow, even allowing for the simplification, I’m pretty sure that framework would have been much easier to deal with if they’d built it in Python instead of Perl. I started working with PyDev in Eclipse, and getting acquainted with the Python unittest module, and that led to pushing code to GitHub, and then pulling it on an AWS instance for execution. 

I wanted to get an idea of how to test a REST API, and that means I needed a REST API to test – right, the tutorial I found for that uses Flask to make an application, and then shows you how to package it in Docker. I might as well do my experiments with extending that application and testing it from Pycharm. Ok, I like Pycharm better than PyDev. On to testing that REST application – there’s pyresttest, but that doesn’t quite look quite right, but hey, there’s this thing called Tavern! It’s pretty new but looks like I can make it do what I want, and you’re supposed to use it via pytest, and yes, pytest looks like better support for what I’m working on learning than unittest, and Pycharm can be made to run pytest for me.

There’s lots more out there,  all this stuff has fascinating references to other things that I’d love to pursue, but I really think I’d better stop collecting tools and work on learning more about what I’ve got so far before I add more. Though I’d like to get that REST application and tests for it into Jenkins … 

I used to trigger a perl script on Friday evening, which would run for about 60 hours. It controlled a test tool to make phone calls, and about once per minute changed configuration to change the nature of those phone calls, and accumulated a file reporting results for each configuration. That sounds like an automated test, but local custom classified it as manual because it was not run using the standard test framework, which among other things reported results into the test management system.

When testing a telephony server, the question isn’t “automated or not”, it is “what/how much are you automating”. You have to test with tools, and those usually are not actual phones. OK, if the product is some sort of Interactive Voice Response system (“If you know your party’s extension, please enter it now. For appointments, press 1 …”), you can do a lot of testing for that IVR UI with a phone. But if the test target is something like “does our product correctly identify a particular protocol header, and process valid data, and detect and handle invalid data without crashing the application or allowing a security violation”, you’re going to be writing test tool scripts, Depending on exactly what you’re doing and why, you might stop at writing and running those tools scripts, or you might make other scripts to figure out if the right thing happened after each test tool run, and another script to run all those scripts, And then you can then get into testing framework territory, and have one-touch execution to configure your SUT first, then run each tool script and its results checker in turn, and maybe write test case results to your test management system.

Automated results checking is generally much harder to get right than automated test execution. If for whatever reason you can’t write a good automated results checker, consider the consequences of providing a bad one. Reporting failures that aren’t will likely get your test junked pretty quickly, and reporting test passed when it shouldn’t, such as “got the expected log message, but didn’t notice that the application then dumped core” – well, that’s really bad.  Maybe you need help in order to write good results checking. Expert advice, or at least someone else to take a look? A tool you don’t have? More time allowed to work on it? Maybe your test is valuable enough to make automated execution with manual verification worth doing, at least for a while.

Twitter pointed me to this link http://simplythetest.tumblr.com/post/169623787610/test-developers-arent-testers.  In this context, “test developer” means “person writing automated tests and/or infrastructure for automated tests”.

The author’s points are that these are 2 different roles, they have different mindsets, sometimes different objectives, and different skills. Some people can do both, or either, and that’s great, but some people in some organizations think they are interchangeable, and they are not.

Yes, true … but I think there’s more of a continuum than division. I found it interesting that the author seems to think that one person doing both roles is a small-organization thing. Well, I worked for a large-ish company, and when testers were assigned features, one of the deliverables was an automated regression test module for the feature, submitted to and accepted by the regression test team. There was a testing framework (homebrew, but with an actual dedicated support team), so most of the time, building that module was a pretty straightforward programming task. And if we needed something the framework didn’t support, we could make our own local edits to that framework while the framework support team worked on the needed enhancement or fix.

Maybe I see more continuum because of my experience? I’ve been testing telephony servers, and for that, even manual tests usually involve some sort of scripting because you need a tool to make the kind of phone calls you need. (Only occasionally is that tool a phone on your desk – and when it is, it is usually not the phone you use to order a pizza.) So the real question is how much automation is there – run a tool script directly, package the tool script in another script, make a script that both runs the tool script and checks the System Under test for results, or make a program that sets up your test environment and implements multiple test cases (run tool script and check results for each) and reports those results into an archive.

I love finding articles that make me think.

The January SQGNE talk was “Proactive Software Quality Assurance (SQA)™ Overcomes SQA’s “Traffic Cop” Resistance”. Interesting … I know that there are organizations where Devs view  SQA (under whatever name) as an obstacle, and I consider myself fortunate to have not worked for any of those.

For me, the useful thing about this talk was the reminder of all the trouble that comes from different definitions. Robin devoted quite a bit of his time to demonstrating that a roomful of software QA and test professionals produced a lot of different answers when asked to define “quality” and “quality assurance”.  I’m guessing that at a lot of companies one would get similar diversity of answers, asking the same questions of developers and QA and their managers, and that a little probing would expose some different expectations about what the QA people were supposed to do

“Done” is another word where different definitions can lead to problems. I remember a time long ago when a Dev I was working with declared that the application. was done Yes, I had done functional testing and he had checked in – but the application was not yet packaged so it could be installed, and it had no help text. So maybe he was done, but that application was not.

Remember to check definitions.

Resolution for 2018: get back into the habit of reading blog posts and articles about testing.  Starting today!

writing documentation for code

This makes some interesting points: (1) Naming practices are your first line of documentation (I knew that). (2)  Code comments that explain why something is being done are more likely to be useful than comments explaining what the code is doing. Usually the code itself is its own best explanation of “what”, unless the code is doing something obscure such as using an interface that is poorly or incorrectly documented. I think that’s a good guideline, although, thinking about test case automation, I think that may have more need for “what” type comments. If you have cause to assume that the maintenance on automated tests is likely to be done by junior people with limited understanding of both the product and the feature that just changed, then some code that is self-documenting for author-level experience may need explaining, and you may need comments to tie pieces of code back to test case steps – although possibly those could be considered as “why” type comments. (3) Unit tests are excellent documentation. I never thought of it that way, but Yes!

I go to some Software Quality Group of New England http://sqgne.org talks, and generally make a point of going when Johanna Rothman is the speaker. I used to work for JR, before she became a management consultant, and (1) it’s always great to see her again and (2) she’s a good speaker, her talks are always interesting and I learn something.

Stuff I learned: there’s a programming technique called “mobbing”, which is similar to pair programming, except more people are involved, and you need a large display so everyone can see. The resulting code still has to get code-reviewed, preferably by someone who wasn’t in the mob, but I can see that this ought to lead to code that all the participants understand very well, so they don’t have to go through the pain of trying to understand it at some later time. Also, “Cucumber” is a tool for doing Behavior Driven Development, and it needs to go on my list of things to do self-study on. (That list keeps growing …)

The talk itself: http://sqgne.org/presentations/2017-18/Rothman-Dec-2017.pdf , “Testers and QA as Agile Leaders”. The announcement email included this statement, “Acting as Agile leaders represents an important but possibly controversial change for QA/Testing folks, especially if you’ve been stifled by Agile projects.” Now, I’ve never worked Agile, though I’ve read about it for years, and I’d love to try it. From what I understand, “testers acting as Agile leaders” shouldn’t be controversial, or a change for that matter. If you’ve got a real self-organizing team that’s delivering working software frequently while dealing with changing requirements as an expected thing, and includes testing specialists who are effective, those testing specialists ought to be right in there contributing in the ways JR’s presentation describes. Maybe people aren’t accustomed to thinking of that as leadership?

And what’s that about some testers being “stifled” by Agile projects? I know that attempts to implement Agile have varying success rates, but I would think that would be a sign of “not a good implementation of Agile”. JR’s presentation does mention that expectations shape reality. That goes both for your expectations of yourself and for other people’s expectations of you, so maybe a tester in that situation needs to work on both? And that just triggered a memory of a presentation I was doing, many years ago, to some developers from a different part of that company, about what we, their new testers, might do for them. At some point one of the Devs said something like “testers aren’t technical enough to do that”, and my mouth dropped open and I said “um, most of us do have Computer Science degrees”*, and that Dev’s mouth then dropped open. It had not occurred to those Devs that testers could do stuff like code reviews, because they had never before had testers who could, and it had not occurred to us that we couldn’t.

*and I thought, but carefully did not say, “and mine’s from MIT, where’s yours from?”