Testing Adventures

I love this job.

Wonderful people to work with – immediate colleagues, immediate manager, developers including their managers, all the way up to my engineering VP – all great. Senior management seems good (though I’m terrible at judging that sort of thing). The company products have enough relationship to things I used to work on for that experience to be useful, and are different enough that there’s plenty of new stuff to learn. I feel that I ought to know my current product much better than I do – but since I’ve spent more than half of my tenure so far *not* working on testing it, that’s probably an unreasonable self-judgement.

I’ve been allowed/encouraged/required to do things I’ve never done before, and that has all turned out pretty well. Some examples:

• We think we’re going to need serious GUI automation soon, have a look at this implementation with an in-house framework that we used for a few things. Sure! I don’t speak browser … but this looks way more complicated than I think it needs to be, let me see what I can do with something open source. Hey, Robot Framework looks promising … and after a few diversions, I’m most of the way through implementing a Robot-based replacement. I’m still not fluent in HTML and related stuff, but I get by.
• We’d like our regression tests to run from Jenkins instead of cron jobs, can you make that happen in a way that gets us better reporting? Well, Jenkins can show results for JUnit tests, which I don’t know anything about … but OK,  I made a wrapper for the pipelines to use, that (1) calls the legacy test framework and then (2) makes fake JUnit test results, so we can use Jenkins JUnit reporting. And my first-ever experience with PHP was to rework a little app for displaying legacy tests results into a version that displayed reorganized results in a Jenkins iframe dashboard.
• We need you to test this thing, and the legacy test framework isn’t a good match, and you’ll need this REST interface. I’d never done REST before, but Robot Framework also does REST, and that worked pretty well. That led to a couple of small REST projects and then to a major one, now well underway.

I’m having a blast. (Well, I could do without the pandemic, but so could we all.)

I have done a few little proof-of-concept GUI projects with Robot Framework, but – finally! – have something underway that I expect to go live.

Almost all of my product’s legacy test library uses an in-house command line test framework, which is entirely appropriate given the way this product works. But there is a little bit of GUI, embedded in a few of those tests. I’m re-implementing those tests with Robot and also implementing mechanisms for the legacy framework to launch Robot tests.

Fun stuff! The interface that needs the GUI automation is GUI-automation-friendly, so that making the Page Object file for it has been easy, and we don’t actually use a lot of what’s there so I can blow off implementing some of it unless/until we decide to use it. I’m cloning the legacy framework’s interface for calling the GUI tests, so the actual test scripts can be updated to use the Robot implementation with a simple global replace.

Why re-implement something that works? The technical debt to deal with the per-release tweaks has become burdensome. We need something that’s a lot easier to maintain, and Robot looks like a reasonable way to get that.

The existing implementation uses a (separate) in-house framework for Selenium-based tests, and that framework was developed for use with a different product. To me right now, that implementation seems like a poor choice for my product, but that decision was made years before I joined the company, and I think I understand why it seemed like a reasonable choice at the time. So, no judgement! People did the best they could at the time they had to decide something, just like I’m doing right now.

Whatever one builds will eventually become legacy, assuming it goes live in the first place, and it has to be maintained. Those changes will include fixes for coding errors, extensions to cover additional requirements, deletions of stuff no longer needed (hope so anyway), and “in retrospect, that was a terrible way to accomplish X”. The “in retrospect” part of that is important. When the immediate reaction is “what was I/he/she thinking?” – remember “it seemed reasonable in the original context”.

My current project is testing a new REST API to be used to integrate my classic product with a product from another part of the company.

OK fine. I’m not an expert with REST yet, but Robot Framework and the RESTinstance library for it is now a toolset I’m pretty comfortable with for functional testing. No problem there.

Thing is … for this project, nobody is worried about functionality and functional testing. It won’t be trivial to get good coverage for all the needed combinations, but we’re confident we can get all that stuff working for single threaded access to small-to-moderate data sets.

Everyone is worried about performance.

This project is meant for large scale deployment. Queries and data retrieval will be going after really large data sets, and while there will be only a small number of simultaneous users of this new interface, there will be much more extensive legacy queries and retrieval going on, hitting the same data at the same time. Resource contention – yeah.

So I’m actually back in familiar territory – load testing, early and often! Get a simple load scenario going as soon as you can get a single transaction to run end-to-end. Except we aren’t talking about phone calls, so the tools I would automatically reach for are not relevant. And though my product has a good performance test environment and tools, this is the first time REST performance has been relevant, so there’s no REST tooling to just pick up and use.

I remembered that about a year ago, I went to a talk about integrating performance testing a web site into the CI Pipeline for it, and the presenter showed us JMeter results. So I did some brief web searches, downloaded JMeter, and followed a getting-started demo to make it run a few “tell me what version you are” API calls into my lab system. Yeah, this does what I need it to, and I’m probably not going to need to go much beyond demo-grade complexity for my test scenarios, because I’m not testing a web site. Tooling problem solved, and as soon as my Dev gets the first search working end-to-end, I’m going to start bashing it. He likes this idea. I am very much looking forward to the demo I’m going to do for my project team.

This is why you go to professional talks about stuff you (currently) have no use for. When you’re suddenly faced with the need to do something you’ve never done before, you remember a presentation about it, and you have a starting point.

My group was in a lull of release-related work, so we put in some time on other needful things. Overhaul of regression test environments, and retire some of them. Some test suites needed maintenance, and some needed to be moved to a different execution cadence, and some needed to be retired. Tools work in preparation for near-future test requirements. And we wanted improvements in test reporting, and that one was for me!

Our classic product has a huge and valuable regression test library, implemented with an in-house testing framework. Traditionally it runs command line scripts (as cron jobs), and results get stored as database records. Reporting is via a few simple in-house applications that display test results based on those database records.

We wanted to move the official regression test execution to Jenkins, and get improved reporting for it. (I have a sneaking suspicion that the wish for improved reporting was partly triggered by envy of the reporting that the Robot Framework Jenkins plugin provides for my next-generation-product Robot tests.) Well, Jenkins can execute scripts just as well as cron can, so no problem with moving the execution. The information provided by the Jenkins plugin Test Results Analyzer looks useful, but that looks at data from JUnit tests. We’d also like to make it easy to find information such as “is this test failing only for certain platforms” and “what tests are reporting failures and who needs to look at them”. Constraint on all this: this work has to be fit in between working on releases, so improvements have to be incremental.

Migrating test execution to Jenkins is pretty easy – take the shell scripts that call the specialized testrunner cli scripts, and turn them into Jenkins pipelines. Yeah, that’s using Jenkins as a cron-clone, but it’s a start and we’ll improve that during the next window for making improvements. Better reporting is a better use of currently available time.

To help with being able to make improvements in small pieces then pause, I made a wrapper for the existing cli testrunner. The wrapper calls the testrunner, then “does stuff” to process the results of that testrunner execution. So, as we convert the shell scripts run by cron to Jenkins pipelines, change the testrunner execution lines to an equivalent execution using the wrapper. Then we can change what the wrapper does to process results without needing to edit all the pipelines.

First step for improved reporting: make the wrapper get the database records produced by its testrunner execution, and export that information as JUnit-format xml files. Then we can use JUnit test reporting available for Jenkins, such as the JUnit publisher and Test Results Analyzer plugin. Making that fake JUnit data turned out to be fairly easy to do, so now we have pretty charts of our test results history.

Second step for improved reporting: take one of those in-house reporting applications, and modify it so it only finds records for tests executed by the Jenkins user, plus rearrange the data presentation a bit, plus restrict the filtering options. Then add it to our Jenkins Dashboard as an “iframe portlet”, which basically tells the Jenkins Dashboard that you want it to display a specified URL. It was my first experience working with PHP, but really, when you start with something already close to what you need, no matter what the language is, some modifications are obvious, and you can look up syntax for the rest. I made it work.

So now our Jenkins Dashboard is a single source of information about the status of our actual regression tests, uncluttered by noise from experimental test execution. It is much easier to tell what is and isn’t reporting green, and who needs to investigate bug vs broken test. And, it was fun to build!

I have some ideas for improvements to the testrunner wrapper, next time I can devote some effort to that. And I’m sure there will be requests for other reporting stuff. More fun with infrastructure to come, but now, back to testing product.

Much of my experience with regression tests has been for code that was not volatile. I’ve built regression tests during the course of testing a feature, and found plenty of bugs during that process, but by the time the feature was done and I was done with it, future runs of the regression tests mostly didn’t expose problems. Occasionally there would be a feature interaction with something new, or a change in the way the feature was supposed to work, but I had gotten used to the idea that most of the value provided by a regression suite came from getting the suite to work in the first place. Some of that was due to the old-school development process which meant that I usually wasn’t able to work with a feature until it was complete or nearly so. But even when I was involved much earlier – once an automated test produced the desired result, it mostly continued to do so, and my Dev gave me advance notice of specification and implementation changes. “Hey, this doesn’t work any more” was quite rare.

That is not true for my current product. It is under active development, has a ways to go yet before it is a shippable product, and there’s a certain amount of “hey, this thing we implemented, it isn’t right, we need it to behave this other way instead”. So, as I implement regression tests for a major subsystem, the initial value is still detecting problems as I get test cases of the regression suite to work. But my suite has been running nightly against the latest smoke-tested build for about 2 months now, gradually increasing its coverage, and I’ve already gotten a few “hey, this thing doesn’t work any more” incidents. Once it was an implementation change that happened while I was on vacation, but some others have been “yeah, that’s a bug that got introduced”. Cool!

That might not seem like a big deal to many people, because duh, that’s what automated regression is supposed to be for. Well, it’s a new thing to me, and I’m happy that my suite is finding those problems.

Robot Framework offers support for data-driven test cases. – write the test once and make test cases by feeding it multiple data sets. You identify a keyword (aka function/subroutine) as your Test Template, then your test cases are just names followed by the arguments expected by that template keyword.

My current test project is data correctness tests (REST based, implemented using Robot) for a feature in our next-generation product – check that each user-defined instance of the feature produces the data expected for that customization. I could have implemented data-driven test cases from the get-go, with test cases named for the test instances, it wouldn’t have been too bad for the first dozen. But keeping that test case list in sync with my test instances as I made the expected several hundred more? That’s tedious, error prone, asking for trouble, and dealing with that problem up front will delay my first deployment. Get something working first, then make it better. I put off dealing with the data-driven test cases problem and made just one test case, with a FOR loop to check all the test instances, and got my initial tests into production.

OK, it’s time to deal with the problem, before I get to those several hundred additional instances. Data-driven test cases as Robot normally offers them would not work well. I figured that what I needed to search for was pointers on automatically generating Robot data-driven test cases for dynamic data? from a CSV file? That is not an easy thing for Robot to do, but yeah, other people have wanted it also. I found some stuff in forums, a blog post, and – score! – a library that does it for you: https://github.com/Snooz82/robotframework-datadriver

My test instances are stored as JSON data, and I have to rebuild that file anyway when I add instances to my test set, so just add a second step to rebuilding to parse that JSON file to find the names of the instances and stuff them into a .csv.

Using that library, I refactored the FOR Loop Of Doom into data-driven test cases automatically generated and named for each instance. It was EASY. The result is better reporting, more maintainable code, and easier problem isolation when a test instance produces an unexpected result. Thank you Snooz82 (whoever you are) for that library!

My first set of Robot Framework based tests have gone live, providing some data correctness regression for a feature of a still-under-development product. Those tests are implemented using REST, not GUI, which is not where I thought I was going with this stuff. Hey, priorities change, and sometimes they drastically change what you’re doing: deal with it. (And sometimes it turns out to be fun!)

I started learning Robot because I thought I’d need to do GUI automation for this new product, because unlike the current product, there’s business logic in the GUI. OK, let’s learn how to do GUI automation!

And then … the various powers-that-be, including my boss, had a collective reality check concerning time frames, priorities, how many people are available, and how much support is required for the existing product. Both current and new products exist to provide information to end users, and that information had better be accurate. So data correctness is THE important thing, for both. The current product has a huge library of data correctness regression tests of various kinds, implemented with the in-house command line testing framework. The new product needs data correctness tests, and automated regression for them, more than any other kinds of tests.

Drop the GUI automation stuff. GUI is not a good vehicle for data correctness automation. We’ll do GUI automation eventually, but not right now.

One class of data correctness tests can be automated using the same REST interfaces that the GUI uses. The in-house framework actually does have some support for calling the new product’s REST interface, which got implemented a while ago during a previous attempt to build this new product. So using that was an option, but I knew Robot has libraries for REST (just not as extensive as for Selenium), and I was really liking working with Robot. And my boss really liked the idea of us working with some new-to-us tools, so sure – REST automation using Robot, and a secondary output of the effort is that it will be an example to show other people in our group how to do it.

This is my first experience with REST and I’ve had a blast doing it. The “Exploring Service APIs through Test Automation” course by Amber Race at the Test Automation University https://testautomationu.applitools.com/ was very useful for me, and the RESTinstance library for Robot https://github.com/asyrjasalo/RESTinstance has been fairly easy to deal with. I uncovered a few product bugs while doing the manual verification for each of the initial test cases, and will likely find more as I add test cases. I am pleased.

I’ll get back to GUI testing eventually. Meanwhile my crude little demo project for Robot GUI automation for the current product has been useful – one of my colleagues started with that and turned it into something that is doing useful work for him, setting up stuff for manual verification. I am pleased.

Yesterday I did the real demo for my GUI automation proof-of-concept project – the one for my own group.

I explained a little about Robot Framework, and gave an overview of how it works using my demo test case for our Next Generation product. This is what a test suite looks like, and this is how you set variables, and this is how you make your own keywords. That product’s GUI is not very automation-friendly, but see, the tool can cope with that stuff. Plus it may not have to, or at least not as much, as we have a commitment from the GUI Dev manager to have his people make changes for us as we target the parts of that GUI that most need automation. Then I ran it, and of course seeing the browser with the “controlled by automated test software” banner executing the steps of the test was a crowd-pleaser.

Then I switched to the other demo I’d made, which targets the current product. The current product doesn’t have much GUI automation right now because it isn’t needed, because most of its GUI is a very thin layer over the interfaces we use for command line automation. But the GUI itself mostly is automation-friendly, so the sample Robot test case for it just needs obvious-looking SeleniumLibrary keywords with obvious-looking locator values. No xpath expressions, no Execute Javascript with weird incantations. It looks easy … because it is. And it runs on the product we’re all familiar with … that product may not need GUI automation, but there are things it would be nice to have automation for.

Then I switched back to the secondary demo test case for the nextgen product. That one demonstrates the data-driven test case style, where you execute the same test for multiple data sets. One data set is deliberately incorrect, so I could show the reporting and logging for a test failure. People liked that too.

We aren’t ready to start writing production test cases yet. The nextgen product isn’t yet ready for that, though I have a list of areas that will most need GUI automation. Also we need some infrastructure, like describing the directory structure for the Gitlab repo we’ll use for those tests, and writing coding guidelines, and establishing a Jenkins setup for test execution. Also I need to make a Robot custom library to call our command line test framework – I don’t know how soon we will need that, I am sure we will need it. But first, get acquainted.

Three hours after I’d finished my demo, someone had installed the tools, pulled the repo for the current product demo, and made and executed his own first test case, which sets configuration for a test he runs. Hey, no reason not to learn a new tool by making it do something useful to you personally.

GUI testing has never been my thing. I’ve done some – a page is missing this thing, and when I select that thing I can’t set all the options I should be able to, and this other display looks fine for a small amount of data but really terrible when I load my large data set. I’m not comfortable with GUI testing, and my understanding of web pages and the technology used to make them is pretty basic.

But I’ve just done a proof-of-concept project for GUI automation, and am now working on making it go live.

Back up a bit … my current product has a huge library of automated regression tests, almost all of them implemented via command line. There are a few automated GUI tests,and a Selenium Grid to run them on, but GUI automation mostly doesn’t make sense for this product. The GUI interface is relatively small and the GUI is mostly a very thin layer over the same interfaces used by the command line tests. Manual testing (both for new features and regression) for the GUI was a deliberate choice.

At present, I know enough about the current product to be useful doing regression testing for a release, but not enough to be able to efficiently do testing for a new feature. So while we aren’t regression testing a release, I’m assigned to the Next Generation product that is under development, and I’ve been doing proof of concept work.

One of the differences between the current and nextgen products is that nextgen has business logic in the GUI layer, so we’re going to have to write GUI automation. No one thinks the current product’s existing GUI automation is a good thing to build on, though it works well enough for its purpose. A while ago, I was given a copy of a proof of concept project that a colleague had done for a very early version of nextgen. It is Python, runs Selenium, and uses a homegrown framework that someone in another product group had made for use with the target platform, and I experimented with that for a bit.

Well … it actually performs a useful test. But it runs for maybe 40 minutes and then fails, then I restart it and it runs an hour then fails, then I restart it and it runs 15 minutes then fails. It is always the same failure at a different time and I have no clue how to debug the problem. (Nor does my colleague. I’m not dissing her, she did a great job with what she’d been given.) Writing more tests like it would be fine for the programmers among our group, but we aren’t all programmers. And a roll-your-own framework for GUI automation? For all I know that may be appropriate for the product this one was built for, but I don’t think it is needed for our product, and we certainly don’t need the maintenance overhead. I was pretty sure there’s an easier solution. (OK, I admit it – my initial reaction was “this looks way too much like work”, and then I thought about why I had that reaction.)

I remembered that during my job search I’d seen postings that specified experience with Robot Framework, Selenium, and Python, so I downloaded that stuff. Once I had everything set up in Pycharm, getting a trivial Robot test working was … trivial.

Here’s a quote from Joe Colantonio: “If you want to use Python for test automation, you can’t go wrong with the Robot Framework.“ https://techbeacon.com/9-top-open-source-testing-automation-frameworks-how-choose

I put that aside for a while, as there was a release to test, then came back to it and started work on the proof of concept test for the nextgen product. That was not trivial, as I discovered that a lot of the GUI is not automation-friendly. Unsurprising, as “must support GUI automation” has never been on the developers radar. But it was obvious from even a cursory reading of the Robot Framework SeleniumLibrary documentation that it expected to have to deal with unfriendly interfaces.

Here’s one of the selling points for using a mature open source tool. No matter what problem you’re having with it, someone else has already had the same problem, and an Internet search will turn up at least one discussion about how to solve it. If you’re working in an unfamiliar domain, as I am, it may take a while to figure out the correct search terms, but the information is out there somewhere. Robot has been open source since 2008, and was in use at Nokia starting 2005. (The Nokia origin gives me an irrational warm-and-fuzzy feeling that “this originated in *my* world, of course it can do what I need it to”.)

Between Internet searches and help from some of the local GUI developers, I found the correct incantations to deal with the unfriendly-to-automation GUI elements. I got my test working, and showed it off during the latest nextgen product “this is what we have working so far” demo session. It was very well received. When GUI automation is not on your radar, seeing your product’s GUI in a browser window with that “controlled by automated test software” banner up top is definitely cool. The bug I submitted for “please make this GUI more automation friendly” is now officially targeted for work as part of the product.

Plus, that demo uncovered the information that someone in my company who works on a different product has been using Robot for a while, for REST and command line tests rather than GUI. OK! Someone in-house I can talk to about building custom libraries. So far I haven’t needed anything except the SeleniumLibrary keywords, but I’m sure that at some point we are going to need to make Robot tests that interleave GUI and command line actions.

I think this will work for our group. Robot’s keyword-driven approach looks sort-of similar to the test case syntax used with our existing command line test framework, so it won’t spook the non-programmers among us. Our Devs won’t fix all the automation-unfriendly GUI elements any time soon, but we should quickly build up enough examples of how to cope that we won’t be slowed down much once we get past the initial ramp-up. I’m sure we’ll run into problems, but I’m also sure we’ll get past them.

I’ve prepared the detailed demo for my group, and I’m working on the “how to set up your environment” instructions. I have a list of what has priority for GUI automation for the nextgen product, though it is not clear when we will be able to start on that. The next step for me is automation infrastructure stuff, such as modifying the “launch test suite on Selenium Grid” functionality in our existing command line test framework so it can also run the Robot suites, and I’ll want to get Robot suites running via Jenkins as well.

Yes, I am feeling very pleased with myself.

I am also very happy to be working for a manager who gave me a chance to do something so far out of my previous experience.

I am happily settling into what I still (after 8 months) think of as “the new job”. By about 5  months in, I understood the current product well enough to be useful helping with regression testing a release, but not well enough to efficiently test a new feature. So – of course! – assign me to the Next Generation product currently under development.

The current and nextgen products both monitor the same data and produce lots and lots of reports about it, and one of the things any existing customer is going to do with a nextgen is point it at the same data their current system is looking at, and compare the reports. So we’d like to test nextgen using the extensive archive of automated data correctness tests we have for the current product. Those tests are all implemented as: run a test-specific report, export the report to CSV, and compare that CSV to an archived CSV file that was carefully vetted for that report when the test was first implemented.

First nextgen project: do a proof of concept for implementing automated tests for nextgen this way. Make a nextgen report that matches a specific current report, run the report and export to CSV, and compare to the CSV archived for the current report test. Complications: while both products can trigger “run a report and get CSV” from the command line, those command lines are quite different, plus, more importantly, the CSV files produced by the two products are different.

Part of the project was figuring out a procedure to compose the automated test cases for nextgen, each corresponding to an existing test for the current product. Yes, doable, and as expected, a manual but straightforward process for a human with brain engaged. (Also, still unclear if it will be possible to make equivalent reports for *all* those existing tests, but at least we can get most of them.)

As for executing those test cases – making the test framework run the new commands is pretty easy. The compare – well, fortunately, and yes it makes sense for what we do with the data, the observed-to-archive CSV compare is not line-by-line, instead each data section gets parsed into a data structure, and the data structures get compared. So, modify the test framework to parse the nextgen observed CSV into current-style data structures, and compare to the specified archive. The CSV differences include some trivial formatting stuff like how to identify the start of a data section, but also different database column IDs for the same data, the way CSV data is laid out for a certain kind of chart, and some database columns that don’t map directly – like “concatenate values for nextgen columns A and C to get the corresponding values for current column X”. The test framework is written in Perl, and CSV files are just text, and Perl was made for text manipulation, so I waded in and got something working. Fun!

I really enjoyed that. I was able to show how to go about building the nextgen report test cases to match the current test cases, and how to run them from the (modified) test framework the same way you run the current test cases. If the nextgen product is doing the right thing, the nextgen observed report data matches the current product archived data.

And then my working prototype test framework code got handed off to the guy who wrote the original code I started from, who understands all the special cases and other stuff I simplified to get that prototype working. So he’s reworking it into production quality test code, which he’s able to do a better job of than I can, while I go on to another proof of concept project. We’ll be able to compose and run those nextgen test cases when we need to.