The Possibilities are Endless...But Your Tests Shouldn't Be

Many years ago now, I was fortunate enough to attend a talk by Cem Kaner about testing. It was a defining moment in my testing ideology's development. The talk involved several points, but a majority of it focused around return on investment.

As it turns out this appears fairly often in the testing field.

This leads into the discussion of test automation. Test automation can have a great ROI, but not always. You can end up spending hours, days, or even weeks automating low-risk and low-importance parts of your project with intricate tests that break constantly, when a manual test would have been far simpler, easier to maintain, and freed up time to grapple with more important problems.

Here's some things to keep in mind for ROI in automating a test:

• How often should the test be run? If it's a throw away test or an exploratory test, it probably shouldn't be automated (though some of the setup can be).
• What is being tested? If you're checking for text on a page, then it's almost always a good candidate for automation. If you're checking that screen elements are in their proper places, or that audio quality is where it should be, that is probably better done with human eyes.
• Are physical devices or objects required for the test? For instance, in a login, is there a One Time Password generator that has to be used? If so, you may be able to write a simulator (or disable it for some tests), but at some point, it's best to use the actual device to make sure that nothing is wrong.
• How fragile is the test? Some tests are fragile because the tested area is still under development. It may be best to re-prioritize that area for later. Other times, the tested object may have some non-deterministic quality that requires constant adjustment. In this case, visual verification may be best.

If you have any other good tips, comments, or disagreements, I'd love to see them in the comments below!

Security, or 'What you have, what you are, what you know'

After reading this blog post by Laurie Gavin, I felt I should post something in response considering my limited time in the security world with a former company.

Learning your 1, 2, 3s

Most people log in to websites or their computers using a 1 factor authentication system. Usually, this is your username and password combination. It's a 'what you know' authentication factor. Technically, it's two pieces of information, but that doesn't really make it more secure.

Issues  with 'what you know' systems:
They're guessable or brute-forcible in some circumstances
They require commitment of memory from the user
If reused, weaker systems can be used to get data from stronger systems

Biometrics, as mentioned in Laurie's post, are usually used in 2 or 3 factor authentication. They represent the 'what you are' factor. This is generally a bit more secure (assuming it is implemented correctly), seeing as you hopefully rarely leave pieces of yourself at home.

Issues with 'what you are' systems:
Your body changes over time, this can cause false negatives
They can still be hacked, though it is more difficult.
There may need to be workarounds for those without the required body parts (missing fingers, hands, or even eyes).

One-Time Passwords, smartcards, and other tokens form the 'what you have' security factor. This is usually a physical item (though can be a bit of software for soft tokens) that the system scans/reads. These sorts of authentication were designed to help prevent someone half the world away from impersonating you.

Issues with 'what you have' systems:
Physical objects can be lost, stolen, or damaged, requiring replacement
Sometimes they can be duplicated (or the data on them duplicated)

Note that any of these factors have vulnerabilities, but using 2 or more provides a layered authentication system, requiring more work to get into your account. If you ask me (and you probably didn't, but I'll tell you anyway), we need to move to at least a two-factor authentication system for any important accounts. There have been far too many leaked passwords to make me comfortable with 1 factor authentication anymore.

For more reading.

Myopia, or Why We Missed That Bug

Back in college, I had a roommate named Matt.

Matt and I both played an MMO called World of Warcraft (who knows, maybe you've heard of it?)

The interesting thing is that we both played it two completely different ways. I used the mouse almost exclusively, and he almost exclusively used the keyboard.

He was shocked I could play it that way (and pretty effectively, as he can still attest).

The blind spots we have in how we go about doing things with computers (like our gaming predilections above), can have a big impact on how we test. I use the mouse far, far more than I use the keyboard when given the option, and it reflects in how I test applications (I click buttons to submit forms, rather than using the enter key), and not giving proper thought to it can cause issues.

I'll give an example:

A while back, I worked on testing a piece of software developed for Palms (so definitely a while ago now). Each screen had many, many tiny drop-downs and text entry fields, most of which I couldn't reliably hit with the stylus during testing. After complaining about this to someone else on the team, it was pointed out to me that I was not the target audience.

I was working on software that would be used by surgeons. As it turns out, they tend to have a much easier time with precision pointing. My entire context for why I thought of the user design as bad was flawed.

And another, from a more recent project:

I had been testing this software for several months when the team was invited to view actual users working with the predecessor to our project. Going through the various groups of users, we eventually came to some who used the product differently than the others. They had on gloves, and worked with substances that they didn't want contacting the keyboard, so their entire interaction with the application was through hotkeys.

Pause a moment, if you would, and look back at how I have admitted to using applications. Can you guess whether I used hotkeys or mouse clicks in my test?

After that day, my tests had a mixture, and I had another viewpoint to look at while writing tests.

How do we fix it?

You can't rid yourself of all of your blind spots and predilections, but you can do a few things to help mitigate your own myopia:

• Other testers can point out flaws in your tests, where you might be missing a way of doing something in the application (do you always click the Save button instead of using Ctrl+S?)
• Groups of users can be polled for feedback (and this is a core way of doing usability testing)
• Be aware of what you are bad at. When you find out about some way of using an application that you don't do naturally, write it down, and try to include it into your way of thinking.

Now, if you'll excuse me, I need to go click some buttons and press some keys.

Windows 8 Wars

I was recently (well, two months ago) tasked (because I was the one who asked about it) with experimenting with TestComplete on Windows 8.

For some background:

TestComplete is a tool for doing system testing here at work. I haven't used it before coming here, and have only used it on Windows 7-based projects since working here. (For more information about TestComplete, feel free to visit the parent company, SmartBear. Don't worry, this blog post will still be here when you get back.)

Attempt 1: The VM Menace

My first attempt was to setup a Windows 8 VM on my work machine, install TestComplete on that, install some software that I had tests for already, execute the tests, and go home happy.

If you look above, I said 'two months' was when I got this task. You can imagine that this did not go as planned.

VMs are great testing tools, but in this case, Windows 8 was not supported very well by my VM technology of choice (VirtualBox). I had issues with getting the network connection working. This caused some fun issues as I tried to debug it. Ultimately, this attempt taught me that VMs are not always the best places for trying out new technology.

One other note is that during this time, SmartBear was running its own testing program for TestComplete for Windows 8. Sadly, I dropped the ball on moving fast enough into the next section to help them back with feedback here.

Attempt 2: The DB Wars

My next attempt was helped out tremendously by the man who volunteered me for this task Raman (I don't have a good link for him...you're on your own for this one). He was able to procure me a laptop with Windows 8 already installed and ready to go. This proved significantly easier than the VM.

Getting TestComplete itself installed was simple, as was getting the necessary files for the application I was going to use for testing.

Unfortunately, once more, I had issues with VMs and Windows 8. This time, it was setting up a server VM on the Windows 8 Machine. After several coworkers offering assistance, I was still unable to connect the VM to the main machine.

After several attempts, Winter break came, leading to...

Attempt 3: Revenge of the Tester

During the break, I rested and plotted my revenge against the Jedi Windows 8. It was during this time that I realized I had another application I could test, one that didn't require setting up another VM or connecting to a server.

After doing some setup in TestComplete, I fiddled with some simple tests, and learned the following:

They worked just fine.

A lot of build up for not much of a shocking discovery is sometimes a good thing.

During this time, I also did some research and found that SmartBear is informing their customers that they do not yet handle Metro applications.

So, to make a long story even longer, some lessons I've taken away from this:

• Time box your exploration of new technologies. While I already knew this, it gets much more difficult when it is more of a side-project.
• When exploring something, know that you may have to drop it for a while if life picks up around you. Keep good notes.
• Always get help when you need it. I have worked at SEP for a while, but it was nice getting a lot of help on this side-project from coworkers, from advice, to laptops, to hosting VMs.
• TestComplete (at least the latest version) works well on Windows 8. I'm sure there will be some gotchas in the future, but I don't see any fatal problems. I look forward to them working out the Metro issue, and being able to test those applications as well.
• I miss the Start Button.
• Every time I use a new laptop, I have to find out how to disable the touchpad again.
• Writing a blog on a Windows 8 machine is basically the same as writing one on a Windows 7 machine.