Monday, July 16, 2007

But it Worked in QA

As a developer how many time have you heard "But it works in QA.", after a bug was found in production.

When a bug can't be reproduced in QA usually there a few reasons why.

The first reason could be that the version of application running in QA is different than in production. This is an easy test and if it is the case then then then next time you do a production promotion the bug should go away.

Another possibility for bugs in production and not QA are configuration settings of the application. Things like connection strings, web service urls and any other custom settings might not be configured properly for the production environment. Again this is an easy thing to check and correct.

These two solutions will probably fix most of the problems when a bug is not reproducible in a QA environment. But what do you do if it doesn't?

This was the case I just recently had. The client stated their users were reporting errors and they were having trouble reproducing them in their QA environment. So I started to look into the problem and saw the actual error on a user machine. The error the users were running into was a low memory error. The error was surprising since our development machines and QA machines were not having the same problem. Turns out the production devices were running on only 512mb of ram and the QA machines has 1gb. See the production devices need to be very strong and unbreakable, so to save money they purchased the base unit without the memory upgrade. Since in testing the devices aren't in the field they were using a different device, however thought the specs to be the same.

Errors like this is why it is very important to have QA and Production environments the same. Hardware, software running, and permissions should all be the same so that applications will react the same in both environments. There may be times that you can't set up both the same due to cost, time or whatever reason, but these differences should be documented and when testing the testers should try to determine the effects the differences will have.

No comments: