What are unit tests, integration tests, smoke tests, and regression tests?
What are unit tests, integration tests, smoke tests, and regression tests? What are the differences between them and which tools can I use for each of them?
Unit test: Specify and test one point of the contract of single method of a class. This should have a very narrow and well defined scope. Complex dependencies and interactions to the outside world are stubbed or mocked.
Integration test: Test the correct inter-operation of multiple subsystems. There is whole spectrum there, from testing integration between two classes, to testing integration with the production environment.
Smoke test (aka sanity check): A simple integration test where we just check that when the system under test is invoked it returns normally and does not blow up.
Regression test: A test that was written when a bug was fixed. It ensures that this specific bug will not occur again. The full name is "non-regression test". It can also be a test made prior to changing an application to make sure the application provides the same outcome.
To this, I will add:
Acceptance test: Test that a feature or use case is correctly implemented. It is similar to an integration test, but with a focus on the use case to provide rather than on the components involved.
System test: Tests a system as a black box. Dependencies on other systems are often mocked or stubbed during the test (otherwise it would be more of an integration test).
Pre-flight check: Tests that are repeated in a production-like environment, to alleviate the 'builds on my machine' syndrome. Often this is realized by doing an acceptance or smoke test in a production like environment.
Read more... Read less...
- Unit test: an automatic test to test the internal workings of a class. It should be a stand-alone test which is not related to other resources.
- Integration test: an automatic test that is done on an environment, so similar to unit tests but with external resources (db, disk access)
- Regression test: after implementing new features or bug fixes, you re-test scenarios which worked in the past. Here you cover the possibility in which your new features break existing features.
- Smoke testing: first tests on which testers can conclude if they will continue testing.
Everyone will have slightly different definitions, and there are often grey areas. However:
- Unit test: does this one little bit (as isolated as possible) work?
- Integration test: do these two (or more) components work together?
- Smoke test: does this whole system (as close to being a production system as possible) hang together reasonably well? (i.e. are we reasonably confident it won't create a black hole?)
- Regression test: have we inadvertently re-introduced any bugs we'd previously fixed?
A new test category I've just become aware of is the canary test. A canary test is an automated, non-destructive test that is run on a regular basis in a live environment, such that if it ever fails, something really bad has happened.
Examples might be:
- Has data that should only ever be available in development/testy appeared live?
- Has a background process failed to run?
- Can a user logon?
Unit test: Verifying that particular component (i.e., class) created or modified functions as designed. This test can be manual or automated, but it does not move beyond the boundary of the component.
Integration test: Verifying that the interaction of particular components function as designed. Integration tests can be performed at the unit level or the system level. These tests can be manual or automated.
Regression test: Verifying that new defects are not introduced into existing code. These tests can be manual or automated.
Depending upon your SDLC (waterfall, RUP, agile, etc.) particular tests may be performed in 'phases' or may all be performed, more or less, at the same time. For example, unit testing may be limited to developers who then turn the code over to testers for integration and regression testing. However, another approach might have developers doing unit testing and some level of integration and regression testing (using a TDD approach along with continuous integration and automated unit and regression tests).
The tool set will depend largely on the codebase, but there are many open source tools for unit testing (JUnit). HP's (Mercury) QTP or Borland's Silk Test are both tools for automated integration and regression testing.