Monday, May 28, 2007
Database Question
What is As the difference among “dropping a table”, “truncating a table” and “deleting all records” from a table.
Answer :
Drop Table - will remove the existence of the table from the database along with its data and structure and all the constraints. The table will be no longer available.
A Truncate Table - will remove all the rows (only the rows) from a table. it will not delete the table. Its a DDL statement that means the deleted rows cannot be reverted back by ROLLBACK statement;
Delete Table - is a DML statement which will delete rows from a table according to the matching criteria mentions in the ‘where’ clause. And these rows can be reverted back by ‘ROLLBACK’ statement if ‘COMMIT’ is not fired.
What are the Large object types supported by Oracle?
Answer :
These are the large object type supported bye oracle
• CLOB and LONG for large fixed-width character data
• NCLOB for large fixed-width national character set data
• BLOB and LONG RAW for storing unstructured data
• BFILE for storing unstructured data in operating system files
What are the different types of joins?
Answer :
Different Types of Joins-
Inner Join
Outer Join - Right Outer Join & Left Outer Join
Cross Join
What is snapshot log?
Answer :
It is a table that maintains a record of modifications to the master table in a snapshot. It is stored in the same database as master table and is only available for simple snapshots. It should be created before creating snapshots.
Explain the difference between MyISAM Static and MyISAM Dynamic.
Answer :
In MyISAM static all the fields have fixed width. The Dynamic MyISAM table would include fields such as TEXT, BLOB, etc. to accommodate the data types with various lengths. MyISAM Static would be easier to restore in case of corruption, since even though you might lose some data, you know exactly where to look for the beginning of the next record.
How can you Enforce Referential Integrity in snapshots?
Answer :
Time the references to occur when master tables are not in use. Perform the reference the manually immediately locking the master tables. We can join tables in snapshots by creating a complex snapshots that will based on the master tables.
What is Two-Phase Commit?
Answer :
Two-phase commit is mechanism that guarantees a distributed transaction either commits on all involved nodes or rolls back on all involved nodes to maintain data consistency across the global distributed database. It has two phase, a Prepare Phase and a Commit Phase.
What is a cluster Key?
Answer :
The related columns of the tables are called the cluster key. The cluster key is indexed using a cluster index and its value is stored only once for multiple tables in the cluster.
What is Read-Only Transaction?
Answer :
A Read-Only transaction ensures that the results of each query executed in the transaction are consistent with respect to the same point in time.
What is Rule-based approach to optimization?
Answer :
Choosing an executing plan based on the access paths available and the ranks of these access paths.
What are constraints? Explain different types of constraints?
Answer
# 1 constraint is a rule which can not be voilated by end users.
Different types of constraints are available.They are:-
1)default constraint:-which is used to define a default
value.
2)primary key:-the key which does not allows duplication
and null values.
3)foreign key:-the key used to refer primary key defined
field in another table and it allows duplication.
4)null:-which allows NULL values.
5)not null:-which does not allows NULL values.
6)unique key:-which does not allows duplication but allows
NULL values.
----------------------------
Testing
Question Find the Cyclometric complexity on the following pseudocode
Do while records remain
read record;
if record field#1=0
then process record;store in buffer;
incremental counter;store in file.
reset counter;end if end do
please let me know the procedure too.
Q.Find the Cyclometric complexity on the following p seudocode Do while records remain read record; if record field#1=0 then process record;store in buffer; incremental counter;store in file. reset counter;end if end do please let me know the procedure too.
Answer
# 1 Listen i will tell u a basic funda:
Cylometric complexity is THE NO. OF LOOPS + 1
so in the question u have to check how many loops r present.
if its 1 then the ans is 2
if its 0 then the nas is 1
-----------
Answer
# 2 The answer for this is 3
3 formula's to calculate Cyclomatic complexity. You can
choose either of them...
C.C = No of bounded regions +1
C.C = No of edges - No of nodes +1
C.C = No of predicate Nodes +1
Now let's consider the first formula to calculate as the
rest 2 diagrams require flow graphs to be drawn.
Do while loop forms 1 bounded region
and IF condition forms 1 bounded region.
So as per this formula, C.C = No of bounded regions +1
2 + 1 = 3
-------------
What is Retesting?
Answer
# 1 Executing the same test cases on the same build but with
different inputs..
Answer
# 2 re execution of the test cases on the same build with
diffrent set of inputs
Answer
# 3 re executing the cases to get the confirmation that defects
are reproducable
What is defect leaking and defect release
Answer
# 1 While verifying the defect if it has been found tobe not
fixed, then it is called defect leaking.
If it has been fixed, then the defect is said to be release
Answer
# 2 when a version is released say ex:version2.0 and is any
modifications have been made to this version and named to
version 2.1 and the u find any (small) defect in the new
version which was left in 2.0 the tht is called defect leakage
A Succesful testcase Means..?
Answer
# 1 it should cover the functionality,it doed not contain if
staement,high probabilty of finding defect
Answer
# 2 A good testcase should satisfy the following two basic
requirements:
1.It should cover all the requirements as mentioned in the
srs
2.For each requirement different scenarios that are
possible should also be mentioned.
Answer
# 3 should cover functionality
should be easy to understand
should be maintanable
Answer
# 7 a Successful test case uncovers as an yet undiscovered
defect
Answer
# 8 to cover all functionality and intended to find bugs
Question Suppose " You are given a build with 5 modules to test as
per the Customer request after finalization of SRS, But the
SRS contains only documentation about 4 modules. On what
basis you will continue the testing of 5th module." ?
Answer
# 1 is developer problem
Answer
# 2 If it is not in SRS, it is bug. Any extra sec. is a bug
also.
Answer
# 3 I think we do Adhoc Testing. Upto 4 modules we first check
the functionality. After that we do the Adhoc on the 5th
module.
Generally we do Adhoc Testing, when there is no clear
specifications or when the time is less.
Answer
# 4 SRS is prepared based on requirements. Even SRS is not
there and have requirements, we analyze the requirements
first, Take the inputs from domain experts,end users.
Consolidate and prepare the SRS on our own and will send
the same for customer approval. based on feed back we can
go with testing.
Answer
# 5 Along with the Build release to the Testing team
Devolopment people releases Build release document, by
studing the release document we are going to conduct
testing on 5 th module.
Answer
# 6 if srs is available prepare test cases and to conduct
formal testing
if SRS is not available in this time we conduct adhoc testing.
Question Testing of Non-Functional requirements does not involove
a) Performance tests
b) Reliability tests
c) usability tests
d) Configuration tests
Answer
# 1 Configuration test
Answer
# 2 reliability tests & configuration tests
Answer
# 3 non functional reuirements includes all ie. performance,
reliability , usability & configiration tests.
Answer
# 4 all are non functional req. i guess...well done poonam
Answer
# 5 Reliability , Usability and Configuration comes under
Functional requirements, and Performance comes under non
functional requirements..
Answer
# 6 In Non-functionality requirements .We consider 1)performance
2)security that's it
What is metrics? explian the type of metrices.
Answer
# 3 Metrics are generally classified into 2 types
1. Process metrics
2. Product metrics
b a metric used to measure the haracteristic of the
methods, techniques and tools employed in eveloping,
implementing and maintaining the software system.
Product metric: a metric used to measure the characteristic
of the documentation and code. The metrics for the test
process would include status of test activities against the
plan, test coverage achieved so far, amongst others. An
important metric is the number ofdefects found in internal
testing compared to the defect found in customer tests,
which indicate the effectiveness of the test process itself.
Answer
# 4 Metrics meaures the strength of testing process
In otherwords it measures how much percentage of testing is
complted
types of metrics
Functional covreage
Softwate release metric
Error discovery rate
stability trend chart
what are the Levels of testing?
Answer
# 1 1. Unit Testing
2. Integration Testing
3. System Testing
4. Acceptance Testing
5. Regression Testing
Answer
# 2 there r 4 levels of testing
level 0 :: Sanity Testing test whether build is testable
or not
level 1 :: Unit testing, Integraion Testing, System Testing
level 2 :: Regression Testing
level 3 :: Final Regression testing or Postmartom Testing
Answer
# 3 There are only 4 levels of testing
1. Unit Testing
2. Integration Testing
3. System Testing
4. User Acceptance Testing
Answer
# 4 1.unit testing
2.Integration testing
a)Top down integration testing
b)Bottom up integration testing
c)Big Bang integration testing
d)Isolation integration testing
e)Incremental integration testing
3.System testing
4.Acceptance testing.
Answer
# 5 Levels of testing
1.Unit testing
2.Module testing
3.Interation testing
a)Top down integration testing
b)Bottom up integration testing
c)Big Bang integration testing
d)Isolation integration testing
e)Incremental integration testing
4.System testing
5.User acceptance testing
What kinds of testing have you done?
Answer
# 1 I performed Installation testing, browser compatability
testing, regression testing, parallel testing.
Answer
# 2 Testers can perform Balck box testing,Grey box
testing,Smoke testing,Sanity testing,endurance
testing,automation testing,volume testing,Database
testing,Stress testing,End-end testing,alpha testing,adhoc
testing,concurrency testing ....List will not end!!!!!!!
Answer
# 3 Tester can perform Black box testing,GUI testing,DataBase
testing,Integration(Tester+Developer)System and UAT.As per
Client requirements.
Answer
# 4 Moreover testers perform sanity testing, black box,
regression, performance, volume, system testing and installtion.
Answer
# 5 I would like to add something here
Black Box & White box testing are two approaches of testing
Most commonly every tester perform
1.Functional testing
2.Regression testing
3.Re-testing
4.Sanity testing
5.Compatibility testing
6.UI testing
7.Exploratory tesing (this is depending upon the type of
project)
8.end to end testing
9.adhoc testing
10.Installation testing
Answer
# 7 Kinds of testing means:
1. Static testing & Dynamic testing.
2. Structural testing & Functional testing.
3. Automation testing & Manual testing.
Testers will do only:
Dynamic testing
Functional testing
Manual testing
what type of documents can u prepare during Testing?
Answer
# 1 Test case document, Bug report, Status reports, Tracebility
Matrix, Use cases, Metric preparation, Release report
Answer
# 2 1.Test Plan : A formal test plan is a document that
provides and records important information about a test
project, for example: Resources, Schedule & Timeline, Test
Milestones, Use cases and/or Test cases
2.Test Case: A set of test inputs, execution conditions and
expected results.
3.Test Data: The actual (set of) values used in the test or
that are necessary to execute the test.
4.Test Scripts: is used to test a particular functionality
(business rule). It may consist of 1 or more Test Cases.
5.Test Log: A chronological record of all relevant details
about the execution of a test
6.Bug Reports: Contains a summary of the bug, its priority
and other details regarding the bug.
Answer
# 3 BRD business requirment document.
srs system requirment specifications.
frs functional requirment specifications,
Test plan: it include what to test,when to test ,where to
test, whom to test,how to test.
test cases : functionality to be test
defwct reporting :
bug report :
Answer
# 4 Here I ma giving all the documents,which are using in our
project and the authorized person,who is preparing them.It
varies from company to company.
Project Plan:Prepared by PM.
QA-Project Plan:Prepared by QA Manager
Test Reports:Generally prepared by Team lead for every
release.
Software Metrics: Team Lead
Module based Test Plan: Testing engineer
Test Results: Test Engineer
Testcase document: Test Engineer
Defect Matrix: Prepared by Team Lead
Traceability Matrix: Team Lead or team member
Bu Reports: Prepared by test engineer.
Define Brain Storming and Cause Effect Graphing? With Eg?
Answer
# 1 Hi...........
BS:
A learning technique involving open group discussion
intended to expand the range of available ideas
OR
A meeting to generate creative ideas. At PEPSI Advertising,
daily, weekly and bi-monthly brainstorming sessions are
held by various work groups within the firm. Our monthly I-
Power brainstorming meeting is attended by the entire
agency staff.
OR
Brainstorming is a highly structured process to help
generate ideas. It is based on the principle that you
cannot generate and evaluate ideas at the same time. To use
brainstorming, you must first gain agreement from the group
to try brainstorming for a fixed interval (eg six minutes).
CEG :
A testing technique that aids in selecting, in a systematic
way, a high-yield set of test cases that logically relates
causes to effects to produce test cases. It has a
beneficial side effect in pointing out incompleteness and
ambiguities in specifications.
What is software development Life Cycle (SDLC)? where exactly the Testing activity begin in SDLC?
Answer
# 1 SDLC is a framework for understanding and developing
information and software succesfully.
Answer
# 2 SDLC is a framework for understanding and developing
information and software succesfully.
Testing activity begins right from the requirement analysis
phase.
For Web Applications what type of tests r u going to do .. tell me the Importent ones..
Answer
# 2 The following are the tests need to be done for Web
application.
Static Testing
Dynamic Testing
Navigation Testing
Compatabilty Testing
Functionality Testing
Link Testing
Answer
# 3 for an web application we need 2 check
URL of appl
navigation(Normal links,broken links_
scalability
reliabilty(recovery testing)
usability
compatbility
forms,cookies
Answer
# 4 for web application mainly tests as follows
userinterface means everyone understand the application is
mportant
funcationality
navigation
performence means speed of acces is imp
links
Answer
# 5 Functional Testing
Smoke Testing(for UI and API's-- Through Automation tool)
Regression Testing
Batch Process Testing(for simulators)
---------
Web-based applications present new challenges, these challenges include:
- Short release cycles;
- Constantly Changing Technology;
- Possible huge number of users during initial website launch;
- Inability to control the user's running environment;
- 24-hour availability of the web site.
The quality of a website must be evident from the Onset. Any difficulty whether in response time, accuracy of information, or ease of use-will compel the user to click to a competitor's site. Such problems translate into lost of users, lost sales, and poor company image.
To overcome these types of problems, use the following techniques:
1. Functionality Testing
Functionality testing involves making Sure the features that most affect user interactions work properly. These include :
· forms
· searches
· pop-up windows
· shopping carts
· online payments
2. Usability Testing
Many users have low tolerance for anything that is difficult to use or that does not work. A user's first impression of the site is important, and many websites have become cluttered with an increasing number of features. For general-use websites frustrated users can easily click over a competitor's site.
Usability testing involves following main steps
· identify the website's purpose;
· identify the indented users;
· define tests and conduct the usability testing
· analyze the acquired information
3. Navigation Testing
Good Navigation is an essential part of a website, especially those that are complex and provide a lot of information. Assessing navigation is a major part of usability Testing.
4. Forms Testing
Websites that use forms need tests to ensure that each field works properly and that the forms posts all data as intended by the designer.
5. Page Content Testing
Each web page must be tested for correct content from the user perspective for correct content from the user perspective. These tests fall into two categories: ensuring that each component functions correctly and ensuring that the content of each is correct.
6. Configuration and Compatibility Testing
A key challenge for web applications is ensuring that the user sees a web page as the designer intended. The user can select different browser software and browser options, use different network software and on-line service, and run other concurrent applications. We execute the application under every browser/platform combination to ensure the web sites work properly under various environments.
7. Reliability and Availability Testing
A key requirement o a website is that it Be available whenever the user requests it, after 24-hours a day, every day. The number of users accessing web site simultaneously may also affect the site's availability.
8. Performance Testing
Performance Testing, which evaluates System performance under normal and heavy usage, is crucial to success of any web application. A system that takes for long to respond may frustrate the user who can then quickly move to a competitor's site. Given enough time, every page request will eventually be delivered. Performance testing seeks to ensure that the website server responds to browser requests within defined parameters.
9. Load Testing
The purpose of Load testing is to model real world experiences, typically by generating many simultaneous users accessing the website. We use automation tools to increases the ability to conduct a valid load test, because it emulates thousand of users by sending simultaneous requests to the application or the server.
10. Stress Testing
Stress Testing consists of subjecting the system to varying and maximum loads to evaluate the resulting performance. We use automated test tools to simulate loads on website and execute the tests continuously for several hours or days.
11. Security Testing
Security is a primary concern when communicating and conducting business- especially sensitive and business- critical transactions - over the internet. The user wants assurance that personal and financial information is secure. Finding the vulnerabilities in an application that would grant an unauthorized user access to the system is important.
Following is the strategy we used in one of my projects:
1. From the requirements, list the priorities of testing.
2. Use a Link Checker to check all the broken text and image links. This will alert you on missing or broken links but not links that have been pointed to the unintended locations/files. But ignore such links for now this is just one of the cursory checks. Report the results.
3. Collect as many scenarios of real users browsing your site. If you are testing an already existing site you can get them from the site Stats. These will probably cover all your existing user base. If your site needs to target additional user base that is a whole other task.
4. Test using the scenarios.
5. From your testing priorities, list the test cases not touched in your previous test scenarios. If you want a complete list or have time to create one, just go ahead or else you can also list them and note the result from your tests earlier in here. We just listed a short description of each test case, it's input and output. It'll be helpful to record your results if you've a column in there to record your results when testing.
6. Start testing using the test cases and record the results.
7. Prioritize and Report the defects. If you don't have time for testing, there might not be sufficient time to fix them all and re-test the defects either. Prioritizing them will help fix at least the major bugs and/or issues. The rest will be fixed or ignored depending on their priority and your project deadlines.
What are the test methodologies??
Answer
# 1 Test Methodolgies defines how to approach the
testing .project type,system domain,risks,at which phase
what type of testing occurs,
all these comes under Methodology
What are the contents in "Requirements Traceability Matrix" not in " Test Responsibility Matrix"?
Answer
# 1 Contents of Requirements Traceability Matrix / RTM :
Release Name
Test Case Name
High level design
Low Level Design
Test Case Execution
Automation Execution
Difference between regression testing and re testing?
Answer
# 1 Retesting- Re test the defects after fixed.
Regression testing-- Nothing but retest, After fix the
defects we are checking the functinality,if the fixed
defect is effected to another functionality.
Answer
# 2 Retesting : Testing the application on the same built
soppose tester found the bugs and developer fixed it.
tested the fixed issues on the same built is called
retesting.
Regression: Testing the application on the different built
Suppose: if the tester found the bugs, and Developer fixed
it and they relised the new version of the build. so
testing on the new version on the built is called Regerssion
Testing
Answer
# 3 re testing:testing the build with multiple test data
regression testing:if the tester find any defect he /she
inform this defect to development team.developement team
fixes that bug, aagin send to the testing team for testing
again whether bug is fixed or not,or the newly developed
build giving any side affects to other modules or not.
Answer
# 4 Re Testing: Once the Bug is fixed Tester has to test the
same bug with multiple number of times with different data
and different scenario. Also once tester find the bug he
has to test multiple number of times bcoz whether the bug
is reproduceable or not.
Regression Testing: Once the Bug is fixed tester has to
test whether bug is fixed or not also test whether the
impact of the bug is shows in other application
What is the diffrence between BUILD & RELEASE ?
Answer
# 1 Difference number one:Builds refer to software that is
still in testing, release refers to software that is
usually no longer in testing.
Difference number two: Builds occur more frequently;
releases occur less frequently.
Answer
# 2 Generally build is a inegrated module,which contains the
fixes for your previous builds bugsGenerally we will
recievie the builds in daily basis,weekly basis.
Once we completed the testing on our software,before we
rolling out that software to our client,we will select a
final build,which is almost defect free and we will take
that build as release build.That release build will be
introduced in the market for end-users.
What is risk analysis,what type of risk analysis u did in u r project
Answer
# 1 hi.................(Must say u have a very good question )
Risk Analysis :
A systematic use of available information to determine how
often specified events and unspecified events may occur and
the magnitude of their likely consequences
OR
procedure to identify threats & vulnerabilities, analyze
them to ascertain the exposures, and highlight how the
impact can be eliminated or reduced
Types :
1.QUANTITATIVE RISK ANALYSIS
2.QUALITATIVE RISK ANALYSIS
what is test case? what it consists? what are the types of test cases?
Answer
# 1 Test Case is a document that describes the input or action
or event and expected response to determine the feature of
the application is working correctly. It contains Test case
No: Action/Input and Expected and Actual Result and Status
(Pass/Fail).
Answer
# 2 Test case is a document specifying inputs, predicted
results, and a set of execution conditions for a test
item.We can name test cases as system test
cases,integration test cases,Unit test cases (that include
validations)
Answer
# 3 Sharavati,
The Test cases naming which you have mentioned here is said
the be levels of testing in testing life cycle of a
software. Hence you cannot name/catogorize the test cases.
Answer
# 4 Test case is a document.It contains inputs,outputs and
outcome .There are three types of test cases
they are
1)Business logic based testcases
2)Input domain based testcases
3)user interface based testcases
Answer
# 5 Test case is a document.It contains inputs,outputs and
outcome .There are three types of test cases
they are
1)Business logic based testcases
2)Input domain based testcases
3)user interface based testcases
Answer
# 6 For me test cases is a document which describes ,
What is we r going to test ,i.e (test area)
followed by test description ,then by the test steps , then
the expected result ,actual result , status(pass/fail_)
Answer
# 7 Test Case:
Definition: To verify the possible values for the
attribute.
It Cosists:
positive and negative values for the attribute
Types:
GUI test case
Functional test case
System level test case
Answer
# 9 Test case is descrption f what is to be tested, what the
data to be used ,what are the actions done to check the
actual results against expected result
What is the difference between QC and QA?
Answer :
Quality assurance is the process where the documents for the product to be tested is verified with actual requirements of the customers. It includes inspection, auditing , code review , meeting etc.
Quality control is the process where the product is actually executed and the expected behavior is verified by comparing with the actual behavior of the software under test. All the testing types like black box testing, white box testing comes under quality control.
Quality assurance is done before quality control.
What is a scenario?
Answer :
A scenario defines the events that occur during each testing session. For example, a scenario defines and controls the number of users to emulate, the actions to be performed, and the machines on which the virtual users run their emulations.
What is Difference betweem Manual and Automation Testing?
Answer :
This answer is quite simple, Manual is when user needs to do many things based on the test case specified, say like click some tab and check if the tab is working fine or click on a particular URL and check if the web site specified opens.
The above stuff can also be done automatically, i.e. through some automated tools like Winrunner , Silk Test etc the things can be automated , so user just has to trigger the tool and the tool will take care of executing , only thing needed is we need to know how to make the testcases automated through that tool which is not very difficult.
What is L10 Testing?
Answer :
L10 Testing is Localization Testing, it verifies whether your products are ready for local markets or not.
What is I18N Testing?
Answer :
I18N Testing is “Internationalization testing”
Determine whether your developed product’s support for international character encoding methods is sufficient and whether your product development methodologies take into account international coding standards.
What is SEI? CMM? CMMI? ISO? IEEE? ANSI?
Answer :
SEI = ‘Software Engineering Institute’ at Carnegie-Mellon University; initiated by the U.S. Defense Department to help improve software development processes.
CMM = ‘Capability Maturity Model’, now called the CMMI (’Capability Maturity Model Integration’), developed by the SEI. It’s a model of 5 levels of process ‘maturity’ that determine effectiveness in delivering quality software. It is geared to large organizations such as large U.S. Defense Department contractors. However, many of the QA processes involved are appropriate to any organization, and if reasonably applied can be helpful. Organizations can receive CMMI ratings by undergoing assessments by qualified auditors.
Level 1 - characterized by chaos, periodic panics, and heroic efforts required by individuals to successfully complete projects. Few if any processes in place; successes may not be repeatable.
Level 2 - software project tracking, requirements management, realistic planning, and configuration management processes are in place; successful practices can be repeated.
Level 3 - standard software development and maintenance processes are integrated throughout an organization; a Software Engineering Process Group is is in place to oversee software processes, and training programs are used to ensure understanding and compliance.
Level 4 - metrics are used to track productivity, processes, and products. Project performance is predictable, and quality is consistently high.
Level 5 - the focus is on continouous process improvement. The impact of new processes and technologies can be predicted and effectively implemented when required.
ISO = ‘International Organisation for Standardization’ - The ISO 9001:2000 standard (which replaces the previous standard of 1994) concerns quality systems that are assessed by outside auditors, and it applies to many kinds of production and manufacturing organizations, not just software. It covers documentation, design, development, production, testing, installation, servicing, and other processes. The full set of standards consists of: (a)Q9001-2000 - Quality Management Systems: Requirements; (b)Q9000-2000 - Quality Management Systems: Fundamentals and Vocabulary; (c)Q9004-2000 - Quality Management Systems: Guidelines for Performance Improvements.
To be ISO 9001 certified, a third-party auditor assesses an organization, and certification is typically good for about 3 years, after which a complete reassessment is required. Note that ISO certification does not necessarily indicate quality products - it indicates only that documented processes are followed.
IEEE = ‘Institute of Electrical and Electronics Engineers’ - among other things, creates standards such as ‘IEEE Standard for Software Test Documentation’ (IEEE/ANSI Standard 829), ‘IEEE Standard of Software Unit Testing (IEEE/ANSI Standard 1008), ‘IEEE Standard for Software Quality Assurance Plans’ (IEEE/ANSI Standard 730) etc
ANSI = ‘American National Standards Institute’, the primary industrial standards body in the U.S.; publishes some software-related standards in conjunction with the IEEE and ASQ (American Society for Quality).
Other software development/IT management process assessment methods besides CMMI and ISO 9000 include SPICE, Trillium, TickIT, Bootstrap, ITIL, MOF, and CobiT.
What is documentation change management?
Answer :
Documentation change management is part of configuration management (CM). CM covers the tools and processes used to control, coordinate and track code, requirements, documentation, problems, change requests, designs, tools, compilers, libraries, patches, changes made to them and who makes the changes
What is the role of documentation in QA?
Answer :
Documentation plays a critical role in QA. QA practices should be documented, so that they are repeatable. Specifications, designs, business rules, inspection reports, configurations, code changes, test plans, test cases, bug reports, user manuals should all be documented. Ideally, there should be a system for easily finding and obtaining of documents and determining what document will have a particular piece of information. Use documentation change management, if possible.
How do you perform integration testing?
Answer :
First, unit testing has to be completed. Upon completion of unit testing, integration testing begins. Integration testing is black box testing. The purpose of integration testing is to ensure distinct components of the application still work in accordance to customer requirements. Test cases are developed with the express purpose of exercising the interfaces between the components. This activity is carried out by the test team.
Integration testing is considered complete, when actual results and expected results are either in line or differences are explainable/acceptable based on client input.
What is clear box testing?
Answer :
Clear box testing is the same as white box testing. It is a testing approach that examines the application’s program structure, and derives test cases from the application’s program logic.
What is closed box testing?
Answer :
Closed box testing is same as black box testing. Black box testing a type of testing that considers only externally visible behavior. Black box testing considers neither the code itself, nor the “inner workings” of the software.
What is open box testing?
Answer :
Open box testing is same as white box testing. It is a testing approach that examines the application’s program structure, and derives test cases from the application’s program logic.
What is ‘Software Quality Assurance’?
Answer :
Software QA involves the entire software development PROCESS - monitoring and improving the process, making sure that any agreed-upon standards and procedures are followed, and ensuring that problems are found and dealt with. It is oriented to ‘prevention’.
What is Security Testing?
Answer :
Security Testing
Application vulnerabilities leave your system open to attacks, Downtime, Data theft, Data corruption and application Defacement. Security within an application or web service is crucial to avoid such vulnerabilities and new threats.
While automated tools can help to eliminate many generic security issues, the detection of application vulnerabilities requires independent evaluation of your specific application’s features and functions by experts. An external security vulnerability review by Third Eye Testing will give you the best possible confidence that your application is as secure as possible.
Q: How do I test a database?A: As far as I know there are four different things that could beconsidered database testing1) using the database as a repository for some user-generatedfront-end activity (client-server)2) using the database as a repository for some application-generatedactivity (transaction logging)3) database migration (bringing one repository from one schema toanother or merging two or more schemas)4) database development (you work for Oracle, MySQL, or a company thatmakes databases)Which one are you trying to do?How to Thoroughly Test a Data WarehouseAuthor(s): Rebecca Cooper/Suellen Arbucklehttp://www.stickyminds.com/r.asp?F=DART_5941Temporal Data Integrity ValidationAuthor(s): Doron Drusinskyhttp://www.stickyminds.com/r.asp?F=DART_5938Testing Merged DatabasesAuthor(s): Bernie Berger/Lawrence NuanezSTQE Issue: Nov/Dec 2002 (Vol. 4 Issue 6)http://www.stickyminds.com/r.asp?F=DART_6181This article requires a PowerPass. It is summarised in the abstract,which I'll paraphrase for you here:This addresses four different problems that have to be overcome in adatabase merger:1) duplicate records (The challenge of duplicate records);2) mismatched columns (Ways to check for mismatched column definitions);3) data corruption (Tips for finding corrupt data);and 4) front-end assumptions (How to verify front- and back-end congruence).
How can u do regression testing tell me procedure?
Answer
# 1 Regression test will be done when we got modified build there we execute test cases to determine whether defects closed or not? or checking 2 ensure that resolved bug has made any impact
on the remaining part of code
Answer
# 2 Regression Testing is as discussed in above answer is done
after the first build release, and in regression testing we
test if the bugs that exited in the Build are actually fixed
or not, we also apply testcases for new requirements (if
any) to check if the system is working fine.
Answer
# 3 Regarding the regression testing procedure, it is same as
testing, we just re-run all those test that were previously
run, but this time we make sure that the test cases match
the new chnages, or new requirements that were not there in
the system.
Answer
# 4 Regreesion testing is nothing but whenever we find out the
bug , it has to be fixed ,given to developers they will fix
the bug , this is known as regression testing
Answer
# 5 Regression Testing-"Testing to ensure that introduction of
new functionality has not affected existing functionality"
Answer
# 6 Whenever tester gets the next fixed build:
1)He tests for the Fixed Bugs.
2)Test the whole application to check whether these fixes,
have not introduced new bugs.
3)If there are some update in the application, then test
cases need to be updated , and executed the same.
All of the above steps used, while doing regression testing.
Answer
# 7 Reexecution of testcases on modified form of build is
called regression testing.
Answer
# 8 regression testing:when ever the bug is fixed from
developement team we can verify whether the bug is fixed or
not and their related functionality is also working fine or not.
Friday, May 25, 2007
What’s a Test Case?
What’s a Test Case?
Let’s start with the basics. What’s a test case?
IEEE Standard 610 (1990) defines test case as follows:
“(1) A set of test inputs, execution conditions, and expected results developed for a
particular objective, such as to exercise a particular program path or to verify compliance
with a specific requirement.
“(2) (IEEE Std 829-1983) Documentation specifying inputs, predicted results, and a set
of execution conditions for a test item.”
According to Ron Patton (2001, p. 65),
“Test cases are the specific inputs that you’ll try and the procedures that you’ll follow
when you test the software.”
Boris Beizer (1995, p. 3) defines a test as
“A sequence of one or more subtests executed as a sequence because the outcome and/or
final state of one subtest is the input and/or initial state of the next. The word ‘test’ is
used to include subtests, tests proper, and test suites.
defines test case:
“A test case specifies the pretest state of the IUT and its environment, the test inputs or
conditions, and the expected result. The expected result specifies what the IUT should
produce from the test inputs. This specification includes messages generated by the IUT,
exceptions, returned values, and resultant state of the IUT and its environment. Test cases
may also specify initial and resulting conditions for other objects that constitute the IUT
and its environment.”
In practice, many things are referred to as test cases even though they are far from being fully
documented.
Brian Marick uses a related term to describe the lightly documented test case, the test idea:
“A test idea is a brief statement of something that should be tested. For example, if you're
testing a square root function, one idea for a test would be ‘test a number less than zero’.
The idea is to check if the code handles an error case.”
In my view, a test case is a question that you ask of the program. The point of running the test is
to gain information, for example whether the program will pass or fail the test.
It may or may not be specified in great procedural detail, as long as it is clear what is the idea of
the test and how to apply that idea to some specific aspect (feature, for example) of the product.
If the documentation is an essential aspect of a test case, in your vocabulary, please substitute the term “test idea” for “test case” in everything that follows.
An important implication of defining a test case as a question is that a test case must be
reasonably capable of revealing information.
Under this definition, the scope of test cases changes as the program gets more
stable. Early in testing, when anything in the program can be broken, trying the
largest “legal” value in a numeric input field is a sensible test. But weeks later, after
the program has passed this test several times over several builds, a standalone test
of this one field is no longer a test case because there is only a miniscule probability
of failure. A more appropriate test case at this point might combine boundaries of ten
different variables at the same time or place the boundary in the context of a longsequence
test or a scenario.
Also, under this definition, the metrics that report the number of test cases are
meaningless. What do you do with a set of 20 single-variable tests that were
interesting a few weeks ago but now should be retired or merged into a combination?
Suppose you create a combination test that includes the 20 tests. Should the metric
report this one test, 20 tests, or 21? What about the tests that you run only once?
What about the tests that you design and implement but never run because the
program design changes in ways that make these tests uninteresting?
Another implication of the definition is that a test is not necessarily designed to expose a defect.
The goal is information. Very often, the information sought involves defects, but not always. (I
owe this insight to Marick, 1997.) To assess the value of a test, we should ask how well it
provides the information we’re looking for.
Information Objectives
So what are we trying to learn or achieve when we run tests? Here are some examples:
Find defects. This is the classic objective of testing. A test is run in order to trigger
failures that expose defects. Generally, we look for defects in all interesting parts of the
product.
Maximize bug count. The distinction between this and “find defects” is that total number
of bugs is more important than coverage. We might focus narrowly, on only a few highrisk
features, if this is the way to find the most bugs in the time available.
Block premature product releases. This tester stops premature shipment by finding bugs
so serious that no one would ship the product until they are fixed. For every releasedecision
meeting, the tester’s goal is to have new showstopper bugs.
Help managers make ship / no-ship decisions. Managers are typically concerned with
risk in the field. They want to know about coverage (maybe not the simplistic code
coverage statistics, but some indicators of how much of the product has been addressed
and how much is left), and how important the known problems are. Problems that appear
significant on paper but will not lead to customer dissatisfaction are probably not relevant
to the ship decision.
Minimize technical support costs. Working in conjunction with a technical support or
help desk group, the test team identifies the issues that lead to calls for support. These are
often peripherally related to the product under test--for example, getting the product to
work with a specific printer or to import data successfully from a third party database
might prevent more calls than a low-frequency, data-corrupting crash.
Assess conformance to specification. Any claim made in the specification is checked.
Program characteristics not addressed in the specification are not (as part of this
objective) checked.
Conform to regulations. If a regulation specifies a certain type of coverage (such as, at
least one test for every claim made about the product), the test group creates the
appropriate tests. If the regulation specifies a style for the specifications or other
documentation, the test group probably checks the style. In general, the test group is
focusing on anything covered by regulation and (in the context of this objective) nothing
that is not covered by regulation.
Minimize safety-related lawsuit risk. Any error that could lead to an accident or injury is
of primary interest. Errors that lead to loss of time or data or corrupt data, but that don’t
carry a risk of injury or damage to physical things are out of scope.
Find safe scenarios for use of the product (find ways to get it to work, in spite of the
bugs). Sometimes, all that you’re looking for is one way to do a task that will consistently
work--one set of instructions that someone else can follow that will reliably deliver the
benefit they are supposed to lead to. In this case, the tester is not looking for bugs. He is
trying out, empirically refining and documenting, a way to do a task.
Assess quality. This is a tricky objective because quality is multi-dimensional. The nature
of quality depends on the nature of the product. For example, a computer game that is
rock solid but not entertaining is a lousy game. To assess quality -- to measure and report
back on the level of quality -- you probably need a clear definition of the most important
quality criteria for this product, and then you need a theory that relates test results to the
definition. For example, reliability is not just about the number of bugs in the product. It
is (or is often defined as being) about the number of reliability-related failures that can be
expected in a period of time or a period of use. (Reliability-related? In measuring
reliability, an organization might not care, for example, about misspellings in error
messages.) To make this prediction, you need a mathematically and empirically sound
model that links test results to reliability. Testing involves gathering the data needed by
the model. This might involve extensive work in areas of the product believed to be
stable as well as some work in weaker areas. Imagine a reliability model based on
counting bugs found (perhaps weighted by some type of severity) per N lines of code or
per K hours of testing. Finding the bugs is important. Eliminating duplicates is important.
Troubleshooting to make the bug report easier to understand and more likely to fix is (in
the context of assessment) out of scope.
Verify correctness of the product. It is impossible to do this by testing. You can prove
that the product is not correct or you can demonstrate that you didn’t find any errors in a
given period of time using a given testing strategy. However, you can’t test exhaustively,
and the product might fail under conditions that you did not test. The best you can do (if
you have a solid, credible model) is assessment--test-based estimation of the probability
of errors. (See the discussion of reliability, above).
Assure quality. Despite the common title, quality assurance, you can’t assure quality by
testing. You can’t assure quality by gathering metrics. You can’t assure quality by setting
standards. Quality assurance involves building a high quality product and for that, you
need skilled people throughout development who have time and motivation and an
appropriate balance of direction and creative freedom. This is out of scope for a test
organization. It is within scope for the project manager and associated executives. The
test organization can certainly help in this process by performing a wide range of
technical investigations, but those investigations are not quality assurance.
Given a testing objective, the good test series provides information directly relevant to that
objective.
Tests Intended to Expose Defects
Let’s narrow our focus to the test group that has two primary objectives:
Find bugs that the rest of the development group will consider relevant (worth reporting)
and Get these bugs fixed.
Even within these objectives, tests can be good in many different ways. For example, we might
say that one test is better than another if it is:
More powerful. I define power in the usual statistical sense as more likely to expose a
bug if it the bug is there. Note that Test 1 can be more powerful than Test 2 for one type
of bug and less powerful than Test 2 for a different type of bug.
More likely to yield significant (more motivating, more persuasive) results. A problem
is significant if a stakeholder with influence would protest if the problem is not fixed. (A
stakeholder is a person who is affected by the product. A stakeholder with influence is
someone whose preference or opinion might result in change to the product.)
More credible. A credible test is more likely to be taken as a realistic (or reasonable) set
of operations by the programmer or another stakeholder with influence. “Corner case” is
an example of a phrase used by programmers to say that a test or bug is non-credible:
“No one would do that.” A test case is credible if some (or all) stakeholders agree that it
is realistic.
Representative of events more likely to be encountered by the customer. A population of
tests can be designed to be highly credible. Set up your population to reflect actual usage
probabilities. The more frequent clusters of activities are more likely to be covered or
covered more thoroughly. (I say cluster of activities to suggest that many features are
used together and so we might track which combinations of features are used and in what
order, and reflect this more specific information in our analysis.) For more details, read
Musa's (1998) work on software reliability engineering.
Easier to evaluate. The question is, did the program pass or fail the test? Ease of
Evaluation. The tester should be able to determine, quickly and easily, whether the
program passed or failed the test. It is not enough that it is possible to tell whether the
program passed or failed. The harder evaluation is, or the longer it takes, the more likely
it is that failures will slip through unnoticed. Faced with time-consuming evaluation, the
tester will take shortcuts and find ways to less expensively guess whether the program is
OK or not. These shortcuts will typically be imperfectly accurate (that is, they may miss
obvious bugs or they may flag correct code as erroneous.)
More useful for troubleshooting. For example, high volume automated tests will often
crash the system under test without providing much information about the relevant test
conditions needed to reproduce the problem. They are not useful for troubleshooting.
Tests that are harder to repeat are less useful for troubleshooting. Tests that are harder to
perform are less likely to be performed correctly the next time, when you are
troubleshooting a failure that was exposed by this test.
More informative. A test provides value to the extent that we learn from it. In most cases,
you learn more from the test that the program passes than the one the program fails, but
the informative test will teach you something (reduce your uncertainty) whether the
program passes it or fails.
o For example, if we have already run a test in several builds, and the program
reliably passed it each time, we will expect the program to pass this test again.
Another "pass" result from the reused test doesn't contribute anything to our
mental model of the program.
o The notion of equivalence classes provides another example of information value.
Behind any test is a set of tests that are sufficiently similar to it that we think of
the other tests as essentially redundant with this one. In traditional jargon, this is
the "equivalence class" or the "equivalence partition." If the tests are sufficiently
similar, there is little added information to be obtained by running the second one
after running the first.
o This criterion is closely related to Karl Popper’s theory of value of experiments
(See Popper 1992). Good experiments involve risky predictions. The theory
predicts something that many people would expect not to be true. Either your
favorite theory is false or lots of people are surprised. Popper’s analysis of what
makes for good experiments (good tests) is a core belief in a mainstream approach
to the philosophy of science.
Perhaps the essential consideration here is that the expected value of what you
will learn from this test has to be balanced against the opportunity cost of
designing and running the test. The time you spend on this test is time you don't
have available for some other test or other activity.
Appropriately complex. A complex test involves many features, or variables, or other
attributes of the software under test. Complexity is less desirable when the program has
changed in many ways, or when you’re testing many new features at once. If the program
has many bugs, a complex test might fail so quickly that you don’t get to run much of it.
Test groups that rely primarily on complex tests complain of blocking bugs. A blocking
bug causes many tests to fail, preventing the test group from learning the other things
about the program that these tests are supposed to expose. Therefore, early in testing,
simple tests are desirable. As the program gets more stable, or (as in eXtreme
Programming or any evolutionary development lifecycle) as more stable features are
incorporated into the program, greater complexity becomes more desirable.
More likely to help the tester or the programmer develop insight into some aspect of the
product, the customer, or the environment. Sometimes, we test to understand the
product, to learn how it works or where its risks might be. Later, we might design tests to
expose faults, but especially early in testing we are interested in learning what it is and
how to test it. Many tests like this are never reused. However, in a test-first design
environment, code changes are often made experimentally, with the expectation that the
(typically, unit) test suite will alert the programmer to side effects. In such an
environment, a test might be designed to flag a performance change, a difference in
rounding error, or some other change that is not a defect. An unexpected change in
program behavior might alert the programmer that her model of the code or of the impact
of her code change is incomplete or wrong, leading her to additional testing and
troubleshooting. (Thanks to Ward Cunningham and Brian Marick for suggesting this
example.)
Test Styles/Types and Test Qualities
Within the field of black box testing, Kaner & Bach (see our course notes, Bach 2003b and
Kaner, 2002, posted at www.testingeducation.org, and see Kaner, Bach & Pettichord, 2002) have described eleven dominant styles of black box testing:
Function testing
Domain testing
Specification-based testing
Risk-based testing
Stress testing
Regression testing
User testing
Scenario testing
State-model based testing
High volume automated testing
Exploratory testing
Bach and I call these "paradigms" of testing because we have seen time and again that one or two of them dominate the thinking of a testing group or a talented tester. An analysis we find
intriguing goes like this:
If I was a "scenario tester" (a person who defines testing primarily in terms of application
of scenario tests), how would I actually test the program? What makes one scenario test
better than another? Why types of problems would I tend to miss, what would be difficult
for me to find or interpret, and what would be particularly easy?
Here are thumbnail sketches of the styles, with some thoughts on how test cases are “good”
within them.
Function Testing
Test each function / feature / variable in isolation.
Most test groups start with fairly simple function testing but then switch to a different style, often
involving the interaction of several functions, once the program passes the mainstream function
tests.
Within this approach, a good test focuses on a single function and tests it with middle-of-theroad
values. We don’t expect the program to fail a test like this, but it will if the algorithm is
fundamentally wrong, the build is broken, or a change to some other part of the program has
fowled this code.
These tests are highly credible and easy to evaluate but not particularly powerful.
Some test groups spend most of their effort on function tests. For them, testing is complete when
every item has been thoroughly tested on its own. In my experience, the tougher function tests
look like domain tests and have their strengths.
Domain Testing
The essence of this type of testing is sampling. We reduce a massive set of possible tests to a
small group by dividing (partitioning) the set into subsets (subdomains) and picking one or two
representatives from each subset.
In domain testing, we focus on variables, initially one variable at time. To test a given variable,
the set includes all the values (including invalid values) that you can imagine being assigned to
the variable. Partition the set into subdomains and test at least one representative from each
subdomain. Typically, you test with a "best representative", that is, with a value that is at least as
likely to expose an error as any other member of the class. If the variable can be mapped to the
number line, the best representatives are typically boundary values.
Most discussions of domain testing are about input variables whose values can be mapped to the
number line. The best representatives of partitions in these cases are typically boundary cases.
A good set of domain tests for a numeric variable hits every boundary value, including the
minimum, the maximum, a value barely below the minimum, and a value barely above the
maximum.
Whittaker (2003) provides an extensive discussion of the many different types of
variables we can analyze in software, including input variables, output variables, results
of intermediate calculations, values stored in a file system, and data sent to devices or
other programs.
Kaner, Falk & Nguyen (1993) provided a detailed analysis of testing with a variable
(printer type, in configuration testing) that can’t be mapped to a number line.
These tests are higher power than tests that don’t use “best representatives” or that skip some of the subdomains (e.g. people often skip cases that are expected to lead to error messages).
The first time these tests are run, or after significant relevant changes, these tests carry a lot of
information value because boundary / extreme-value errors are common.
Bugs found with these tests are sometimes dismissed, especially when you test extreme values of several variables at the same time. (These tests are called corner cases.) They are not necessarily credible, they don’t necessarily represent what customers will do, and thus they are not necessarily very motivating to stakeholders.
Specification-Based Testing
Check the program against every claim made in a reference document, such as a design
specification, a requirements list, a user interface description, a published model, or a user
manual.
These tests are highly significant (motivating) in companies that take their specifications
seriously. For example, if the specification is part of a contract, conformance to the spec is very
important. Similarly products must conform to their advertisements, and life-critical products
must conform to any safety-related specification.
Specification-driven tests are often weak, not particularly powerful representatives of the class of
tests that could test a given specification item.
Some groups that do specification-based testing focus narrowly on what is written in the
document. To them, a good set of tests includes an unambiguous and relevant test for each claim
made in the spec.
Other groups look further, for problems in the specification. They find that the most informative
tests in a well-specified product are often the ones that explore ambiguities in the spec or
examine aspects of the product that were not well-specified.
Risk-Based Testing
Imagine a way the program could fail and then design one or more tests to check whether the
program will actually fail that in way.
A “complete” set of risk-based tests would be based on an exhaustive risk list, a list of every way
the program could fail.
A good risk-based test is a powerful representative of the class of tests that address a given risk.
To the extent that the tests tie back to significant failures in the field or well known failures in a
competitor’s product, a risk-based failure will be highly credible and highly motivating.
However, many risk-based tests are dismissed as academic (unlikely to occur in real use). Being
able to tie the “risk” (potential failure) you test for to a real failure in the field is very valuable,
and makes tests more credible.
Good Test Cases Copyright © Cem Kaner 2003. All rights reserved. Page 9
Risk-based tests tend to carry high information value because you are testing for a problem that
you have some reason to believe might actually exist in the product. We learn a lot whether the
program passes the test or fails it.
Stress Testing
There are a few different definition of stress tests.
Under one common definition, you hit the program with a peak burst of activity and see it
fail.
IEEE Standard 610.12-1990 defines it as "Testing conducted to evaluate a system or
component at or beyond the limits of its specified requirements with the goal of causing
the system to fail."
A third approach involves driving the program to failure in order to watch how the
program fails. For example, if the test involves excessive input, you don’t just test near
the specified limits. You keep increasing the size or rate of input until either the program
finally fails or you become convinced that further increases won’t yield a failure. The fact
that the program eventually fails might not be particularly surprising or motivating. The
interesting thinking happens when you see the failure and ask what vulnerabilities have
been exposed and which of them might be triggered under less extreme circumstances.
Jorgensen (2003) provides a fascinating example of this style of work.
I work from this third definition.
These tests have high power.
Some people dismiss stress test results as not representative of customer use, and therefore not
credible and not motivating. Another problem with stress testing is that a failure may not be
useful unless the test provides good troubleshooting information, or the lead tester is extremely
familiar with the application.
A good stress test pushes the limit you want to push, and includes enough diagnostic support to
make it reasonably easy for you to investigate a failure once you see it.
Some testers, such as Alberto Savoia (2000), use stress-like tests to expose failures that are hard
to see if the system is not running several tasks concurrently. These failures often show up well
within the theoretical limits of the system and so they are more credible and more motivating.
They are not necessarily easy to troubleshoot.
Regression Testing
Design, develop and save tests with the intent of regularly reusing them, Repeat the tests after
making changes to the program.
This is a good point (consideration of regression testing) to note that this is not an orthogonal list
of test types. You can put domain tests or specification-based tests or any other kinds of tests into
your set of regression tests.
So what’s the difference between these and the others? I’ll answer this by example:
Suppose a tester creates a suite of domain tests and saves them for reuse. Is this domain testing or
regression testing?
I think of it as primarily domain testing if the tester is primarily thinking about
partitioning variables and finding good representatives when she creates the tests.
I think of it as primarily regression testing if the tester is primarily thinking about
building a set of reusable tests.
Regression tests may have been powerful, credible, and so on, when they were first designed.
However, after a test has been run and passed many times, it’s not likely that the program will
fail it the next time, unless there have been major changes or changes in part of the code directly
involved with this test. Thus, most of the time, regression tests carry little information value.
A good regression test is designed for reuse. It is adequately documented and maintainable. (For
suggestions that improve maintainability of GUI-level tests, see Graham & Fewster, 1999;
Kaner, 1998; Pettichord, 2002, and the papers at www.pettichord.com in general). A good
regression test is designed to be likely to fail if changes induce errors in the function(s) or area(s)
of the program addressed by the regression test.
User Testing
User testing is done by users. Not by testers pretending to be users. Not by secretaries or
executives pretending to be testers pretending to be users. By users. People who will make use of the finished product. User tests might be designed by the users or by testers or by other people (sometimes even by lawyers, who included them as acceptance tests in a contract for custom software).
The set of user tests might include boundary tests, stress tests, or any other type of test.
Some user tests are designed in such detail that the user merely executes them and reports
whether the program passed or failed them. This is a good way to design tests if your goal is to
provide a carefully scripted demonstration of the system, without much opportunity for wrong
things to show up as wrong.
If your goal is to discover what problems a user will encounter in real use of the system, your
task is much more difficult. Beta tests are often described as cheap, effective user tests but in
practice they can be quite expensive to administer and they may not yield much information. For some suggestions on beta tests, see Kaner, Falk & Nguyen (1993).
A good user test must allow enough room for cognitive activity by the user while providing
enough structure for the user to report the results effectively (in a way that helps readers
understand and troubleshoot the problem).
Failures found in user testing are typically credible and motivating. Few users run particularly
powerful tests. However, some users run complex scenarios that put the program through its
paces.
Scenario Testing
A scenario is a story that describes a hypothetical situation. In testing, you check how the
program copes with this hypothetical situation.
The ideal scenario test is credible, motivating, easy to evaluate, and complex.
In practice, many scenarios will be weak in at least one of these attributes, but people will still
call them scenarios. The key message of this pattern is that you should keep these four attributes in mind when you design a scenario test and try hard to achieve them.
An important variation of the scenario test involves a harsher test. The story will often involve a
sequence, or data values, that would rarely be used by typical users. They might arise, however,
out of user error or in the course of an unusual but plausible situation, or in the behavior of a
hostile user. Hans Buwalda (2000a, 2000b) calls these "killer soaps" to distinguish them from
normal scenarios, which he calls "soap operas." Such scenarios are common in security testing or
other forms of stress testing.
In the Rational Unified Process, scenarios come from use cases. (Jacobson, Booch, &
Rumbaugh, 1999). Scenarios specify actors, roles, business processes, the goal(s) of the actor(s),
and events that can occur in the course of attempting to achieve the goal. A scenario is an
instantiation of a use case. A simple scenario traces through a single use case, specifying the data
values and thus the path taken through the case. A more complex use case involves
concatenation of several use cases, to track through a given task, end to end. (See also Bittner &
However they are derived, good scenario tests have high power the first time they’re run.
Groups vary in how often they run a given scenario test.
Some groups create a pool of scenario tests as regression tests.
Others (like me) run a scenario once or a small number of times and then design another
scenario rather than sticking with the ones they’ve used before.
Testers often develop scenarios to develop insight into the product. This is especially true early
in testing and again late in testing (when the product has stabilized and the tester is trying to
understand advanced uses of the product.)
In general, comparisons of software behavior to the model are done using automated tests and so
the failures that are found are found easily (easy to evaluate).
In general, state-model-based tests are credible, motivating and easy to troubleshoot. However,
state-based testing often involves simplifications, looking at transitions between operational
modes rather than states, because there are too many states (El-Far 1995). Some abstractions to operational modes are obvious and credible, but others can seem overbroad or otherwise odd to some stakeholders, thereby reducing the value of the tests. Additionally, if the model is
oversimplified, failures exposed by the model can be difficult to troubleshoot (Houghtaling,
2001).
Talking about his experiences in creating state models of software, Harry Robinson (2001)
reported that much of the bug-finding happens while doing the modeling, well before the
automated tests are coded. Elisabeth Hendrickson (2002) trains testers to work with state models
as an exploratory testing tool--her models might never result in automated tests, their value is
that they guide the analysis by the tester.
El-Far, Thompson & Mottay (2001) and El-Far (2001) discuss some of the considerations in
building a good suite of model-based tests. There are important tradeoffs, involving, for
example, the level of detail (more detailed models find more bugs but can be much harder to read
and maintain). For much more, see the papers at www.model-based-testing.org.
High-Volume Automated Testing
High-volume automated testing involves massive numbers of tests, comparing the results against one or more partial oracles.
The simplest partial oracle is running versus crashing. If the program crashes, there must
be a bug. See Nyman (1998, 2002) for details and experience reports.
State-model-based testing can be high volume if the stopping rule is based on the results
of the tests rather than on a coverage criterion. For the general notion of stochastic statebased
testing, see Whittaker (1997). For discussion of state-model-based testing ended
by a coverage stopping rule, see Al-Ghafees & Whittaker (2002).
Jorgensen (2002) provides another example of high-volume testing. He starts with a file
that is valid for the application under test. Then he corrupts it in many ways, in many
places, feeding the corrupted files to the application. The application rejects most of the
bad files and crashes on some. Sometimes, some applications lose control when handling
these files. Buffer overruns or other failures allow the tester to take over the application
or the machine running the application. Any program that will read any type of data
stream can be subject to this type of attack if the tester can modify the data stream before
it reaches the program.
Kaner (2000) describes several other examples of high-volume automated testing
approaches. One classic approach repeatedly feeds random data to the application under
test and to another application that serves as a reference for comparison, an oracle.
Another approach runs an arbitrarily long random sequence of regression tests, tests that
the program has shown it can pass one by one. Memory leaks, stack corruption, wild
pointers or other garbage that cumulates over time finally causes failures in these long
sequences. Yet another approach attacks the program with long sequences of activity and
uses probes (tests built into the program that log warning or failure messages in response
to unexpected conditions) to expose problems.
High-volume testing is a diverse grouping. The essence of it is that the structure of this type of
testing is designed by a person, but the individual test cases are developed, executed, and
interpreted by the computer, which flags suspected failures for human review. The almostcomplete
automation is what makes it possible to run so many tests.
The individual tests are often weak. They make up for low power with massive numbers.
Because the tests are not handcrafted, some tests that expose failures may not be
particularly credible or motivating. A skilled tester often works with a failure to imagine
a broader or more significant range of circumstances under which the failure might arise,
and then craft a test to prove it.
Some high-volume test approaches yield failures that are very hard to troubleshoot. It is
easy to see that the failure occurred in a given test, but one of the necessary conditions
that led to the failure might have been set up thousands of tests before the one that
actually failed. Building troubleshooting support into these tests is a design challenge that
some test groups have tackled more effectively than others.
Exploratory Testing
Exploratory testing is “any testing to the extent that the tester actively controls the design of the
tests as those tests are performed and uses information gained while testing to design new and
better tests” (Bach 2003a).
Bach points out that tests span a continuum between purely scripted (the tester does precisely
what the script specifies and nothing else) to purely exploratory (none of the tester’s activities
are pre-specified and the tester is not required to generate any test documentation beyond bug
reports). Any given testing effort falls somewhere on this continuum. Even predominantly prescripted
Let’s start with the basics. What’s a test case?
IEEE Standard 610 (1990) defines test case as follows:
“(1) A set of test inputs, execution conditions, and expected results developed for a
particular objective, such as to exercise a particular program path or to verify compliance
with a specific requirement.
“(2) (IEEE Std 829-1983) Documentation specifying inputs, predicted results, and a set
of execution conditions for a test item.”
According to Ron Patton (2001, p. 65),
“Test cases are the specific inputs that you’ll try and the procedures that you’ll follow
when you test the software.”
Boris Beizer (1995, p. 3) defines a test as
“A sequence of one or more subtests executed as a sequence because the outcome and/or
final state of one subtest is the input and/or initial state of the next. The word ‘test’ is
used to include subtests, tests proper, and test suites.
defines test case:
“A test case specifies the pretest state of the IUT and its environment, the test inputs or
conditions, and the expected result. The expected result specifies what the IUT should
produce from the test inputs. This specification includes messages generated by the IUT,
exceptions, returned values, and resultant state of the IUT and its environment. Test cases
may also specify initial and resulting conditions for other objects that constitute the IUT
and its environment.”
In practice, many things are referred to as test cases even though they are far from being fully
documented.
Brian Marick uses a related term to describe the lightly documented test case, the test idea:
“A test idea is a brief statement of something that should be tested. For example, if you're
testing a square root function, one idea for a test would be ‘test a number less than zero’.
The idea is to check if the code handles an error case.”
In my view, a test case is a question that you ask of the program. The point of running the test is
to gain information, for example whether the program will pass or fail the test.
It may or may not be specified in great procedural detail, as long as it is clear what is the idea of
the test and how to apply that idea to some specific aspect (feature, for example) of the product.
If the documentation is an essential aspect of a test case, in your vocabulary, please substitute the term “test idea” for “test case” in everything that follows.
An important implication of defining a test case as a question is that a test case must be
reasonably capable of revealing information.
Under this definition, the scope of test cases changes as the program gets more
stable. Early in testing, when anything in the program can be broken, trying the
largest “legal” value in a numeric input field is a sensible test. But weeks later, after
the program has passed this test several times over several builds, a standalone test
of this one field is no longer a test case because there is only a miniscule probability
of failure. A more appropriate test case at this point might combine boundaries of ten
different variables at the same time or place the boundary in the context of a longsequence
test or a scenario.
Also, under this definition, the metrics that report the number of test cases are
meaningless. What do you do with a set of 20 single-variable tests that were
interesting a few weeks ago but now should be retired or merged into a combination?
Suppose you create a combination test that includes the 20 tests. Should the metric
report this one test, 20 tests, or 21? What about the tests that you run only once?
What about the tests that you design and implement but never run because the
program design changes in ways that make these tests uninteresting?
Another implication of the definition is that a test is not necessarily designed to expose a defect.
The goal is information. Very often, the information sought involves defects, but not always. (I
owe this insight to Marick, 1997.) To assess the value of a test, we should ask how well it
provides the information we’re looking for.
Information Objectives
So what are we trying to learn or achieve when we run tests? Here are some examples:
Find defects. This is the classic objective of testing. A test is run in order to trigger
failures that expose defects. Generally, we look for defects in all interesting parts of the
product.
Maximize bug count. The distinction between this and “find defects” is that total number
of bugs is more important than coverage. We might focus narrowly, on only a few highrisk
features, if this is the way to find the most bugs in the time available.
Block premature product releases. This tester stops premature shipment by finding bugs
so serious that no one would ship the product until they are fixed. For every releasedecision
meeting, the tester’s goal is to have new showstopper bugs.
Help managers make ship / no-ship decisions. Managers are typically concerned with
risk in the field. They want to know about coverage (maybe not the simplistic code
coverage statistics, but some indicators of how much of the product has been addressed
and how much is left), and how important the known problems are. Problems that appear
significant on paper but will not lead to customer dissatisfaction are probably not relevant
to the ship decision.
Minimize technical support costs. Working in conjunction with a technical support or
help desk group, the test team identifies the issues that lead to calls for support. These are
often peripherally related to the product under test--for example, getting the product to
work with a specific printer or to import data successfully from a third party database
might prevent more calls than a low-frequency, data-corrupting crash.
Assess conformance to specification. Any claim made in the specification is checked.
Program characteristics not addressed in the specification are not (as part of this
objective) checked.
Conform to regulations. If a regulation specifies a certain type of coverage (such as, at
least one test for every claim made about the product), the test group creates the
appropriate tests. If the regulation specifies a style for the specifications or other
documentation, the test group probably checks the style. In general, the test group is
focusing on anything covered by regulation and (in the context of this objective) nothing
that is not covered by regulation.
Minimize safety-related lawsuit risk. Any error that could lead to an accident or injury is
of primary interest. Errors that lead to loss of time or data or corrupt data, but that don’t
carry a risk of injury or damage to physical things are out of scope.
Find safe scenarios for use of the product (find ways to get it to work, in spite of the
bugs). Sometimes, all that you’re looking for is one way to do a task that will consistently
work--one set of instructions that someone else can follow that will reliably deliver the
benefit they are supposed to lead to. In this case, the tester is not looking for bugs. He is
trying out, empirically refining and documenting, a way to do a task.
Assess quality. This is a tricky objective because quality is multi-dimensional. The nature
of quality depends on the nature of the product. For example, a computer game that is
rock solid but not entertaining is a lousy game. To assess quality -- to measure and report
back on the level of quality -- you probably need a clear definition of the most important
quality criteria for this product, and then you need a theory that relates test results to the
definition. For example, reliability is not just about the number of bugs in the product. It
is (or is often defined as being) about the number of reliability-related failures that can be
expected in a period of time or a period of use. (Reliability-related? In measuring
reliability, an organization might not care, for example, about misspellings in error
messages.) To make this prediction, you need a mathematically and empirically sound
model that links test results to reliability. Testing involves gathering the data needed by
the model. This might involve extensive work in areas of the product believed to be
stable as well as some work in weaker areas. Imagine a reliability model based on
counting bugs found (perhaps weighted by some type of severity) per N lines of code or
per K hours of testing. Finding the bugs is important. Eliminating duplicates is important.
Troubleshooting to make the bug report easier to understand and more likely to fix is (in
the context of assessment) out of scope.
Verify correctness of the product. It is impossible to do this by testing. You can prove
that the product is not correct or you can demonstrate that you didn’t find any errors in a
given period of time using a given testing strategy. However, you can’t test exhaustively,
and the product might fail under conditions that you did not test. The best you can do (if
you have a solid, credible model) is assessment--test-based estimation of the probability
of errors. (See the discussion of reliability, above).
Assure quality. Despite the common title, quality assurance, you can’t assure quality by
testing. You can’t assure quality by gathering metrics. You can’t assure quality by setting
standards. Quality assurance involves building a high quality product and for that, you
need skilled people throughout development who have time and motivation and an
appropriate balance of direction and creative freedom. This is out of scope for a test
organization. It is within scope for the project manager and associated executives. The
test organization can certainly help in this process by performing a wide range of
technical investigations, but those investigations are not quality assurance.
Given a testing objective, the good test series provides information directly relevant to that
objective.
Tests Intended to Expose Defects
Let’s narrow our focus to the test group that has two primary objectives:
Find bugs that the rest of the development group will consider relevant (worth reporting)
and Get these bugs fixed.
Even within these objectives, tests can be good in many different ways. For example, we might
say that one test is better than another if it is:
More powerful. I define power in the usual statistical sense as more likely to expose a
bug if it the bug is there. Note that Test 1 can be more powerful than Test 2 for one type
of bug and less powerful than Test 2 for a different type of bug.
More likely to yield significant (more motivating, more persuasive) results. A problem
is significant if a stakeholder with influence would protest if the problem is not fixed. (A
stakeholder is a person who is affected by the product. A stakeholder with influence is
someone whose preference or opinion might result in change to the product.)
More credible. A credible test is more likely to be taken as a realistic (or reasonable) set
of operations by the programmer or another stakeholder with influence. “Corner case” is
an example of a phrase used by programmers to say that a test or bug is non-credible:
“No one would do that.” A test case is credible if some (or all) stakeholders agree that it
is realistic.
Representative of events more likely to be encountered by the customer. A population of
tests can be designed to be highly credible. Set up your population to reflect actual usage
probabilities. The more frequent clusters of activities are more likely to be covered or
covered more thoroughly. (I say cluster of activities to suggest that many features are
used together and so we might track which combinations of features are used and in what
order, and reflect this more specific information in our analysis.) For more details, read
Musa's (1998) work on software reliability engineering.
Easier to evaluate. The question is, did the program pass or fail the test? Ease of
Evaluation. The tester should be able to determine, quickly and easily, whether the
program passed or failed the test. It is not enough that it is possible to tell whether the
program passed or failed. The harder evaluation is, or the longer it takes, the more likely
it is that failures will slip through unnoticed. Faced with time-consuming evaluation, the
tester will take shortcuts and find ways to less expensively guess whether the program is
OK or not. These shortcuts will typically be imperfectly accurate (that is, they may miss
obvious bugs or they may flag correct code as erroneous.)
More useful for troubleshooting. For example, high volume automated tests will often
crash the system under test without providing much information about the relevant test
conditions needed to reproduce the problem. They are not useful for troubleshooting.
Tests that are harder to repeat are less useful for troubleshooting. Tests that are harder to
perform are less likely to be performed correctly the next time, when you are
troubleshooting a failure that was exposed by this test.
More informative. A test provides value to the extent that we learn from it. In most cases,
you learn more from the test that the program passes than the one the program fails, but
the informative test will teach you something (reduce your uncertainty) whether the
program passes it or fails.
o For example, if we have already run a test in several builds, and the program
reliably passed it each time, we will expect the program to pass this test again.
Another "pass" result from the reused test doesn't contribute anything to our
mental model of the program.
o The notion of equivalence classes provides another example of information value.
Behind any test is a set of tests that are sufficiently similar to it that we think of
the other tests as essentially redundant with this one. In traditional jargon, this is
the "equivalence class" or the "equivalence partition." If the tests are sufficiently
similar, there is little added information to be obtained by running the second one
after running the first.
o This criterion is closely related to Karl Popper’s theory of value of experiments
(See Popper 1992). Good experiments involve risky predictions. The theory
predicts something that many people would expect not to be true. Either your
favorite theory is false or lots of people are surprised. Popper’s analysis of what
makes for good experiments (good tests) is a core belief in a mainstream approach
to the philosophy of science.
Perhaps the essential consideration here is that the expected value of what you
will learn from this test has to be balanced against the opportunity cost of
designing and running the test. The time you spend on this test is time you don't
have available for some other test or other activity.
Appropriately complex. A complex test involves many features, or variables, or other
attributes of the software under test. Complexity is less desirable when the program has
changed in many ways, or when you’re testing many new features at once. If the program
has many bugs, a complex test might fail so quickly that you don’t get to run much of it.
Test groups that rely primarily on complex tests complain of blocking bugs. A blocking
bug causes many tests to fail, preventing the test group from learning the other things
about the program that these tests are supposed to expose. Therefore, early in testing,
simple tests are desirable. As the program gets more stable, or (as in eXtreme
Programming or any evolutionary development lifecycle) as more stable features are
incorporated into the program, greater complexity becomes more desirable.
More likely to help the tester or the programmer develop insight into some aspect of the
product, the customer, or the environment. Sometimes, we test to understand the
product, to learn how it works or where its risks might be. Later, we might design tests to
expose faults, but especially early in testing we are interested in learning what it is and
how to test it. Many tests like this are never reused. However, in a test-first design
environment, code changes are often made experimentally, with the expectation that the
(typically, unit) test suite will alert the programmer to side effects. In such an
environment, a test might be designed to flag a performance change, a difference in
rounding error, or some other change that is not a defect. An unexpected change in
program behavior might alert the programmer that her model of the code or of the impact
of her code change is incomplete or wrong, leading her to additional testing and
troubleshooting. (Thanks to Ward Cunningham and Brian Marick for suggesting this
example.)
Test Styles/Types and Test Qualities
Within the field of black box testing, Kaner & Bach (see our course notes, Bach 2003b and
Kaner, 2002, posted at www.testingeducation.org, and see Kaner, Bach & Pettichord, 2002) have described eleven dominant styles of black box testing:
Function testing
Domain testing
Specification-based testing
Risk-based testing
Stress testing
Regression testing
User testing
Scenario testing
State-model based testing
High volume automated testing
Exploratory testing
Bach and I call these "paradigms" of testing because we have seen time and again that one or two of them dominate the thinking of a testing group or a talented tester. An analysis we find
intriguing goes like this:
If I was a "scenario tester" (a person who defines testing primarily in terms of application
of scenario tests), how would I actually test the program? What makes one scenario test
better than another? Why types of problems would I tend to miss, what would be difficult
for me to find or interpret, and what would be particularly easy?
Here are thumbnail sketches of the styles, with some thoughts on how test cases are “good”
within them.
Function Testing
Test each function / feature / variable in isolation.
Most test groups start with fairly simple function testing but then switch to a different style, often
involving the interaction of several functions, once the program passes the mainstream function
tests.
Within this approach, a good test focuses on a single function and tests it with middle-of-theroad
values. We don’t expect the program to fail a test like this, but it will if the algorithm is
fundamentally wrong, the build is broken, or a change to some other part of the program has
fowled this code.
These tests are highly credible and easy to evaluate but not particularly powerful.
Some test groups spend most of their effort on function tests. For them, testing is complete when
every item has been thoroughly tested on its own. In my experience, the tougher function tests
look like domain tests and have their strengths.
Domain Testing
The essence of this type of testing is sampling. We reduce a massive set of possible tests to a
small group by dividing (partitioning) the set into subsets (subdomains) and picking one or two
representatives from each subset.
In domain testing, we focus on variables, initially one variable at time. To test a given variable,
the set includes all the values (including invalid values) that you can imagine being assigned to
the variable. Partition the set into subdomains and test at least one representative from each
subdomain. Typically, you test with a "best representative", that is, with a value that is at least as
likely to expose an error as any other member of the class. If the variable can be mapped to the
number line, the best representatives are typically boundary values.
Most discussions of domain testing are about input variables whose values can be mapped to the
number line. The best representatives of partitions in these cases are typically boundary cases.
A good set of domain tests for a numeric variable hits every boundary value, including the
minimum, the maximum, a value barely below the minimum, and a value barely above the
maximum.
Whittaker (2003) provides an extensive discussion of the many different types of
variables we can analyze in software, including input variables, output variables, results
of intermediate calculations, values stored in a file system, and data sent to devices or
other programs.
Kaner, Falk & Nguyen (1993) provided a detailed analysis of testing with a variable
(printer type, in configuration testing) that can’t be mapped to a number line.
These tests are higher power than tests that don’t use “best representatives” or that skip some of the subdomains (e.g. people often skip cases that are expected to lead to error messages).
The first time these tests are run, or after significant relevant changes, these tests carry a lot of
information value because boundary / extreme-value errors are common.
Bugs found with these tests are sometimes dismissed, especially when you test extreme values of several variables at the same time. (These tests are called corner cases.) They are not necessarily credible, they don’t necessarily represent what customers will do, and thus they are not necessarily very motivating to stakeholders.
Specification-Based Testing
Check the program against every claim made in a reference document, such as a design
specification, a requirements list, a user interface description, a published model, or a user
manual.
These tests are highly significant (motivating) in companies that take their specifications
seriously. For example, if the specification is part of a contract, conformance to the spec is very
important. Similarly products must conform to their advertisements, and life-critical products
must conform to any safety-related specification.
Specification-driven tests are often weak, not particularly powerful representatives of the class of
tests that could test a given specification item.
Some groups that do specification-based testing focus narrowly on what is written in the
document. To them, a good set of tests includes an unambiguous and relevant test for each claim
made in the spec.
Other groups look further, for problems in the specification. They find that the most informative
tests in a well-specified product are often the ones that explore ambiguities in the spec or
examine aspects of the product that were not well-specified.
Risk-Based Testing
Imagine a way the program could fail and then design one or more tests to check whether the
program will actually fail that in way.
A “complete” set of risk-based tests would be based on an exhaustive risk list, a list of every way
the program could fail.
A good risk-based test is a powerful representative of the class of tests that address a given risk.
To the extent that the tests tie back to significant failures in the field or well known failures in a
competitor’s product, a risk-based failure will be highly credible and highly motivating.
However, many risk-based tests are dismissed as academic (unlikely to occur in real use). Being
able to tie the “risk” (potential failure) you test for to a real failure in the field is very valuable,
and makes tests more credible.
Good Test Cases Copyright © Cem Kaner 2003. All rights reserved. Page 9
Risk-based tests tend to carry high information value because you are testing for a problem that
you have some reason to believe might actually exist in the product. We learn a lot whether the
program passes the test or fails it.
Stress Testing
There are a few different definition of stress tests.
Under one common definition, you hit the program with a peak burst of activity and see it
fail.
IEEE Standard 610.12-1990 defines it as "Testing conducted to evaluate a system or
component at or beyond the limits of its specified requirements with the goal of causing
the system to fail."
A third approach involves driving the program to failure in order to watch how the
program fails. For example, if the test involves excessive input, you don’t just test near
the specified limits. You keep increasing the size or rate of input until either the program
finally fails or you become convinced that further increases won’t yield a failure. The fact
that the program eventually fails might not be particularly surprising or motivating. The
interesting thinking happens when you see the failure and ask what vulnerabilities have
been exposed and which of them might be triggered under less extreme circumstances.
Jorgensen (2003) provides a fascinating example of this style of work.
I work from this third definition.
These tests have high power.
Some people dismiss stress test results as not representative of customer use, and therefore not
credible and not motivating. Another problem with stress testing is that a failure may not be
useful unless the test provides good troubleshooting information, or the lead tester is extremely
familiar with the application.
A good stress test pushes the limit you want to push, and includes enough diagnostic support to
make it reasonably easy for you to investigate a failure once you see it.
Some testers, such as Alberto Savoia (2000), use stress-like tests to expose failures that are hard
to see if the system is not running several tasks concurrently. These failures often show up well
within the theoretical limits of the system and so they are more credible and more motivating.
They are not necessarily easy to troubleshoot.
Regression Testing
Design, develop and save tests with the intent of regularly reusing them, Repeat the tests after
making changes to the program.
This is a good point (consideration of regression testing) to note that this is not an orthogonal list
of test types. You can put domain tests or specification-based tests or any other kinds of tests into
your set of regression tests.
So what’s the difference between these and the others? I’ll answer this by example:
Suppose a tester creates a suite of domain tests and saves them for reuse. Is this domain testing or
regression testing?
I think of it as primarily domain testing if the tester is primarily thinking about
partitioning variables and finding good representatives when she creates the tests.
I think of it as primarily regression testing if the tester is primarily thinking about
building a set of reusable tests.
Regression tests may have been powerful, credible, and so on, when they were first designed.
However, after a test has been run and passed many times, it’s not likely that the program will
fail it the next time, unless there have been major changes or changes in part of the code directly
involved with this test. Thus, most of the time, regression tests carry little information value.
A good regression test is designed for reuse. It is adequately documented and maintainable. (For
suggestions that improve maintainability of GUI-level tests, see Graham & Fewster, 1999;
Kaner, 1998; Pettichord, 2002, and the papers at www.pettichord.com in general). A good
regression test is designed to be likely to fail if changes induce errors in the function(s) or area(s)
of the program addressed by the regression test.
User Testing
User testing is done by users. Not by testers pretending to be users. Not by secretaries or
executives pretending to be testers pretending to be users. By users. People who will make use of the finished product. User tests might be designed by the users or by testers or by other people (sometimes even by lawyers, who included them as acceptance tests in a contract for custom software).
The set of user tests might include boundary tests, stress tests, or any other type of test.
Some user tests are designed in such detail that the user merely executes them and reports
whether the program passed or failed them. This is a good way to design tests if your goal is to
provide a carefully scripted demonstration of the system, without much opportunity for wrong
things to show up as wrong.
If your goal is to discover what problems a user will encounter in real use of the system, your
task is much more difficult. Beta tests are often described as cheap, effective user tests but in
practice they can be quite expensive to administer and they may not yield much information. For some suggestions on beta tests, see Kaner, Falk & Nguyen (1993).
A good user test must allow enough room for cognitive activity by the user while providing
enough structure for the user to report the results effectively (in a way that helps readers
understand and troubleshoot the problem).
Failures found in user testing are typically credible and motivating. Few users run particularly
powerful tests. However, some users run complex scenarios that put the program through its
paces.
Scenario Testing
A scenario is a story that describes a hypothetical situation. In testing, you check how the
program copes with this hypothetical situation.
The ideal scenario test is credible, motivating, easy to evaluate, and complex.
In practice, many scenarios will be weak in at least one of these attributes, but people will still
call them scenarios. The key message of this pattern is that you should keep these four attributes in mind when you design a scenario test and try hard to achieve them.
An important variation of the scenario test involves a harsher test. The story will often involve a
sequence, or data values, that would rarely be used by typical users. They might arise, however,
out of user error or in the course of an unusual but plausible situation, or in the behavior of a
hostile user. Hans Buwalda (2000a, 2000b) calls these "killer soaps" to distinguish them from
normal scenarios, which he calls "soap operas." Such scenarios are common in security testing or
other forms of stress testing.
In the Rational Unified Process, scenarios come from use cases. (Jacobson, Booch, &
Rumbaugh, 1999). Scenarios specify actors, roles, business processes, the goal(s) of the actor(s),
and events that can occur in the course of attempting to achieve the goal. A scenario is an
instantiation of a use case. A simple scenario traces through a single use case, specifying the data
values and thus the path taken through the case. A more complex use case involves
concatenation of several use cases, to track through a given task, end to end. (See also Bittner &
However they are derived, good scenario tests have high power the first time they’re run.
Groups vary in how often they run a given scenario test.
Some groups create a pool of scenario tests as regression tests.
Others (like me) run a scenario once or a small number of times and then design another
scenario rather than sticking with the ones they’ve used before.
Testers often develop scenarios to develop insight into the product. This is especially true early
in testing and again late in testing (when the product has stabilized and the tester is trying to
understand advanced uses of the product.)
In general, comparisons of software behavior to the model are done using automated tests and so
the failures that are found are found easily (easy to evaluate).
In general, state-model-based tests are credible, motivating and easy to troubleshoot. However,
state-based testing often involves simplifications, looking at transitions between operational
modes rather than states, because there are too many states (El-Far 1995). Some abstractions to operational modes are obvious and credible, but others can seem overbroad or otherwise odd to some stakeholders, thereby reducing the value of the tests. Additionally, if the model is
oversimplified, failures exposed by the model can be difficult to troubleshoot (Houghtaling,
2001).
Talking about his experiences in creating state models of software, Harry Robinson (2001)
reported that much of the bug-finding happens while doing the modeling, well before the
automated tests are coded. Elisabeth Hendrickson (2002) trains testers to work with state models
as an exploratory testing tool--her models might never result in automated tests, their value is
that they guide the analysis by the tester.
El-Far, Thompson & Mottay (2001) and El-Far (2001) discuss some of the considerations in
building a good suite of model-based tests. There are important tradeoffs, involving, for
example, the level of detail (more detailed models find more bugs but can be much harder to read
and maintain). For much more, see the papers at www.model-based-testing.org.
High-Volume Automated Testing
High-volume automated testing involves massive numbers of tests, comparing the results against one or more partial oracles.
The simplest partial oracle is running versus crashing. If the program crashes, there must
be a bug. See Nyman (1998, 2002) for details and experience reports.
State-model-based testing can be high volume if the stopping rule is based on the results
of the tests rather than on a coverage criterion. For the general notion of stochastic statebased
testing, see Whittaker (1997). For discussion of state-model-based testing ended
by a coverage stopping rule, see Al-Ghafees & Whittaker (2002).
Jorgensen (2002) provides another example of high-volume testing. He starts with a file
that is valid for the application under test. Then he corrupts it in many ways, in many
places, feeding the corrupted files to the application. The application rejects most of the
bad files and crashes on some. Sometimes, some applications lose control when handling
these files. Buffer overruns or other failures allow the tester to take over the application
or the machine running the application. Any program that will read any type of data
stream can be subject to this type of attack if the tester can modify the data stream before
it reaches the program.
Kaner (2000) describes several other examples of high-volume automated testing
approaches. One classic approach repeatedly feeds random data to the application under
test and to another application that serves as a reference for comparison, an oracle.
Another approach runs an arbitrarily long random sequence of regression tests, tests that
the program has shown it can pass one by one. Memory leaks, stack corruption, wild
pointers or other garbage that cumulates over time finally causes failures in these long
sequences. Yet another approach attacks the program with long sequences of activity and
uses probes (tests built into the program that log warning or failure messages in response
to unexpected conditions) to expose problems.
High-volume testing is a diverse grouping. The essence of it is that the structure of this type of
testing is designed by a person, but the individual test cases are developed, executed, and
interpreted by the computer, which flags suspected failures for human review. The almostcomplete
automation is what makes it possible to run so many tests.
The individual tests are often weak. They make up for low power with massive numbers.
Because the tests are not handcrafted, some tests that expose failures may not be
particularly credible or motivating. A skilled tester often works with a failure to imagine
a broader or more significant range of circumstances under which the failure might arise,
and then craft a test to prove it.
Some high-volume test approaches yield failures that are very hard to troubleshoot. It is
easy to see that the failure occurred in a given test, but one of the necessary conditions
that led to the failure might have been set up thousands of tests before the one that
actually failed. Building troubleshooting support into these tests is a design challenge that
some test groups have tackled more effectively than others.
Exploratory Testing
Exploratory testing is “any testing to the extent that the tester actively controls the design of the
tests as those tests are performed and uses information gained while testing to design new and
better tests” (Bach 2003a).
Bach points out that tests span a continuum between purely scripted (the tester does precisely
what the script specifies and nothing else) to purely exploratory (none of the tester’s activities
are pre-specified and the tester is not required to generate any test documentation beyond bug
reports). Any given testing effort falls somewhere on this continuum. Even predominantly prescripted
QTP Questions / Load Runner
What are the Features & Benefits of Quick Test Pro (QTP 8.0)? - Operates stand-alone, or integrated into Mercury Business Process Testing and Mercury Quality Center. Introduces next-generation zero-configuration Keyword Driven testing technology in Quick Test Professional 8.0 allowing for fast test creation, easier maintenance, and more powerful data-driving capability. Identifies objects with Unique Smart Object Recognition, even if they change from build to build, enabling reliable unattended script execution. Collapses test documentation and test creation to a single step with Auto-documentation technology. Enables thorough validation of applications through a full complement of checkpoints.
How to handle the exceptions using recovery scenario manager in QTP? - There are 4 trigger events during which a recovery scenario should be activated. A pop up window appears in an opened application during the test run: A property of an object changes its state or value, A step in the test does not run successfully, An open application fails during the test run, These triggers are considered as exceptions. You can instruct QTP to recover unexpected events or errors that occurred in your testing environment during test run. Recovery scenario manager provides a wizard that guides you through the defining recovery scenario. Recovery scenario has three steps: 1. Triggered Events 2. Recovery steps 3. Post Recovery Test-Run
What is the use of Text output value in QTP? - Output values enable to view the values that the application talks during run time. When parameterized, the values change for each iteration. Thus by creating output values, we can capture the values that the application takes for each run and output them to the data table.
How to use the Object spy in QTP 8.0 version? - There are two ways to Spy the objects in QTP: 1) Thru file toolbar, In the File Toolbar click on the last toolbar button (an icon showing a person with hat). 2) True Object repository Dialog, In Object repository dialog click on the button object spy. In the Object spy Dialog click on the button showing hand symbol. The pointer now changes in to a hand symbol and we have to point out the object to spy the state of the object if at all the object is not visible. or window is minimized then, hold the Ctrl button and activate the required window to and release the Ctrl button.
How Does Run time data (Parameterization) is handled in QTP? - You can then enter test data into the Data Table, an integrated spreadsheet with the full functionality of Excel, to manipulate data sets and create multiple test iterations, without programming, to expand test case coverage. Data can be typed in or imported from databases, spreadsheets, or text files.
What is keyword view and Expert view in QTP? - Quick Test’s Keyword Driven approach, test automation experts have full access to the underlying test and object properties, via an integrated scripting and debugging environment that is round-trip synchronized with the Keyword View. Advanced testers can view and edit their tests in the Expert View, which reveals the underlying industry-standard VBScript that Quick Test Professional automatically generates. Any changes made in the Expert View are automatically synchronized with the Keyword View.
Explain about the Test Fusion Report of QTP? - Once a tester has run a test, a Test Fusion report displays all aspects of the test run: a high-level results overview, an expandable Tree View of the test specifying exactly where application failures occurred, the test data used, application screen shots for every step that highlight any discrepancies, and detailed explanations of each checkpoint pass and failure. By combining Test Fusion reports with Quick Test Professional, you can share reports across an entire QA and development team.
Which environments does QTP support? - Quick Test Professional supports functional testing of all enterprise environments, including Windows, Web,..NET, Java/J2EE, SAP, Siebel, Oracle, PeopleSoft, Visual Basic, ActiveX, mainframe terminal emulators, and Web services.
What is QTP? - Quick Test is a graphical interface record-playback automation tool. It is able to work with any web, java or windows client application. Quick Test enables you to test standard web objects and ActiveX controls. In addition to these environments, Quick Test Professional also enables you to test Java applets and applications and multimedia objects on Applications as well as standard Windows applications, Visual Basic 6 applications and.NET framework applications
Explain QTP Testing process? - Quick Test testing process consists of 6 main phases:
Create your test plan - Prior to automating there should be a detailed description of the test including the exact steps to follow, data to be input, and all items to be verified by the test. The verification information should include both data validations and existence or state verifications of objects in the application.
Recording a session on your application - As you navigate through your application, Quick Test graphically displays each step you perform in the form of a collapsible icon-based test tree. A step is any user action that causes or makes a change in your site, such as clicking a link or image, or entering data in a form.
Enhancing your test - Inserting checkpoints into your test lets you search for a specific value of a page, object or text string, which helps you identify whether or not your application is functioning correctly. NOTE: Checkpoints can be added to a test as you record it or after the fact via the Active Screen. It is much easier and faster to add the checkpoints during the recording process. Broadening the scope of your test by replacing fixed values with parameters lets you check how your application performs the same operations with multiple sets of data. Adding logic and conditional statements to your test enables you to add sophisticated checks to your test.
Debugging your test - If changes were made to the script, you need to debug it to check that it operates smoothly and without interruption.
Running your test on a new version of your application - You run a test to check the behavior of your application. While running, Quick Test connects to your application and performs each step in your test.
Analyzing the test results - You examine the test results to pinpoint defects in your application.
Reporting defects - As you encounter failures in the application when analyzing test results, you will create defect reports in Defect Reporting Tool.
Explain the QTP Tool interface. - It contains the following key elements: Title bar, displaying the name of the currently open test, Menu bar, displaying menus of Quick Test commands, File toolbar, containing buttons to assist you in managing tests, Test toolbar, containing buttons used while creating and maintaining tests, Debug toolbar, containing buttons used while debugging tests. Note: The Debug toolbar is not displayed when you open Quick Test for the first time. You can display the Debug toolbar by choosing View — Toolbars — Debug. Action toolbar, containing buttons and a list of actions, enabling you to view the details of an individual action or the entire test flow. Note: The Action toolbar is not displayed when you open Quick Test for the first time. You can display the Action toolbar by choosing View — Toolbars — Action. If you insert a reusable or external action in a test, the Action toolbar is displayed automatically. Test pane, containing two tabs to view your test-the Tree View and the Expert View ,Test Details pane, containing the Active Screen. Data Table, containing two tabs, Global and Action, to assist you in parameterizing your test. Debug Viewer pane, containing three tabs to assist you in debugging your test-Watch Expressions, Variables, and Command. (The Debug Viewer pane can be opened only when a test run pauses at a breakpoint.) Status bar, displaying the status of the test
How does QTP recognize Objects in AUT? - Quick Test stores the definitions for application objects in a file called the Object Repository. As you record your test, Quick Test will add an entry for each item you interact with. Each Object Repository entry will be identified by a logical name (determined automatically by Quick Test), and will contain a set of properties (type, name, etc) that uniquely identify each object. Each line in the Quick Test script will contain a reference to the object that you interacted with, a call to the appropriate method (set, click, check) and any parameters for that method (such as the value for a call to the set method). The references to objects in the script will all be identified by the logical name, rather than any physical, descriptive properties.
What are the types of Object Repositories in QTP? - Quick Test has two types of object repositories for storing object information: shared object repositories and action object repositories. You can choose which type of object repository you want to use as the default type for new tests, and you can change the default as necessary for each new test. The object repository per-action mode is the default setting. In this mode, Quick Test automatically creates an object repository file for each action in your test so that you can create and run tests without creating, choosing, or modifying object repository files. However, if you do modify values in an action object repository, your changes do not have any effect on other actions. Therefore, if the same test object exists in more than one action and you modify an object’s property values in one action, you may need to make the same change in every action (and any test) containing the object.
Explain the check points in QTP? - A checkpoint verifies that expected information is displayed in an Application while the test is running. You can add eight types of checkpoints to your test for standard web objects using QTP. A page checkpoint checks the characteristics of an Application. A text checkpoint checks that a text string is displayed in the appropriate place on an Application. An object checkpoint (Standard) checks the values of an object on an Application. An image checkpoint checks the values of an image on an Application. A table checkpoint checks information within a table on a Application. An Accessibility checkpoint checks the web page for Section 508 compliance. An XML checkpoint checks the contents of individual XML data files or XML documents that are part of your Web application. A database checkpoint checks the contents of databases accessed by your web site
In how many ways we can add check points to an application using QTP? - We can add checkpoints while recording the application or we can add after recording is completed using Active screen (Note: To perform the second one The Active screen must be enabled while recording).
How does QTP identify objects in the application? - QTP identifies the object in the application by Logical Name and Class.
What is Parameterizing Tests? - When you test your application, you may want to check how it performs the same operations with multiple sets of data. For example, suppose you want to check how your application responds to ten separate sets of data. You could record ten separate tests, each with its own set of data. Alternatively, you can create a parameterized test that runs ten times: each time the test runs, it uses a different set of data.
What is test object model in QTP? - The test object model is a large set of object types or classes that Quick Test uses to represent the objects in your application. Each test object class has a list of properties that can uniquely identify objects of that class and a set of relevant methods that Quick Test can record for it. A test object is an object that Quick Test creates in the test or component to represent the actual object in your application. Quick Test stores information about the object that will help it identify and check the object during the run session.
What is Object Spy in QTP? - Using the Object Spy, you can view the properties of any object in an open application. You use the Object Spy pointer to point to an object. The Object Spy displays the selected object’s hierarchy tree and its properties and values in the Properties tab of the Object Spy dialog box.
What is the Diff between Image check-point and Bit map Check point? - Image checkpoints enable you to check the properties of a Web image. You can check an area of a Web page or application as a bitmap. While creating a test or component, you specify the area you want to check by selecting an object. You can check an entire object or any area within an object. Quick Test captures the specified object as a bitmap, and inserts a checkpoint in the test or component. You can also choose to save only the selected area of the object with your test or component in order to save disk Space. For example, suppose you have a Web site that can display a map of a city the user specifies. The map has control keys for zooming. You can record the new map that is displayed after one click on the control key that zooms in the map. Using the bitmap checkpoint, you can check that the map zooms in correctly. You can create bitmap checkpoints for all supported testing environments (as long as the appropriate add-ins are loaded). Note: The results of bitmap checkpoints may be affected by factors such as operating system, screen resolution, and color settings.
How many ways we can parameterize data in QTP? - There are four types of parameters: Test, action or component parameters enable you to use values passed from your test or component, or values from other actions in your test. Data Table parameters enable you to create a data-driven test (or action) that runs several times using the data you supply. In each repetition, or iteration, Quick Test uses a different value from the Data Table. Environment variable parameters enable you to use variable values from other sources during the run session. These may be values you supply, or values that Quick Test generates for you based on conditions and options you choose. Random number parameters enable you to insert random numbers as values in your test or component. For example, to check how your application handles small and large ticket orders, you can have Quick Test generate a random number and insert it in a number of tickets edit field.
How do u do batch testing in WR & is it possible to do in QTP, if so explain? - Batch Testing in WR is nothing but running the whole test set by selecting Run Test set from the Execution Grid. The same is possible with QTP also. If our test cases are automated then by selecting Run Test set all the test scripts can be executed. In this process the Scripts get executed one by one by keeping all the remaining scripts in Waiting mode.
If I give some thousand tests to execute in 2 days what do u do? – Ad-hoc testing is done. It Covers the least basic functionalities to verify that the system is working fine.
What does it mean when a check point is in red color? what do u do? - A red color indicates failure. Here we analyze the cause for failure whether it is a Script Issue or Environment Issue or a Application issue.
What is Object Spy in QTP? - Using the Object Spy, you can view the properties of any object in an open application. You use the Object Spy pointer to point to an object. The Object Spy displays the selected object’s hierarchy tree and its properties and values in the Properties tab of the Object Spy dialog box.
What is the file extension of the code file & object repository file in QTP? - Code file extension is.vbs and object repository is.tsr
Explain the concept of object repository & how QTP recognizes objects? - Object Repository: displays a tree of all objects in the current component or in the current action or entire test (depending on the object repository mode you selected). We can view or modify the test object description of any test object in the repository or to add new objects to the repository. Quicktest learns the default property values and determines in which test object class it fits. If it is not enough it adds assistive properties, one by one to the description until it has compiled the unique description. If no assistive properties are available, then it adds a special Ordinal identifier such as objects location on the page or in the source code.
What are the properties you would use for identifying a browser & page when using descriptive programming? - Name would be another property apart from title that we can use.
Give me an example where you have used a COM interface in your QTP project? - Com interface appears in the scenario of front end and back end. for eg:if you r using oracle as back end and front end as VB or any language then for better compatibility we will go for an interface. of which COM will be one among those interfaces. Create object creates handle to the instance of the specified object so that we program can use the methods on the specified object. It is used for implementing Automation(as defined by Microsoft).
Explain in brief about the QTP Automation Object Model. - Essentially all configuration and run functionality provided via the Quick Test interface is in some way represented in the Quick Test automation object model via objects, methods, and properties. Although a one-on-one comparison cannot always be made, most dialog boxes in Quick Test have a corresponding automation object, most options in dialog boxes can be set and/or retrieved using the corresponding object property, and most menu commands and other operations have corresponding automation methods. You can use the objects, methods, and properties exposed by the Quick Test automation object model, along with standard programming elements such as loops and conditional statements to design your program.
Load Runner Questions
1. What is load testing? - Load testing is to test that if the application works fine with the loads that result from large number of simultaneous users, transactions and to determine weather it can handle peak usage periods.
2. What is Performance testing? - Timing for both read and update transactions should be gathered to determine whether system functions are being performed in an acceptable timeframe. This should be done standalone and then in a multi user environment to determine the effect of multiple transactions on the timing of a single transaction.
3. Did u use LoadRunner? What version? - Yes. Version 7.2.
4. Explain the Load testing process? -Step 1: Planning the test. Here, we develop a clearly defined test plan to ensure the test scenarios we develop will accomplish load-testing objectives. Step 2: Creating Vusers. Here, we create Vuser scripts that contain tasks performed by each Vuser, tasks performed by Vusers as a whole, and tasks measured as transactions. Step 3: Creating the scenario. A scenario describes the events that occur during a testing session. It includes a list of machines, scripts, and Vusers that run during the scenario. We create scenarios using LoadRunner Controller. We can create manual scenarios as well as goal-oriented scenarios. In manual scenarios, we define the number of Vusers, the load generator machines, and percentage of Vusers to be assigned to each script. For web tests, we may create a goal-oriented scenario where we define the goal that our test has to achieve. LoadRunner automatically builds a scenario for us. Step 4: Running the scenario.We emulate load on the server by instructing multiple Vusers to perform tasks simultaneously. Before the testing, we set the scenario configuration and scheduling. We can run the entire scenario, Vuser groups, or individual Vusers. Step 5: Monitoring the scenario.We monitor scenario execution using the LoadRunner online runtime, transaction, system resource, Web resource, Web server resource, Web application server resource, database server resource, network delay, streaming media resource, firewall server resource, ERP server resource, and Java performance monitors. Step 6: Analyzing test results. During scenario execution, LoadRunner records the performance of the application under different loads. We use LoadRunner’s graphs and reports to analyze the application’s performance.
5. When do you do load and performance Testing? - We perform load testing once we are done with interface (GUI) testing. Modern system architectures are large and complex. Whereas single user testing primarily on functionality and user interface of a system component, application testing focuses on performance and reliability of an entire system. For example, a typical application-testing scenario might depict 1000 users logging in simultaneously to a system. This gives rise to issues such as what is the response time of the system, does it crash, will it go with different software applications and platforms, can it hold so many hundreds and thousands of users, etc. This is when we set do load and performance testing.
6. What are the components of LoadRunner? - The components of LoadRunner are The Virtual User Generator, Controller, and the Agent process, LoadRunner Analysis and Monitoring, LoadRunner Books Online.
7. What Component of LoadRunner would you use to record a Script? - The Virtual User Generator (VuGen) component is used to record a script. It enables you to develop Vuser scripts for a variety of application types and communication protocols.
8. What Component of LoadRunner would you use to play Back the script in multi user mode? - The Controller component is used to playback the script in multi-user mode. This is done during a scenario run where a vuser script is executed by a number of vusers in a group.
9. What is a rendezvous point? - You insert rendezvous points into Vuser scripts to emulate heavy user load on the server. Rendezvous points instruct Vusers to wait during test execution for multiple Vusers to arrive at a certain point, in order that they may simultaneously perform a task. For example, to emulate peak load on the bank server, you can insert a rendezvous point instructing 100 Vusers to deposit cash into their accounts at the same time.
10. What is a scenario? - A scenario defines the events that occur during each testing session. For example, a scenario defines and controls the number of users to emulate, the actions to be performed, and the machines on which the virtual users run their emulations.
11. Explain the recording mode for web Vuser script? - We use VuGen to develop a Vuser script by recording a user performing typical business processes on a client application. VuGen creates the script by recording the activity between the client and the server. For example, in web based applications, VuGen monitors the client end of the database and traces all the requests sent to, and received from, the database server. We use VuGen to: Monitor the communication between the application and the server; Generate the required function calls; and Insert the generated function calls into a Vuser script.
12. Why do you create parameters? - Parameters are like script variables. They are used to vary input to the server and to emulate real users. Different sets of data are sent to the server each time the script is run. Better simulate the usage model for more accurate testing from the Controller; one script can emulate many different users on the system.
13. What is correlation? Explain the difference between automatic correlation and manual correlation? - Correlation is used to obtain data which are unique for each run of the script and which are generated by nested queries. Correlation provides the value to avoid errors arising out of duplicate values and also optimizing the code (to avoid nested queries). Automatic correlation is where we set some rules for correlation. It can be application server specific. Here values are replaced by data which are created by these rules. In manual correlation, the value we want to correlate is scanned and create correlation is used to correlate.
14. How do you find out where correlation is required? Give few examples from your projects? - Two ways: First we can scan for correlations, and see the list of values which can be correlated. From this we can pick a value to be correlated. Secondly, we can record two scripts and compare them. We can look up the difference file to see for the values which needed to be correlated. In my project, there was a unique id developed for each customer, it was nothing but Insurance Number, it was generated automatically and it was sequential and this value was unique. I had to correlate this value, in order to avoid errors while running my script. I did using scan for correlation.
15. Where do you set automatic correlation options? - Automatic correlation from web point of view can be set in recording options and correlation tab. Here we can enable correlation for the entire script and choose either issue online messages or offline actions, where we can define rules for that correlation. Automatic correlation for database can be done using show output window and scan for correlation and picking the correlate query tab and choose which query value we want to correlate. If we know the specific value to be correlated, we just do create correlation for the value and specify how the value to be created.
16. What is a function to capture dynamic values in the web Vuser script? - Web_reg_save_param function saves dynamic data information to a parameter.
17. When do you disable log in Virtual User Generator, When do you choose standard and extended logs? - Once we debug our script and verify that it is functional, we can enable logging for errors only. When we add a script to a scenario, logging is automatically disabled. Standard Log Option: When you selectStandard log, it creates a standard log of functions and messages sent during script execution to use for debugging. Disable this option for large load testing scenarios. When you copy a script to a scenario, logging is automatically disabled Extended Log Option: Selectextended log to create an extended log, including warnings and other messages. Disable this option for large load testing scenarios. When you copy a script to a scenario, logging is automatically disabled. We can specify which additional information should be added to the extended log using the Extended log options.
18. How do you debug a LoadRunner script? - VuGen contains two options to help debug Vuser scripts-the Run Step by Step command and breakpoints. The Debug settings in the Options dialog box allow us to determine the extent of the trace to be performed during scenario execution. The debug information is written to the Output window. We can manually set the message class within your script using the lr_set_debug_message function. This is useful if we want to receive debug information about a small section of the script only.
19. How do you write user defined functions in LR? Give me few functions you wrote in your previous project? - Before we create the User Defined functions we need to create the externallibrary (DLL) with the function. We add this library to VuGen bin directory. Once the library is added then we assign user defined function as a parameter. The function should have the following format: __declspec (dllexport) char*(char*, char*)Examples of user defined functions are as follows:GetVersion, GetCurrentTime, GetPltform are some of the user defined functions used in my earlier project.
20. What are the changes you can make in run-time settings? - The Run Time Settings that we make are: a) Pacing - It has iteration count. b) Log - Under this we have Disable Logging Standard Log and c) Extended Think Time - In think time we have two options like Ignore think time and Replay think time. d) General - Under general tab we can set the vusers as process or as multithreading and whether each step as a transaction.
21. Where do you set Iteration for Vuser testing? - We set Iterations in the Run Time Settings of the VuGen. The navigation for this is Run time settings, Pacing tab, set number of iterations.
22. How do you perform functional testing under load? - Functionality under load can be tested by running several Vusers concurrently. By increasing the amount of Vusers, we can determine how much load the server can sustain.
23. What is Ramp up? How do you set this? - This option is used to gradually increase the amount of Vusers/load on the server. An initial value is set and a value to wait between intervals can bespecified. To set Ramp Up, go to ‘Scenario Scheduling Options’
24. What is the advantage of running the Vuser as thread? - VuGen provides the facility to use multithreading. This enables more Vusers to be run pergenerator. If the Vuser is run as a process, the same driver program is loaded into memory for each Vuser, thus taking up a large amount of memory. This limits the number of Vusers that can be run on a singlegenerator. If the Vuser is run as a thread, only one instance of the driver program is loaded into memory for the given number ofVusers (say 100). Each thread shares the memory of the parent driver program, thus enabling more Vusers to be run per generator.
25. If you want to stop the execution of your script on error, how do you do that? - The lr_abort function aborts the execution of a Vuser script. It instructs the Vuser to stop executing the Actions section, execute the vuser_end section and end the execution. This function is useful when you need to manually abort a script execution as a result of a specific error condition. When you end a script using this function, the Vuser is assigned the status "Stopped". For this to take effect, we have to first uncheck the “Continue on error” option in Run-Time Settings.
26. What is the relation between Response Time and Throughput? - The Throughput graph shows the amount of data in bytes that the Vusers received from the server in a second. When we compare this with the transaction response time, we will notice that as throughput decreased, the response time also decreased. Similarly, the peak throughput and highest response time would occur approximately at the same time.
27. Explain the Configuration of your systems? - The configuration of our systems refers to that of the client machines on which we run the Vusers. The configuration of any client machine includes its hardware settings, memory, operating system, software applications, development tools, etc. This system component configuration should match with the overall system configuration that would include the network infrastructure, the web server, the database server, and any other components that go with this larger system so as to achieve the load testing objectives.
28. How do you identify the performance bottlenecks? - Performance Bottlenecks can be detected by using monitors. These monitors might be application server monitors, web server monitors, database server monitors and network monitors. They help in finding out the troubled area in our scenario which causes increased response time. The measurements made are usually performance response time, throughput, hits/sec, network delay graphs, etc.
29. If web server, database and Network are all fine where could be the problem? - The problem could be in the system itself or in the application server or in the code written for the application.
30. How did you find web server related issues? - Using Web resource monitors we can find the performance of web servers. Using these monitors we can analyze throughput on the web server, number of hits per second thatoccurred during scenario, the number of http responses per second, the number of downloaded pages per second.
31. How did you find database related issues? - By running “Database” monitor and help of “Data Resource Graph” we can find database related issues. E.g. You can specify the resource you want to measure on before running the controller and than you can see database related issues
32. Explain all the web recording options?
33. What is the difference between Overlay graph and Correlate graph? - Overlay Graph: It overlay the content of two graphs that shares a common x-axis. Left Y-axis on the merged graph show’s the current graph’s value & Right Y-axis show the value of Y-axis of the graph that was merged. Correlate Graph: Plot the Y-axis of two graphs against each other. The active graph’s Y-axis becomes X-axis of merged graph. Y-axis of the graph that was merged becomes merged graph’s Y-axis.
34. How did you plan the Load? What are the Criteria? - Load test is planned to decide the number of users, what kind of machines we are going to use and from where they are run. It is based on 2 important documents, Task Distribution Diagram and Transaction profile. Task Distribution Diagram gives us the information on number of users for a particular transaction and the time of the load. The peak usage and off-usage are decided from this Diagram. Transaction profile gives us the information about the transactions name and their priority levels with regard to the scenario we are deciding.
35. What does vuser_init action contain? - Vuser_init action contains procedures to login to a server.
36. What does vuser_end action contain? - Vuser_end section contains log off procedures.
37. What is think time? How do you change the threshold? - Think time is the time that a real user waits between actions. Example: When a user receives data from a server, the user may wait several seconds to review the data before responding. This delay is known as the think time. Changing the Threshold: Threshold level is the level below which the recorded think time will be ignored. The default value is five (5) seconds. We can change the think time threshold in the Recording options of the Vugen.
38. What is the difference between standard log and extended log? - The standard log sends a subset of functions and messages sent during script execution to a log. The subset depends on the Vuser type Extended log sends a detailed script execution messages to the output log. This is mainly used during debugging when we want information about: Parameter substitution. Data returned by the server. Advanced trace.
39. Explain the following functions: - lr_debug_message - The lr_debug_message function sends a debug message to the output log when the specified message class is set. lr_output_message - The lr_output_message function sends notifications to the Controller Output window and the Vuser log file. lr_error_message - The lr_error_message function sends an error message to the LoadRunner Output window. lrd_stmt - The lrd_stmt function associates a character string (usually a SQL statement) with a cursor. This function sets a SQL statement to be processed. lrd_fetch - The lrd_fetch function fetches the next row from the result set.
40. Throughput - If the throughput scales upward as time progresses and the number of Vusers increase, this indicates that the bandwidth is sufficient. If the graph were to remain relatively flat as the number of Vusers increased, it wouldbe reasonable to conclude that the bandwidth is constraining the volume ofdata delivered.
41. Types of Goals in Goal-Oriented Scenario - Load Runner provides you with five different types of goals in a goal oriented scenario:
o The number of concurrent Vusers
o The number of hits per second
o The number of transactions per second
o The number of pages per minute
o The transaction response time that you want your scenario
42. Analysis Scenario (Bottlenecks): In Running Vuser graph correlated with the response time graph you can see that as the number of Vusers increases, the average response time of the check itinerary transaction very gradually increases. In other words, the average response time steadily increases as the loadincreases. At 56 Vusers, there is a sudden, sharp increase in the average responsetime. We say that the test broke the server. That is the mean time before failure (MTBF). The response time clearly began to degrade when there were more than 56 Vusers running simultaneously.
43. What is correlation? Explain the difference between automatic correlation and manual correlation? - Correlation is used to obtain data which are unique for each run of the script and which are generated by nested queries. Correlation provides the value to avoid errors arising out of duplicate values and also optimizing the code (to avoid nested queries). Automatic correlation is where we set some rules for correlation. It can be application server specific. Here values are replaced by data which are created by these rules. In manual correlation, the value we want to correlate is scanned and create correlation is used to correlate.
44. Where do you set automatic correlation options? - Automatic correlation from web point of view, can be set in recording options and correlation tab. Here we can enable correlation for the entire script and choose either issue online messages or offline actions, where we can define rules for that correlation. Automatic correlation for database, can be done using show output window and scan for correlation and picking the correlate query tab and choose which query value we want to correlate. If we know the specific value to be correlated, we just do create correlation for the value and specify how the value to be created.
45. What is a function to capture dynamic values in the web user script? – Web reg_save_param function saves dynamic data information to a parameter
Questions on Win Runner
How you used Win Runner in your project? - Yes, I have been using Win Runner for creating automated scripts for GUI, functional and regression testing of the AUT.
Explain Win Runner testing process? - Win Runner testing process involves six main stages
Create GUI Map File so that Win Runner can recognize the GUI objects in the application being tested
Create test scripts by recording, programming, or a combination of both. While recording tests, insert checkpoints where you want to check the response of the application being tested.
Debug Test: run tests in Debug mode to make sure they run smoothly
Run Tests: run tests in Verify mode to test your application.
View Results: determines the success or failure of the tests.
Report Defects: If a test run fails due to a defect in the application being tested, you can report information about the defect directly from the Test Results window.
What is contained in the GUI map? - Win Runner stores information it learns about a window or object in a GUI Map. When Win Runner runs a test, it uses the GUI map to locate objects. It reads an object’s description in the GUI map and then looks for an object with the same properties in the application being tested. Each of these objects in the GUI Map file will be having a logical name and a physical description. There are 2 types of GUI Map files. Global GUI Map file: a single GUI Map file for the entire application. GUI Map File per Test: Win Runner automatically creates a GUI Map file for each test created.
How does Win Runner recognize objects on the application? - Win Runner uses the GUI Map file to recognize objects on the application. When Win Runner runs a test, it uses the GUI map to locate objects. It reads an object’s description in the GUI map and then looks for an object with the same properties in the application being tested.
Have you created test scripts and what is contained in the test scripts? - Yes I have created test scripts. It contains the statement in Mercury Interactive’s Test Script Language (TSL). These statements appear as a test script in a test window. You can then enhance your recorded test script, either by typing in additional TSL functions and programming elements or by using WinRunner’s visual programming tool, the Function Generator.
How does WinRunner evaluate test results? - Following each test run, Win Runner displays the results in a report. The report details all the major events that occurred during the run, such as checkpoints, error messages, system messages, or user messages. If mismatches are detected at checkpoints during the test run, you can view the expected results and the actual results from the Test Results window.
Have you performed debugging of the scripts? - Yes, I have performed debugging of scripts. We can debug the script by executing the script in the debug mode. We can also debug script using the Step, Step Into, Step out functionalities provided by the Win Runner.
How do you run your test scripts? - We run tests in Verify mode to test your application. Each time WinRunner encounters a checkpoint in the test script, it compares the current data of the application being tested to the expected data captured earlier. If any mismatches are found, WinRunner captures them as actual results.
How do you analyze results and report the defects? - Following each test run, Win Runner displays the results in a report. The report details all the major events that occurred during the run, such as checkpoints, error messages, system messages, or user messages. If mismatches are detected at checkpoints during the test run, you can view the expected results and the actual results from the Test Results window. If a test run fails due to a defect in the application being tested, you can report information about the defect directly from the Test Results window. This information is sent via e-mail to the quality assurance manager, who tracks the defect until it is fixed.
What is the use of Test Director software? - Test Director is Mercury Interactive’s software test management tool. It helps quality assurance personnel plan and organize the testing process. With TestDirector you can create a database of manual and automated tests, build test cycles, run tests, and report and track defects. You can also create reports and graphs to help review the progress of planning tests, running tests, and tracking defects before a software release.
Have you integrated your automated scripts from Test Director? - When you work with Win Runner, you can choose to save your tests directly to your Test Director database or while creating a test case in the Test Director we can specify whether the script in automated or manual. And if it is automated script then Test Director will build a skeleton for the script that can be later modified into one which could be used to test the AUT.
What are the different modes of recording? - There are two type of recording in Win Runner. Context Sensitive recording records the operations you perform on your application by identifying Graphical User Interface (GUI) objects. Analog recording records keyboard input, mouse clicks, and the precise x- and y-coordinates traveled by the mouse pointer across the screen.
What is the purpose of loading Win Runner Add-Ins? - Add-Ins are used in Win Runner to load functions specific to the particular add-in to the memory. While creating a script only those functions in the add-in selected will be listed in the function generator and while executing the script only those functions in the loaded add-in will be executed else Win Runner will give an error message saying it does not recognize the function.
What are the reasons that Win Runner fails to identify an object on the GUI? - Win Runner fails to identify an object in a GUI due to various reasons. The object is not a standard windows object. If the browser used is not compatible with the Win Runner version, GUI Map Editor will not be able to learn any of the objects displayed in the browser window.
What is meant by the logical name of the object? - An object’s logical name is determined by its class. In most cases, the logical name is the label that appears on an object.
If the object does not have a name then what will be the logical name? - If the object does not have a name then the logical name could be the attached text.
What is the different between GUI map and GUI map files? - The GUI map is actually the sum of one or more GUI map files. There are two modes for organizing GUI map files. Global GUI Map file: a single GUI Map file for the entire application. GUI Map File per Test: Win Runner automatically creates a GUI Map file for each test created.
GUI Map file is a file which contains the windows and the objects learned by the Win Runner with its logical name and their physical description.
How do you view the contents of the GUI map? - GUI Map editor displays the content of a GUI Map. We can invoke GUI Map Editor from the Tools Menu in WinRunner. The GUI Map Editor displays the various GUI Map files created and the windows and objects learned in to them with their logical name and physical description.
When you create GUI map do you record all the objects of specific objects? - If we are learning a window then WinRunner automatically learns all the objects in the window else we will we identifying those object, which are to be learned in a window, since we will be working with only those objects while creating scripts.
How to handle the exceptions using recovery scenario manager in QTP? - There are 4 trigger events during which a recovery scenario should be activated. A pop up window appears in an opened application during the test run: A property of an object changes its state or value, A step in the test does not run successfully, An open application fails during the test run, These triggers are considered as exceptions. You can instruct QTP to recover unexpected events or errors that occurred in your testing environment during test run. Recovery scenario manager provides a wizard that guides you through the defining recovery scenario. Recovery scenario has three steps: 1. Triggered Events 2. Recovery steps 3. Post Recovery Test-Run
What is the use of Text output value in QTP? - Output values enable to view the values that the application talks during run time. When parameterized, the values change for each iteration. Thus by creating output values, we can capture the values that the application takes for each run and output them to the data table.
How to use the Object spy in QTP 8.0 version? - There are two ways to Spy the objects in QTP: 1) Thru file toolbar, In the File Toolbar click on the last toolbar button (an icon showing a person with hat). 2) True Object repository Dialog, In Object repository dialog click on the button object spy. In the Object spy Dialog click on the button showing hand symbol. The pointer now changes in to a hand symbol and we have to point out the object to spy the state of the object if at all the object is not visible. or window is minimized then, hold the Ctrl button and activate the required window to and release the Ctrl button.
How Does Run time data (Parameterization) is handled in QTP? - You can then enter test data into the Data Table, an integrated spreadsheet with the full functionality of Excel, to manipulate data sets and create multiple test iterations, without programming, to expand test case coverage. Data can be typed in or imported from databases, spreadsheets, or text files.
What is keyword view and Expert view in QTP? - Quick Test’s Keyword Driven approach, test automation experts have full access to the underlying test and object properties, via an integrated scripting and debugging environment that is round-trip synchronized with the Keyword View. Advanced testers can view and edit their tests in the Expert View, which reveals the underlying industry-standard VBScript that Quick Test Professional automatically generates. Any changes made in the Expert View are automatically synchronized with the Keyword View.
Explain about the Test Fusion Report of QTP? - Once a tester has run a test, a Test Fusion report displays all aspects of the test run: a high-level results overview, an expandable Tree View of the test specifying exactly where application failures occurred, the test data used, application screen shots for every step that highlight any discrepancies, and detailed explanations of each checkpoint pass and failure. By combining Test Fusion reports with Quick Test Professional, you can share reports across an entire QA and development team.
Which environments does QTP support? - Quick Test Professional supports functional testing of all enterprise environments, including Windows, Web,..NET, Java/J2EE, SAP, Siebel, Oracle, PeopleSoft, Visual Basic, ActiveX, mainframe terminal emulators, and Web services.
What is QTP? - Quick Test is a graphical interface record-playback automation tool. It is able to work with any web, java or windows client application. Quick Test enables you to test standard web objects and ActiveX controls. In addition to these environments, Quick Test Professional also enables you to test Java applets and applications and multimedia objects on Applications as well as standard Windows applications, Visual Basic 6 applications and.NET framework applications
Explain QTP Testing process? - Quick Test testing process consists of 6 main phases:
Create your test plan - Prior to automating there should be a detailed description of the test including the exact steps to follow, data to be input, and all items to be verified by the test. The verification information should include both data validations and existence or state verifications of objects in the application.
Recording a session on your application - As you navigate through your application, Quick Test graphically displays each step you perform in the form of a collapsible icon-based test tree. A step is any user action that causes or makes a change in your site, such as clicking a link or image, or entering data in a form.
Enhancing your test - Inserting checkpoints into your test lets you search for a specific value of a page, object or text string, which helps you identify whether or not your application is functioning correctly. NOTE: Checkpoints can be added to a test as you record it or after the fact via the Active Screen. It is much easier and faster to add the checkpoints during the recording process. Broadening the scope of your test by replacing fixed values with parameters lets you check how your application performs the same operations with multiple sets of data. Adding logic and conditional statements to your test enables you to add sophisticated checks to your test.
Debugging your test - If changes were made to the script, you need to debug it to check that it operates smoothly and without interruption.
Running your test on a new version of your application - You run a test to check the behavior of your application. While running, Quick Test connects to your application and performs each step in your test.
Analyzing the test results - You examine the test results to pinpoint defects in your application.
Reporting defects - As you encounter failures in the application when analyzing test results, you will create defect reports in Defect Reporting Tool.
Explain the QTP Tool interface. - It contains the following key elements: Title bar, displaying the name of the currently open test, Menu bar, displaying menus of Quick Test commands, File toolbar, containing buttons to assist you in managing tests, Test toolbar, containing buttons used while creating and maintaining tests, Debug toolbar, containing buttons used while debugging tests. Note: The Debug toolbar is not displayed when you open Quick Test for the first time. You can display the Debug toolbar by choosing View — Toolbars — Debug. Action toolbar, containing buttons and a list of actions, enabling you to view the details of an individual action or the entire test flow. Note: The Action toolbar is not displayed when you open Quick Test for the first time. You can display the Action toolbar by choosing View — Toolbars — Action. If you insert a reusable or external action in a test, the Action toolbar is displayed automatically. Test pane, containing two tabs to view your test-the Tree View and the Expert View ,Test Details pane, containing the Active Screen. Data Table, containing two tabs, Global and Action, to assist you in parameterizing your test. Debug Viewer pane, containing three tabs to assist you in debugging your test-Watch Expressions, Variables, and Command. (The Debug Viewer pane can be opened only when a test run pauses at a breakpoint.) Status bar, displaying the status of the test
How does QTP recognize Objects in AUT? - Quick Test stores the definitions for application objects in a file called the Object Repository. As you record your test, Quick Test will add an entry for each item you interact with. Each Object Repository entry will be identified by a logical name (determined automatically by Quick Test), and will contain a set of properties (type, name, etc) that uniquely identify each object. Each line in the Quick Test script will contain a reference to the object that you interacted with, a call to the appropriate method (set, click, check) and any parameters for that method (such as the value for a call to the set method). The references to objects in the script will all be identified by the logical name, rather than any physical, descriptive properties.
What are the types of Object Repositories in QTP? - Quick Test has two types of object repositories for storing object information: shared object repositories and action object repositories. You can choose which type of object repository you want to use as the default type for new tests, and you can change the default as necessary for each new test. The object repository per-action mode is the default setting. In this mode, Quick Test automatically creates an object repository file for each action in your test so that you can create and run tests without creating, choosing, or modifying object repository files. However, if you do modify values in an action object repository, your changes do not have any effect on other actions. Therefore, if the same test object exists in more than one action and you modify an object’s property values in one action, you may need to make the same change in every action (and any test) containing the object.
Explain the check points in QTP? - A checkpoint verifies that expected information is displayed in an Application while the test is running. You can add eight types of checkpoints to your test for standard web objects using QTP. A page checkpoint checks the characteristics of an Application. A text checkpoint checks that a text string is displayed in the appropriate place on an Application. An object checkpoint (Standard) checks the values of an object on an Application. An image checkpoint checks the values of an image on an Application. A table checkpoint checks information within a table on a Application. An Accessibility checkpoint checks the web page for Section 508 compliance. An XML checkpoint checks the contents of individual XML data files or XML documents that are part of your Web application. A database checkpoint checks the contents of databases accessed by your web site
In how many ways we can add check points to an application using QTP? - We can add checkpoints while recording the application or we can add after recording is completed using Active screen (Note: To perform the second one The Active screen must be enabled while recording).
How does QTP identify objects in the application? - QTP identifies the object in the application by Logical Name and Class.
What is Parameterizing Tests? - When you test your application, you may want to check how it performs the same operations with multiple sets of data. For example, suppose you want to check how your application responds to ten separate sets of data. You could record ten separate tests, each with its own set of data. Alternatively, you can create a parameterized test that runs ten times: each time the test runs, it uses a different set of data.
What is test object model in QTP? - The test object model is a large set of object types or classes that Quick Test uses to represent the objects in your application. Each test object class has a list of properties that can uniquely identify objects of that class and a set of relevant methods that Quick Test can record for it. A test object is an object that Quick Test creates in the test or component to represent the actual object in your application. Quick Test stores information about the object that will help it identify and check the object during the run session.
What is Object Spy in QTP? - Using the Object Spy, you can view the properties of any object in an open application. You use the Object Spy pointer to point to an object. The Object Spy displays the selected object’s hierarchy tree and its properties and values in the Properties tab of the Object Spy dialog box.
What is the Diff between Image check-point and Bit map Check point? - Image checkpoints enable you to check the properties of a Web image. You can check an area of a Web page or application as a bitmap. While creating a test or component, you specify the area you want to check by selecting an object. You can check an entire object or any area within an object. Quick Test captures the specified object as a bitmap, and inserts a checkpoint in the test or component. You can also choose to save only the selected area of the object with your test or component in order to save disk Space. For example, suppose you have a Web site that can display a map of a city the user specifies. The map has control keys for zooming. You can record the new map that is displayed after one click on the control key that zooms in the map. Using the bitmap checkpoint, you can check that the map zooms in correctly. You can create bitmap checkpoints for all supported testing environments (as long as the appropriate add-ins are loaded). Note: The results of bitmap checkpoints may be affected by factors such as operating system, screen resolution, and color settings.
How many ways we can parameterize data in QTP? - There are four types of parameters: Test, action or component parameters enable you to use values passed from your test or component, or values from other actions in your test. Data Table parameters enable you to create a data-driven test (or action) that runs several times using the data you supply. In each repetition, or iteration, Quick Test uses a different value from the Data Table. Environment variable parameters enable you to use variable values from other sources during the run session. These may be values you supply, or values that Quick Test generates for you based on conditions and options you choose. Random number parameters enable you to insert random numbers as values in your test or component. For example, to check how your application handles small and large ticket orders, you can have Quick Test generate a random number and insert it in a number of tickets edit field.
How do u do batch testing in WR & is it possible to do in QTP, if so explain? - Batch Testing in WR is nothing but running the whole test set by selecting Run Test set from the Execution Grid. The same is possible with QTP also. If our test cases are automated then by selecting Run Test set all the test scripts can be executed. In this process the Scripts get executed one by one by keeping all the remaining scripts in Waiting mode.
If I give some thousand tests to execute in 2 days what do u do? – Ad-hoc testing is done. It Covers the least basic functionalities to verify that the system is working fine.
What does it mean when a check point is in red color? what do u do? - A red color indicates failure. Here we analyze the cause for failure whether it is a Script Issue or Environment Issue or a Application issue.
What is Object Spy in QTP? - Using the Object Spy, you can view the properties of any object in an open application. You use the Object Spy pointer to point to an object. The Object Spy displays the selected object’s hierarchy tree and its properties and values in the Properties tab of the Object Spy dialog box.
What is the file extension of the code file & object repository file in QTP? - Code file extension is.vbs and object repository is.tsr
Explain the concept of object repository & how QTP recognizes objects? - Object Repository: displays a tree of all objects in the current component or in the current action or entire test (depending on the object repository mode you selected). We can view or modify the test object description of any test object in the repository or to add new objects to the repository. Quicktest learns the default property values and determines in which test object class it fits. If it is not enough it adds assistive properties, one by one to the description until it has compiled the unique description. If no assistive properties are available, then it adds a special Ordinal identifier such as objects location on the page or in the source code.
What are the properties you would use for identifying a browser & page when using descriptive programming? - Name would be another property apart from title that we can use.
Give me an example where you have used a COM interface in your QTP project? - Com interface appears in the scenario of front end and back end. for eg:if you r using oracle as back end and front end as VB or any language then for better compatibility we will go for an interface. of which COM will be one among those interfaces. Create object creates handle to the instance of the specified object so that we program can use the methods on the specified object. It is used for implementing Automation(as defined by Microsoft).
Explain in brief about the QTP Automation Object Model. - Essentially all configuration and run functionality provided via the Quick Test interface is in some way represented in the Quick Test automation object model via objects, methods, and properties. Although a one-on-one comparison cannot always be made, most dialog boxes in Quick Test have a corresponding automation object, most options in dialog boxes can be set and/or retrieved using the corresponding object property, and most menu commands and other operations have corresponding automation methods. You can use the objects, methods, and properties exposed by the Quick Test automation object model, along with standard programming elements such as loops and conditional statements to design your program.
Load Runner Questions
1. What is load testing? - Load testing is to test that if the application works fine with the loads that result from large number of simultaneous users, transactions and to determine weather it can handle peak usage periods.
2. What is Performance testing? - Timing for both read and update transactions should be gathered to determine whether system functions are being performed in an acceptable timeframe. This should be done standalone and then in a multi user environment to determine the effect of multiple transactions on the timing of a single transaction.
3. Did u use LoadRunner? What version? - Yes. Version 7.2.
4. Explain the Load testing process? -Step 1: Planning the test. Here, we develop a clearly defined test plan to ensure the test scenarios we develop will accomplish load-testing objectives. Step 2: Creating Vusers. Here, we create Vuser scripts that contain tasks performed by each Vuser, tasks performed by Vusers as a whole, and tasks measured as transactions. Step 3: Creating the scenario. A scenario describes the events that occur during a testing session. It includes a list of machines, scripts, and Vusers that run during the scenario. We create scenarios using LoadRunner Controller. We can create manual scenarios as well as goal-oriented scenarios. In manual scenarios, we define the number of Vusers, the load generator machines, and percentage of Vusers to be assigned to each script. For web tests, we may create a goal-oriented scenario where we define the goal that our test has to achieve. LoadRunner automatically builds a scenario for us. Step 4: Running the scenario.We emulate load on the server by instructing multiple Vusers to perform tasks simultaneously. Before the testing, we set the scenario configuration and scheduling. We can run the entire scenario, Vuser groups, or individual Vusers. Step 5: Monitoring the scenario.We monitor scenario execution using the LoadRunner online runtime, transaction, system resource, Web resource, Web server resource, Web application server resource, database server resource, network delay, streaming media resource, firewall server resource, ERP server resource, and Java performance monitors. Step 6: Analyzing test results. During scenario execution, LoadRunner records the performance of the application under different loads. We use LoadRunner’s graphs and reports to analyze the application’s performance.
5. When do you do load and performance Testing? - We perform load testing once we are done with interface (GUI) testing. Modern system architectures are large and complex. Whereas single user testing primarily on functionality and user interface of a system component, application testing focuses on performance and reliability of an entire system. For example, a typical application-testing scenario might depict 1000 users logging in simultaneously to a system. This gives rise to issues such as what is the response time of the system, does it crash, will it go with different software applications and platforms, can it hold so many hundreds and thousands of users, etc. This is when we set do load and performance testing.
6. What are the components of LoadRunner? - The components of LoadRunner are The Virtual User Generator, Controller, and the Agent process, LoadRunner Analysis and Monitoring, LoadRunner Books Online.
7. What Component of LoadRunner would you use to record a Script? - The Virtual User Generator (VuGen) component is used to record a script. It enables you to develop Vuser scripts for a variety of application types and communication protocols.
8. What Component of LoadRunner would you use to play Back the script in multi user mode? - The Controller component is used to playback the script in multi-user mode. This is done during a scenario run where a vuser script is executed by a number of vusers in a group.
9. What is a rendezvous point? - You insert rendezvous points into Vuser scripts to emulate heavy user load on the server. Rendezvous points instruct Vusers to wait during test execution for multiple Vusers to arrive at a certain point, in order that they may simultaneously perform a task. For example, to emulate peak load on the bank server, you can insert a rendezvous point instructing 100 Vusers to deposit cash into their accounts at the same time.
10. What is a scenario? - A scenario defines the events that occur during each testing session. For example, a scenario defines and controls the number of users to emulate, the actions to be performed, and the machines on which the virtual users run their emulations.
11. Explain the recording mode for web Vuser script? - We use VuGen to develop a Vuser script by recording a user performing typical business processes on a client application. VuGen creates the script by recording the activity between the client and the server. For example, in web based applications, VuGen monitors the client end of the database and traces all the requests sent to, and received from, the database server. We use VuGen to: Monitor the communication between the application and the server; Generate the required function calls; and Insert the generated function calls into a Vuser script.
12. Why do you create parameters? - Parameters are like script variables. They are used to vary input to the server and to emulate real users. Different sets of data are sent to the server each time the script is run. Better simulate the usage model for more accurate testing from the Controller; one script can emulate many different users on the system.
13. What is correlation? Explain the difference between automatic correlation and manual correlation? - Correlation is used to obtain data which are unique for each run of the script and which are generated by nested queries. Correlation provides the value to avoid errors arising out of duplicate values and also optimizing the code (to avoid nested queries). Automatic correlation is where we set some rules for correlation. It can be application server specific. Here values are replaced by data which are created by these rules. In manual correlation, the value we want to correlate is scanned and create correlation is used to correlate.
14. How do you find out where correlation is required? Give few examples from your projects? - Two ways: First we can scan for correlations, and see the list of values which can be correlated. From this we can pick a value to be correlated. Secondly, we can record two scripts and compare them. We can look up the difference file to see for the values which needed to be correlated. In my project, there was a unique id developed for each customer, it was nothing but Insurance Number, it was generated automatically and it was sequential and this value was unique. I had to correlate this value, in order to avoid errors while running my script. I did using scan for correlation.
15. Where do you set automatic correlation options? - Automatic correlation from web point of view can be set in recording options and correlation tab. Here we can enable correlation for the entire script and choose either issue online messages or offline actions, where we can define rules for that correlation. Automatic correlation for database can be done using show output window and scan for correlation and picking the correlate query tab and choose which query value we want to correlate. If we know the specific value to be correlated, we just do create correlation for the value and specify how the value to be created.
16. What is a function to capture dynamic values in the web Vuser script? - Web_reg_save_param function saves dynamic data information to a parameter.
17. When do you disable log in Virtual User Generator, When do you choose standard and extended logs? - Once we debug our script and verify that it is functional, we can enable logging for errors only. When we add a script to a scenario, logging is automatically disabled. Standard Log Option: When you selectStandard log, it creates a standard log of functions and messages sent during script execution to use for debugging. Disable this option for large load testing scenarios. When you copy a script to a scenario, logging is automatically disabled Extended Log Option: Selectextended log to create an extended log, including warnings and other messages. Disable this option for large load testing scenarios. When you copy a script to a scenario, logging is automatically disabled. We can specify which additional information should be added to the extended log using the Extended log options.
18. How do you debug a LoadRunner script? - VuGen contains two options to help debug Vuser scripts-the Run Step by Step command and breakpoints. The Debug settings in the Options dialog box allow us to determine the extent of the trace to be performed during scenario execution. The debug information is written to the Output window. We can manually set the message class within your script using the lr_set_debug_message function. This is useful if we want to receive debug information about a small section of the script only.
19. How do you write user defined functions in LR? Give me few functions you wrote in your previous project? - Before we create the User Defined functions we need to create the externallibrary (DLL) with the function. We add this library to VuGen bin directory. Once the library is added then we assign user defined function as a parameter. The function should have the following format: __declspec (dllexport) char*
20. What are the changes you can make in run-time settings? - The Run Time Settings that we make are: a) Pacing - It has iteration count. b) Log - Under this we have Disable Logging Standard Log and c) Extended Think Time - In think time we have two options like Ignore think time and Replay think time. d) General - Under general tab we can set the vusers as process or as multithreading and whether each step as a transaction.
21. Where do you set Iteration for Vuser testing? - We set Iterations in the Run Time Settings of the VuGen. The navigation for this is Run time settings, Pacing tab, set number of iterations.
22. How do you perform functional testing under load? - Functionality under load can be tested by running several Vusers concurrently. By increasing the amount of Vusers, we can determine how much load the server can sustain.
23. What is Ramp up? How do you set this? - This option is used to gradually increase the amount of Vusers/load on the server. An initial value is set and a value to wait between intervals can bespecified. To set Ramp Up, go to ‘Scenario Scheduling Options’
24. What is the advantage of running the Vuser as thread? - VuGen provides the facility to use multithreading. This enables more Vusers to be run pergenerator. If the Vuser is run as a process, the same driver program is loaded into memory for each Vuser, thus taking up a large amount of memory. This limits the number of Vusers that can be run on a singlegenerator. If the Vuser is run as a thread, only one instance of the driver program is loaded into memory for the given number ofVusers (say 100). Each thread shares the memory of the parent driver program, thus enabling more Vusers to be run per generator.
25. If you want to stop the execution of your script on error, how do you do that? - The lr_abort function aborts the execution of a Vuser script. It instructs the Vuser to stop executing the Actions section, execute the vuser_end section and end the execution. This function is useful when you need to manually abort a script execution as a result of a specific error condition. When you end a script using this function, the Vuser is assigned the status "Stopped". For this to take effect, we have to first uncheck the “Continue on error” option in Run-Time Settings.
26. What is the relation between Response Time and Throughput? - The Throughput graph shows the amount of data in bytes that the Vusers received from the server in a second. When we compare this with the transaction response time, we will notice that as throughput decreased, the response time also decreased. Similarly, the peak throughput and highest response time would occur approximately at the same time.
27. Explain the Configuration of your systems? - The configuration of our systems refers to that of the client machines on which we run the Vusers. The configuration of any client machine includes its hardware settings, memory, operating system, software applications, development tools, etc. This system component configuration should match with the overall system configuration that would include the network infrastructure, the web server, the database server, and any other components that go with this larger system so as to achieve the load testing objectives.
28. How do you identify the performance bottlenecks? - Performance Bottlenecks can be detected by using monitors. These monitors might be application server monitors, web server monitors, database server monitors and network monitors. They help in finding out the troubled area in our scenario which causes increased response time. The measurements made are usually performance response time, throughput, hits/sec, network delay graphs, etc.
29. If web server, database and Network are all fine where could be the problem? - The problem could be in the system itself or in the application server or in the code written for the application.
30. How did you find web server related issues? - Using Web resource monitors we can find the performance of web servers. Using these monitors we can analyze throughput on the web server, number of hits per second thatoccurred during scenario, the number of http responses per second, the number of downloaded pages per second.
31. How did you find database related issues? - By running “Database” monitor and help of “Data Resource Graph” we can find database related issues. E.g. You can specify the resource you want to measure on before running the controller and than you can see database related issues
32. Explain all the web recording options?
33. What is the difference between Overlay graph and Correlate graph? - Overlay Graph: It overlay the content of two graphs that shares a common x-axis. Left Y-axis on the merged graph show’s the current graph’s value & Right Y-axis show the value of Y-axis of the graph that was merged. Correlate Graph: Plot the Y-axis of two graphs against each other. The active graph’s Y-axis becomes X-axis of merged graph. Y-axis of the graph that was merged becomes merged graph’s Y-axis.
34. How did you plan the Load? What are the Criteria? - Load test is planned to decide the number of users, what kind of machines we are going to use and from where they are run. It is based on 2 important documents, Task Distribution Diagram and Transaction profile. Task Distribution Diagram gives us the information on number of users for a particular transaction and the time of the load. The peak usage and off-usage are decided from this Diagram. Transaction profile gives us the information about the transactions name and their priority levels with regard to the scenario we are deciding.
35. What does vuser_init action contain? - Vuser_init action contains procedures to login to a server.
36. What does vuser_end action contain? - Vuser_end section contains log off procedures.
37. What is think time? How do you change the threshold? - Think time is the time that a real user waits between actions. Example: When a user receives data from a server, the user may wait several seconds to review the data before responding. This delay is known as the think time. Changing the Threshold: Threshold level is the level below which the recorded think time will be ignored. The default value is five (5) seconds. We can change the think time threshold in the Recording options of the Vugen.
38. What is the difference between standard log and extended log? - The standard log sends a subset of functions and messages sent during script execution to a log. The subset depends on the Vuser type Extended log sends a detailed script execution messages to the output log. This is mainly used during debugging when we want information about: Parameter substitution. Data returned by the server. Advanced trace.
39. Explain the following functions: - lr_debug_message - The lr_debug_message function sends a debug message to the output log when the specified message class is set. lr_output_message - The lr_output_message function sends notifications to the Controller Output window and the Vuser log file. lr_error_message - The lr_error_message function sends an error message to the LoadRunner Output window. lrd_stmt - The lrd_stmt function associates a character string (usually a SQL statement) with a cursor. This function sets a SQL statement to be processed. lrd_fetch - The lrd_fetch function fetches the next row from the result set.
40. Throughput - If the throughput scales upward as time progresses and the number of Vusers increase, this indicates that the bandwidth is sufficient. If the graph were to remain relatively flat as the number of Vusers increased, it wouldbe reasonable to conclude that the bandwidth is constraining the volume ofdata delivered.
41. Types of Goals in Goal-Oriented Scenario - Load Runner provides you with five different types of goals in a goal oriented scenario:
o The number of concurrent Vusers
o The number of hits per second
o The number of transactions per second
o The number of pages per minute
o The transaction response time that you want your scenario
42. Analysis Scenario (Bottlenecks): In Running Vuser graph correlated with the response time graph you can see that as the number of Vusers increases, the average response time of the check itinerary transaction very gradually increases. In other words, the average response time steadily increases as the loadincreases. At 56 Vusers, there is a sudden, sharp increase in the average responsetime. We say that the test broke the server. That is the mean time before failure (MTBF). The response time clearly began to degrade when there were more than 56 Vusers running simultaneously.
43. What is correlation? Explain the difference between automatic correlation and manual correlation? - Correlation is used to obtain data which are unique for each run of the script and which are generated by nested queries. Correlation provides the value to avoid errors arising out of duplicate values and also optimizing the code (to avoid nested queries). Automatic correlation is where we set some rules for correlation. It can be application server specific. Here values are replaced by data which are created by these rules. In manual correlation, the value we want to correlate is scanned and create correlation is used to correlate.
44. Where do you set automatic correlation options? - Automatic correlation from web point of view, can be set in recording options and correlation tab. Here we can enable correlation for the entire script and choose either issue online messages or offline actions, where we can define rules for that correlation. Automatic correlation for database, can be done using show output window and scan for correlation and picking the correlate query tab and choose which query value we want to correlate. If we know the specific value to be correlated, we just do create correlation for the value and specify how the value to be created.
45. What is a function to capture dynamic values in the web user script? – Web reg_save_param function saves dynamic data information to a parameter
Questions on Win Runner
How you used Win Runner in your project? - Yes, I have been using Win Runner for creating automated scripts for GUI, functional and regression testing of the AUT.
Explain Win Runner testing process? - Win Runner testing process involves six main stages
Create GUI Map File so that Win Runner can recognize the GUI objects in the application being tested
Create test scripts by recording, programming, or a combination of both. While recording tests, insert checkpoints where you want to check the response of the application being tested.
Debug Test: run tests in Debug mode to make sure they run smoothly
Run Tests: run tests in Verify mode to test your application.
View Results: determines the success or failure of the tests.
Report Defects: If a test run fails due to a defect in the application being tested, you can report information about the defect directly from the Test Results window.
What is contained in the GUI map? - Win Runner stores information it learns about a window or object in a GUI Map. When Win Runner runs a test, it uses the GUI map to locate objects. It reads an object’s description in the GUI map and then looks for an object with the same properties in the application being tested. Each of these objects in the GUI Map file will be having a logical name and a physical description. There are 2 types of GUI Map files. Global GUI Map file: a single GUI Map file for the entire application. GUI Map File per Test: Win Runner automatically creates a GUI Map file for each test created.
How does Win Runner recognize objects on the application? - Win Runner uses the GUI Map file to recognize objects on the application. When Win Runner runs a test, it uses the GUI map to locate objects. It reads an object’s description in the GUI map and then looks for an object with the same properties in the application being tested.
Have you created test scripts and what is contained in the test scripts? - Yes I have created test scripts. It contains the statement in Mercury Interactive’s Test Script Language (TSL). These statements appear as a test script in a test window. You can then enhance your recorded test script, either by typing in additional TSL functions and programming elements or by using WinRunner’s visual programming tool, the Function Generator.
How does WinRunner evaluate test results? - Following each test run, Win Runner displays the results in a report. The report details all the major events that occurred during the run, such as checkpoints, error messages, system messages, or user messages. If mismatches are detected at checkpoints during the test run, you can view the expected results and the actual results from the Test Results window.
Have you performed debugging of the scripts? - Yes, I have performed debugging of scripts. We can debug the script by executing the script in the debug mode. We can also debug script using the Step, Step Into, Step out functionalities provided by the Win Runner.
How do you run your test scripts? - We run tests in Verify mode to test your application. Each time WinRunner encounters a checkpoint in the test script, it compares the current data of the application being tested to the expected data captured earlier. If any mismatches are found, WinRunner captures them as actual results.
How do you analyze results and report the defects? - Following each test run, Win Runner displays the results in a report. The report details all the major events that occurred during the run, such as checkpoints, error messages, system messages, or user messages. If mismatches are detected at checkpoints during the test run, you can view the expected results and the actual results from the Test Results window. If a test run fails due to a defect in the application being tested, you can report information about the defect directly from the Test Results window. This information is sent via e-mail to the quality assurance manager, who tracks the defect until it is fixed.
What is the use of Test Director software? - Test Director is Mercury Interactive’s software test management tool. It helps quality assurance personnel plan and organize the testing process. With TestDirector you can create a database of manual and automated tests, build test cycles, run tests, and report and track defects. You can also create reports and graphs to help review the progress of planning tests, running tests, and tracking defects before a software release.
Have you integrated your automated scripts from Test Director? - When you work with Win Runner, you can choose to save your tests directly to your Test Director database or while creating a test case in the Test Director we can specify whether the script in automated or manual. And if it is automated script then Test Director will build a skeleton for the script that can be later modified into one which could be used to test the AUT.
What are the different modes of recording? - There are two type of recording in Win Runner. Context Sensitive recording records the operations you perform on your application by identifying Graphical User Interface (GUI) objects. Analog recording records keyboard input, mouse clicks, and the precise x- and y-coordinates traveled by the mouse pointer across the screen.
What is the purpose of loading Win Runner Add-Ins? - Add-Ins are used in Win Runner to load functions specific to the particular add-in to the memory. While creating a script only those functions in the add-in selected will be listed in the function generator and while executing the script only those functions in the loaded add-in will be executed else Win Runner will give an error message saying it does not recognize the function.
What are the reasons that Win Runner fails to identify an object on the GUI? - Win Runner fails to identify an object in a GUI due to various reasons. The object is not a standard windows object. If the browser used is not compatible with the Win Runner version, GUI Map Editor will not be able to learn any of the objects displayed in the browser window.
What is meant by the logical name of the object? - An object’s logical name is determined by its class. In most cases, the logical name is the label that appears on an object.
If the object does not have a name then what will be the logical name? - If the object does not have a name then the logical name could be the attached text.
What is the different between GUI map and GUI map files? - The GUI map is actually the sum of one or more GUI map files. There are two modes for organizing GUI map files. Global GUI Map file: a single GUI Map file for the entire application. GUI Map File per Test: Win Runner automatically creates a GUI Map file for each test created.
GUI Map file is a file which contains the windows and the objects learned by the Win Runner with its logical name and their physical description.
How do you view the contents of the GUI map? - GUI Map editor displays the content of a GUI Map. We can invoke GUI Map Editor from the Tools Menu in WinRunner. The GUI Map Editor displays the various GUI Map files created and the windows and objects learned in to them with their logical name and physical description.
When you create GUI map do you record all the objects of specific objects? - If we are learning a window then WinRunner automatically learns all the objects in the window else we will we identifying those object, which are to be learned in a window, since we will be working with only those objects while creating scripts.
Subscribe to:
Posts (Atom)
