|
 Автор: Михаил Давыдов
Статья изначально написана для западного заказчика по результатам предварительного evaluation IBM Rational Functional Tester (к моменту написания статьи автор уже имел продолжительный опыт работы с Mercury QuickTest Pro 6.5). Целью evaluation был (предварительный) выбор инструмента автоматизированного функционального тестирования Web-приложения, разрабатываемого для заказчика нашей компанией. Статья не претендует на исчерпывающий анализ обоих инструментов — такой цели и не ставилось. Цель статьи — проанализировать сферу применимости обоих инструментов, обозначить их наиболее сильные и наиболее слабые стороны. Приношу читателям извинения за свой, не всегда адекватный английски язык (а так же то, что я не удосужился перевести статью на русский).
Software Comparison: Rational Functional Tester v.6.1 vs. Mercury Quick Test Pro 8.1
1. Introduction
1.1. Initial choice of commercial tools for automated functional testing of Web applications.
Only tools, aimed at (primarily) Web testing, were considered. The majority of market-share of regression-testing tools is now shared by only few vendors: Mercury Interactive, IBM Rational, CompuWare, Segue and Empirix. Rest vendors share less then 15% of market. Mercury Interactive has the largest market share, IBM Rational has the second. Besides commercial tools, there is a few of open source ones.
Both market leaders — Mercury Corp. and IBM Rational have (each) 2 product lines for automated functional testing. Mercury has WinRunner and Quick Test Pro. IBM Rational has Rational Robot and Functional Tester (formerly RobotJ and, later, XDE Tester for Java and Web). WinRunner and Rational Robot are rather "old" products: both have more than 10 years history. Another major player on the market, Segue, has only one product line — Silk Test — it also has rather long history. Both, Rational Robot and WinRunner, were not initially aimed at web testing, but rather, on testing "classical" Windows applications (WinRunner, even had a few versions for Windows 3.11). Both tools support Web testing, but in less efficient way then they successors (QTP and Functional Tester). The other difference between these two generations of functional testing tools is in the scripting language that is used for automated scripts development. All three (mentioned above) "first generation" functional testing tools (Rational Robot, WinRunner, Silk Test) use so-called "vendor-scripts", e.g., they have their own scripting languages. It’s not very good — because of additional time to learn these languages, compatibility problems and overall relative weakness comparing to common scripting languages. Quick Test Pro uses Microsoft VBScript as its scripting language. Rational Functional Tester uses Java.
Both Mercury Interactive and IBM Rational are, step by step, shrinking support for their "first-generation" tools (WinRunner and Rational Robot, respectingly). That’s why we can take these tools out of our consideration.
1.2. Quick Test Pro: Overview.
Quick Test Pro, formerly known as "Astra Quick Test" was launched by Mercury as an alternative to WinRunner in the field of (primarily) Web Testing. In private conversations, Mercury employees say that "Mercury has learnt many lessons from WinRunner", and QTP is a result of such an analysis. As a sign of such "lessons learnt" we can consider scripting language change, as well as changing Test Objects model.
QTP uses Microsoft VBScript as its scripting language. It’s good because VBScript fully support all Microsoft component technologies, such as COM, OLE Automation and ActiveX. VBScript, also, was initially "Web-oriented" language — it’s widely used on client-side in web applications. Another VBScript advantage is that it’s very easy to learn it. From the other hand, the choice of VBScript is not very good, because VBScript has some significant limitations: for instance, it doesn’t support modules/libraries and cannot be considered as "truly" object-oriented language: it does not support inheritance of classes. Since VBScript does not support libraries/modules, debugging of functions outside script itself, is not supported — it makes overall script debugging harder. Another issue in VBScript that makes debugging harder is very weak syntax checks (before running scripts).
QTP benefits full advantage of object support of VBScript. Unlike Rational Robot and WinRunner, GUI Objects in QTP have multi-level hierarchy. It makes QTP scripts more compact and removes unnecessary commands, such as "SetWindow", "SetFrame" (Rational Robot) and "set_window "(WinRunner). Another advantage is a "natural" connection to GUI Object’s native properties and methods. Once you get ".object" property, of, say, HTML control, you can do with it anything you can do in VBScripts on HTML page, and in absolutely same way. Functional Tester, for the comparison, uses so-called "proxies" for GUI object properties and methods usage (mainly, because of Java cross-platform nature: VBScript is on more friendly terms with COM and OLE Automation).
Unlike Rational Functional Tester, QTP’s object hierarchy, in case of Web tests, doesn’t fully correspond to HTML DOM (Document Object Model). QTP hierarchy for Web pages is always following: Browser->Page->Frame->Objects in frame. It simplifies scripting, but, in some cases, makes some problems in objects recognition.
QTP, as well as Functional Tester and WinRunner, uses some kind of object map (so-called "Object Repository") for AUT GUI objects management. QTP’s Object Repository is similar to GUI Map of Win Runner’s GUI Map in the way it’s used in scripts. Object Repository provides script developer with logical names for each GUI object in AUT, so that tester can use it instead of describing all recognition properties of object (as it’s done in Rational Robot). Object Repositories in QTP scripts are stored in a separate file in proprietary binary format. Object Repositories management in QTP is rather limited, because of their complex structure. User cannot copy a "child" of one, say, frame to another, can’t (easily) copy an object from one OR to another, cannot manually insert new object (without launching AUT),etc. However, in Rational Functional Tester, things are hardly better (unlike WinRunner). It’s a charge vendors have to pay for making Object model more complex.
Fig.1.2.1.Snapshot: QTP Object Repository.
QTP has some unique IDE features that no other tool from other vendors has — such as Active Screen, Tree View, Method Wizard and some others. These features are aimed at simplification of script development for inexperienced users. However, experienced testers (AFAIK), often avoid their usage. Overall, QTP’s IDE rests its user with mixed feelings. It lacks some very common (for, even, most simple code editors) features, while provides user with some very advanced features, related to managing test assets (Objects Repository, Data pool, Checkpoints, Test script as a tree).
Fig.1.2.2. Snapshot: QTP Tree View.
Fig.1.2.3. Snapshot: QTP’s Active Screen feature.
Besides specific IDE features, QTP has some other unique features (mostly, related to tool configuration and replay options), some of them are very handy and are often used even by experienced testers. The example of such a feature is a Recovery Manager (it helps handling unexpected behavior of Software (Application) Under Test (SUT, AUT), such as appearance of error dialog boxes, application crashes and so on). The absence of this feature in Rational Functional Tester is an important factor in testing tools comparison.
Fig.1.2.4. Snapshot: QTP’s Recovery Scenario.
Very significant strength point of Mercury Interactive Corp. as a software vendor is its support services (so-called "Customer Support Office"). It includes on-line services (forums, Knowledge Base, users-contributed Knowledge Base articles and uploads) and Service-request services (Web, phone and e-mail service requests).
Documentation for QTP is also very good. All the product features are well documented, for every feature there are code examples. I can’t remember a thing that I could not find in Help.
1.3. Functional Tester. Overview
Rational Functional Tester was first introduced by Rational as "RobotJ" and was aimed at Java and Web testing. Rational made a stake on Java as a scripts language and Eclipse as an IDE. Though, this choice is a bit disputable (Java is not a language that is very often used for scripting, in the first order, because it takes more time to learn and requires some additional efforts in development), it was a strong move. RobotJ was first functional testing product from Rational with Object Map feature. Though it simplified significantly management of recorded scripts, it as well, added some difficulties: "descriptive" programming (e.g., by recognition properties descriptions instead of referencing Object Map) in RobotJ is very complicated (especially taking into account its not very good documentation). Together with rather complex object model, it frightened away inexperienced testers from this product. Rational Robot was still most common choice (from Rational Family) as a functional testing tool.
In product’s new version, Rational seemed to have learnt the lesson. Descriptive programming support was added ("atChild","atDescendant","atList" functions), new scripting language and IDE (VB .Net + Visual Studio) were introduced (they didn’t replace Java scripting + Eclipse, it’s just an addition), documentation was re-worked (though, it’s still worse than QTP’s one). Now Rational Functional Tester is a matured, flexible and highly extensible product with "true" IDE (- not a Notepad-like IDE of Rational Robot or lack-of-editing-capabilities QTP’s one. I think it’s enough to say that Eclipse one of most popular Java IDEs, Visual Studio also does not need recommendations). Functional Tester takes full advantage of Java (and Visual Basic .Net in VB .Net scripting edition) as an OOPL. Documentation is still its weak point, but it’s not a stopper-issue.
Fig.1.3.1.Functional Tester IDE (Java Scripting)
Rational FT API differs significantly from QTP’s one. It’s mainly because of its Java nature. All the internals are visible (though, unfortunately, not well documented and without source codes), some of core classes allow inheritance — it makes FT more scalable. But, on the other hand, Functional Tester’s API is more complex that QTP’s one. It adds complexity to scripts, as a charge for better flexibility.
Object Map feature implementation also differs quite significantly from QTP’s Object Repository: it’s not used directly from scripts. Rather, Object Map is used for automatic generation of "Helper script" objects (strictly speaking, helper methods). Object Map objects are not necessary used in scripts. For instance, if you need to do some action on HTML button, you don’t need to mention Frame, Page and Browser — they appear neither in main script, nor in helper script. There is non-obvious advantage from this approach: you may have 3 browsers with same description, but it absolutely will not affect object recognition when script replays: only browser that has given child will be selected. In QTP it would be a problem. Another advantage is that several scripts using same Object Map may use differently named helper objects. Another difference is that Functional Tester’s object map fully corresponds to DOM, unlike QTP.
Fig.1.3.2. Functional Tester Object Map.
Test Object implementation also differs from QTP’s one. In QTP Test Object is defined by its description. It‘s not necessary "connected" to run-time AUT object. You can create test object without having it displayed in AUT. It’s not so in Functional Tester. In Functional Tester, you create Testing Object immediately before using it, and it’s always connected to AUT. It adds some extra complexity in scripts development, because after using Test Object variable you have to unregister it immediately after you don’t need it (otherwise, AUT may experience some difficulties freeing it).
IBM Rational Support is known to be a bit less advanced that Mercury’s one because of fewer resources, available online and fewer Support Engineers (though, it may be only rumors. I didn’t have an experience with IBM Rational support). Meanwhile, Functional Tester is less expensive then QTP.
2. Rational Functional Tester vs. Mercury Quick Test Pro comparison.
2.1. Scripting language
QTP uses VBScript as scripting language, while Rational Functional Tester uses Java or VB .Net. Let’s compare strength and weak points of their implementation in both tools:
| Criteria | VBScript | Java and Visual Basic .Net |
|---|
| Learning Curve |
Easy to learn. Language is mostly procedural, with no strict variable types and no inheritance of classes. Usage of objects (especially, COM and OLE automation objects) is very easy. Exception handling is also quite easy. VBScript is not very feature-rich: about 100 functions + a few "Scripting" COM objects. Syntax is rather strange and abundant, but language itself (in form of VBA or Visual Basic) is very widespread.
Score: 9
|
More complex than VBScript. Script developer must be aware of classes hierarchy, method types (public, static, abstract, overloaded, etc.), variable types.
Since both Java and VB.Net are more feature-rich, of course, some time is needed to learn most useful features.
However, both Java and VB.Net are easier than other OOPLs, such as C++ or Delphi.
Score: 5
|
| Simplicity / Complexity |
Developer doesn’t need to care about variable types, classes, and so on: less functionality means more simplicity.
Score: 8
|
Once developer had learnt language, development is rather easy. Of course, developer should take care of much more things than in VBScript, but after he/she gets use of it, it’s not a real problem.
Score: 7
|
| Scalability |
Lack of native libraries support (it makes debugging of externally defined functions in QTP impossible). Lack of inheritance of classes. No events. Very limited exception handling. Classes in external files are not fully supported.
Score: 2
|
True, full-scale OOPL (both VB.Net and Java).
Score: 10
|
| Debugging |
See above. Since libraries are not supported, debugging them is always a headache. Besides that, debugging capabilities are rather standard.
Score: 4
|
No "Retry" option. If exception occurs, you can’t retry last step.
Score: 8
|
| Support for COM/OLE/ActiveX |
Full support. Extremely easy to use.
Score: 10
|
No built-in support. External libraries ("COM bridges") are needed.
Score: 4
|
| Features (functions, built-in objects, 3rd party libraries, etc.) |
Score: 5 |
Score: 9 |
| Summary |
Pros:
Easy to learn. Advantage: dealing with COM is easy.
Contras:
Not enough powerful and scalable. No support for libraries/modules
Overall score: depends on criteria
|
Pros:
Very feature-rich and extremely scalable.
Contras:
Not very easy to learn. More complex. No built-in support for COM.
Overall score: depends on criteria
|
2.2. IDE
Classical IDE features:
Rational Functional Tester:
Pros:
Visual Studio .Net or Eclipse. No need to add a word.
Contras:
No
Summary:
Score: 10
Quick Test Pro:
Pros:
No (at least, over Rational Functional Tester).
Contras:
Limited editing capabilities; Rather weak "auto-complete" (it does not work with external libraries and user-defined functions and classes); no syntax checks; a bit buggy integration with Object Repository (at least, in QTP 6.5).
Summary:
Score: 3
Testing-specific IDE features:
Rational Functional Tester:
Pros:
All necessary testing functionality (such as Verification Points, new recording, Test Objects) is inserted quite easily. All test assets are easily available and editable. Object Map functionality and Object Spy are mighty, though are not perfect.
Contras:
No Tree View/Keyword View. No Active Screen. No testing-oriented commands on right-click in editor. Data pool editor has very limited functionality.
Summary:
Score: 6.
Mercury Quick Test Pro:
Pros:
Unique tree view/keyword view feature; Unique Active Screen feature; Items-rich right-click menu; very accessible and rather powerful Data Table editor. Auto complete for Object Repository objects.
Contras:
Most right-click pop-up menu items are available only for recorded script steps. Checkpoints are available only from right-click pop-up menu in editor. Object Repository functionality is less feature-rich than Object Map from Rational.
Summary:
Score: 8.
2.3. Test Objects model and API
| Feature | Mercury QuickTest Pro | Rational Functional Tester |
|---|
| Test Object Concept |
Test Object is the description of an object in AUT. It can be connected to AUT (and, I know one bug in QTP when test object can’t restore connection to Browser), but, generally speaking, Test Object is rather a description of GUI object than a representation of "live" object in AUT. Test Object can be created using "literal" description, using Description object, or by referencing object map. Also, some methods of Test Objects may return other Test Objects (say, children). Test Objects are, primarily GUI Objects — e.g., combo items and, moreover, processes are not Test Objects. |
Test Object is the representation of an object in live application. It can’t exist in script if it does not exist in application. It’s not necessary, GUI Object. It can be a process, a combo item, a "Root Test Object", a "Domain" object. Test Objects are always created by invoking some methods (most commonly, "find" method) of parent Test Object (the only exception is the root Test Object that is returned by a method "getRootTestObject" of RationalTestScript object). After Test Object is explicitly created, it should be unregistered. |
| Referencing objects by description |
Pros: Very simple syntax. All objects in hierarchy can be referenced by description in one operator.
Contras: All objects in hierarchy should always be mentioned. E.g., you can’t reference to button in frame as Browser->WebButton, but should reference it as Browser->Page->Frame->Button. Since you should give correct and non-ambiguous description for each object in hierarchy, it’s not good for script reliability.
|
Pros: You don’t have to describe full object hierarchy to reference child object. "atDescendant" function is very useful.
Contras: If you want to reference object by description in one operator, "Parent" object must be a helper object, otherwise you should write some custom functions to do that. Also, only 2-tier hierarchy is available: e.g., Parent->Descendant (though, not necessary, direct) and not Parent->Child->Grandchild.
|
| Referencing objects from GUI Map |
Pros: Referencing mapped objects by logical name is, perhaps, most simple and natural way. You can have same logical names for objects that are not same level children of same parent. It may be quite handy. Also, navigation in tree-like Object Map is much easier (when script is enough long) than in "flat" helper objects list.
Contras: You (again), always have to describe (now, by logical name) each object in object hierarchy. It’s not compact, and, again, leads to problems with ambiguous recognition of parent objects.
|
Pros: Using helper objects instead of using full object hierarchy is very compact way of referencing objects. Again, there are no problems with ambiguous recognition of parent objects.
Contras: Naming problems:
For instance, if you have 3 buttons "OK" in 3 several dialogs, you would have to name each one differently. If script is quite big, navigating in huge flat helper objects list may become a problem.
|
| Dealing with "native" object properties and methods. |
Via ".object property". Very easy and convenient. |
Via "proxies": in FT, there is no difference between "native" and "common" properties. Native methods are called via special methods, I didn’t test it yet. |
| Dealing with common object properties |
Very simple syntax. Rather slow, especially on large HTML documents (because QTP searches for object each time it’s methods called or properties retrieved) |
Quite similar to QTP (in syntax). Fast (when working with TestObejct by reference). |
| Retrieving specific object data (Table cells, list items, etc.) |
Very simple syntax. Slow (see above). |
Via very complex interfaces (so-called TestData interfaces), but can be very easily "wrapped" into user-defined functions. |
| Test Object common methods |
Rich functionality. |
Standard functionality. For instance, there is no "waitProperty" method (but, again, there is no problem to write a user-defined function). |
| Summary |
Overall score: depends on criteria
Simplicity: 9
Syntax compactness: 4
Speed of execution: 3 (Note: actually, if .object property is used, speed is the same as in FT, but it’s not a common practice)
Flexibility: 6
Compatibility (native objects methods/properties): 9
|
Overall score: depends on criteria
Simplicity: 4
Syntax compactness: 6
Speed of execution: 8 (Note: speed of object recognition is the same as in QTP, but developer is forced to use "run-time" objects).
Flexibility: 9
Compatibility: 5-7 (not tested)
|
2.4. Object Recognition and GUI Map
| Feature |
Mercury QuickTest Pro |
Rational Functional Tester |
| What properties are recorded? |
Configurable (for each Test Object Class). Three kinds of recognition properties: mandatory (recorded always), additional (recorded if more than 1 object have same mandatory properties values) and "order" (used if both mandatory and additional properties are used, but recognition is still ambiguous). 4th kind of properties stand a bit aside: "Smart Recognition" properties are used optionally, and only if Smart Recognition is enabled. |
Not configurable. |
| "Order" properties |
Index, Location, CreationTime (for Browser objects only) |
.classIndex |
| Additional recognition features |
"Smart Recognition" — ordered list of "filtering" properties. Test Objects are filtered by one property after another, till only on Test Object rest, it wins. |
"ScriptAssure" (each property has it’s "weight" factor, test object with most weight "wins" in competition) |
| Mapping to Test Object class |
Yes (Not needed for Web testing). |
No. |
| GUI Map |
Name: "Object Repository". Structure is simplified (comparing to DOM). Key field is path (nodes of same parent with same name are not allowed). File format: proprietary binary. Types: Shared, per-action. |
Name: "Objects Map". Structure fully corresponds to DOM (in case of Web testing). Key field is internal "Id" property. File format: XML-based. Types: Shared, Private (per-script) |
| Regular expression as recognition property value |
Yes |
Yes |
| Summary |
Object recognition properties recording can be configured very finely, but improper configuration often leads to unusable scripts. Proper configuration of object recognition is one of most complex tasks in QTP.
Score: 5
|
Recording of recognition properties is not configurable. However, though I have not tested it thoroughly, it seems that it just works.
Score: 4-7 (more testing is required)
|
2.4. Recording
| Feature |
Mercury QuickTest Pro |
Rational Functional Tester |
| Can be paused? |
Yes |
Yes |
| Inserting verification points while recording |
Yes |
Yes |
| Editing script while recording |
Yes |
No |
| Editing GUI Map while recording |
No |
No |
| Inserting Test Object methods while recording |
Yes |
Limited |
| Correct recognition of objects, already in GUI Map |
Problematic (object recognition configuration should match exactly set of object’s properties in GUI Map) |
Needs further investigation |
| Insert recording in script |
Yes |
Yes |
| Summary |
Problems with correct recognition of objects, already in Object Repository: often new Objects in Repository are created
Score: 6
|
Standard. Correctness of recognition Object Map objects in recording time is not tested yet.
Score: 5-7
|
2.5. Replay, Recovery and Debugging
| Feature |
Mercury QuickTest Pro |
Rational Functional Tester |
| Can be paused? |
Yes (Normal mode) |
Yes (Debug mode) |
| Retries on exception |
Yes |
No |
| Replay Modes |
Normal, Fast. Error handling is configurable |
Standard, Debugging. Error handling not configurable |
| Log available in replay time? |
No |
No |
| "Recovery Management" |
Handled:
— Test exceptions (Object not found, Ambiguous Recognition, Object disabled, etc.)
— AUT crashes
— Pop-up windows
— Appearance of specific objects (very handy to catch "Internal Error" messages)
Recovery actions:
— key press
— call function
—restart Windows ?
Post-recovery:
— retry step
— skip to next step
— skip to next action
— Stop script.
|
No (test exceptions, such as "Ambiguous recognition" can be handled by custom methods of Superscript class). Actually, most of "recovery management" can be done this way. The only (but very important) exception of this rule is appearance of specific Test Objects — they should be caught by explicit calls to corresponding custom method (that will search for such objects and raise exception if they’re found) — at least, it’s the only way I see. |
| "Arrow" in code editor while replaying script |
Yes (in Normal mode) |
No |
| Script editable when debugging |
No |
Needs further investigation |
| Object Repository editable while replay? |
Tricky (possible only because of a defect in QTP 6.5) |
Needs further investigation |
| "Run Wizard" (feature of WinRunner, allowing automatic script and GUI Map correcting while run) |
No |
No |
| "Update Mode" |
Yes (Baselines, Test object descriptions and ActiveScreen snapshot can be updated together or separately). |
No |
| Debugging inside libraries/modules
(not included in score calculation — already included in "Language" topic)
|
No |
Yes |
| Summary |
Recovery management is a very handy and enough flexible tool. Update Mode is not enough flexible for updating both baselines and Test Object descriptions, that’s why it is used quite rarely.
Score: 7.5
|
Quite standard.
Score: 5-6
|
2.6. Logging
| Feature |
Mercury QuickTest Pro |
Rational Functional Tester |
| Tree structure |
Yes |
No |
| Event type filtering in viewer |
Yes |
No |
| Event type filtering in replay time (by changing logging options in run-time) |
Yes |
Partial (all errors are displayed in special list control) |
| Screenshot capture on errors and warnings (standard API functions only) |
Yes |
No |
| Multiple event types |
Yes |
Yes |
| Known bugs |
Tree structure is often corrupted on recovery scenarios activation and "action events" (such as action stop). |
Unknown |
| Summary |
Score: 8 |
Score: 3.5 |
2.7. Data Pool (Data Table) and data-driven testing
| Feature |
Mercury QuickTest Pro |
Rational Functional Tester |
| Typed variables/parameters |
No |
Yes |
| API |
Very convenient |
Purely undocumented, and, there for, not tested. Data pool commands are inserted, mainly, through wizards. |
| Speed |
Slow |
Needs investigation |
| Next row |
On next action iteration |
Needs investigation |
| Usage in verification points |
Yes |
Yes |
| Usage in GUI Map |
Yes |
Needs investigation |
| Summary |
It is not very important feature (because, generally, it’s easier to write custom functions to read data from files), but, generally, QTP "Data tables" seem to me to be much more convenient that FT’s Data pools.
Score: 7
|
Not tested due to poor documentation. Most probably, can be easily replaced with custom classes.
Score: 2-5
|
2.8. Verification Points/Checkpoints
| Feature |
Mercury QuickTest Pro |
Rational Functional Tester |
| Properties checkpoints |
Yes |
Yes |
| Table checkpoints |
Yes |
Yes |
| Regular expressions support |
Yes |
Yes |
| Case sensitive option |
Yes |
Yes |
| Ignore spaces option |
No |
Yes |
| Data pool option |
Yes |
Yes |
| Numeric range option |
Yes |
Yes |
| Image Checkpoints |
Yes |
Needs (free) add-in |
| Database Checkpoints |
Yes (ODBC) |
No |
| Table checkpoints advanced features |
Intuitive, not enough flexible |
Rather complex, but very feature-rich |
| Format |
Binary |
XML |
| Dynamic checkpoints (baseline is saved (interactively) on first script run) |
No |
Yes |
| Manual checkpoints (baseline is either saved non-interactively on first script run or is supplied as method argument) |
No |
Yes |
| Navigation through checkpoints |
No |
Yes |
| Delete checkpoint |
No |
Yes |
| Non-Test Object verification points |
No |
Yes |
| Summary |
Score: 5 |
Score: 7 |
2.9. Test assets editors and tools (GUI Map editor, Verification Points editor, Data pool/Data Table editor, Object Spy).
| Feature |
Mercury QuickTest Pro |
Rational Functional Tester |
| GUI Map Editor |
Save as: Yes
Version Control integration: No
Edit, Delete properties: Yes
Add properties: No
Edit additional recognition properties: Yes ("Smart Identification")
Add object from AUT: Yes
Delete object: Yes
Add object manually: No
Move Node: No
Batch node edit: No
Integration with script (on logical name change) — buggy (in QTP 6.5 it may corrupt script)
Score: 6
|
Save as: No
Version Control integration: Yes
Edit, Delete properties: Yes
Add properties: Only in Batch modify mode
Edit additional recognition properties: Yes ("Script Assure": properties weights)
Add object from AUT: Yes
Delete object: Yes
Add object manually: No
Move Node: Yes
Batch node edit: Yes
Integration with script: Yes
Score: 9
|
| Verification Points Editor |
Score: 6 |
Score: 6 |
| DataTable Editor |
Multiple cell’s select: Yes
Multiple cell’s copy: Yes
Formulae (a’la Excel): Yes
Import/Export: Yes
Known bugs (!)
Score: 6
|
Multiple cell’s select: No
Multiple cell’s copy: No
Formulae (a’la Excel): No
Import/Export: No (only once)
Score: 2
|
| Object Spy |
Only properties, configured as "recognition properties" are shown.
Known bugs (!)
Score: 5
|
Score: 7 |
| Summary |
Known data corruption bugs (in Object Repository editor and DataTable editor)
Average capabilities, average usability.
Overall Score: 5.8
|
Very capable Verification Points editor and better (comparing to QTP) Object Map editor. Very feature-poor data pool editor.
Overall Score: 6.8
|
2.10. Command-line options, Automation model.
| Feature |
Mercury QuickTest Pro |
Rational Functional Tester |
| OLE Automation Interface |
Yes (all QTP options, commands and settings are controllable through automation) |
No |
| Run script from command-line |
Yes |
Yes |
| Log file path from command line |
Yes |
Yes |
| Run options from command line |
Not investigated (Since it’s more convenient to run QTP from automation scripts) |
No |
| Summary |
Overall score: 10 |
Overall score: 4 |
2.11. Integrations
| Integration |
Mercury QuickTest Pro |
Rational Functional Tester |
| Version Control Systems |
Through TestDirector only. Only test as whole can be checked in or checked out. |
Built-in integrations with version control systems:
— CVS
— ClarCase
Integration with other version control systems is, probably, possible (using Eclipse plugins), but it needs additional investigation.
Files can be checked-in separately.
|
| Test Management and Remote test runners |
TestDirector (Store, Run remotely, Store logs) |
TestManager (Run remotely, Store logs) |
| Summary |
Version control needs TestDirector.
Score: depends on required integrations
|
Version control does not need any additional commercial software.
Score: depends on required integrations
|
2.12. Documentation
| Feature |
Mercury QuickTest Pro |
Rational Functional Tester |
| User Guides |
Score: 7 |
Score: 5
(Less detailed, comparing to QTP User Guide) |
| API documentation |
QTP Automation API
Score: 6
QTP Object Model Reference
Score: 7
|
FT API Help: Very few examples, some methods are purely undocumented
Score: 3 |
| Supplied Scripting Language Documentation |
Windows Scripting Technologies help from MSDN
Score: 8 |
javadoc comments
Score: 4
|
| Summary |
Overall Score: 7 |
Overall Score: 4 |
2.13 Support
| Feature |
Mercury QuickTest Pro |
Rational Functional Tester |
| Access to support resources (documentation, forums, articles) |
For users with current maintenance contracts only |
For all users |
| Patches and new versions |
For users with current maintenance contracts only |
For users with current maintenance contracts only |
| Knowledge Base |
Yes (for users with current maintenance contracts only) |
No |
| Forum |
Yes (for users with current maintenance contracts only) |
Yes |
| Service Requests |
Yes (for users with current maintenance contracts only) |
Yes (for users with current maintenance contracts only) |
| User-contributed downloads |
Yes (for users with current maintenance contracts only) |
No |
| User-contributed KBA |
Yes (for users with current maintenance contracts only) |
No |
| Technical Articles |
Yes (for users with current maintenance contracts only) |
No |
| Summary |
Support is better and includes more options, but is more commercialized
Score: 7
|
Support includes fewer options, but online resources are free.
Score: 4.5
|
2.14 Pricing
| Option |
Mercury QuickTest Pro (Average prices)* |
Rational Functional Tester * |
| Site (node-locked) license |
~6-7.5K EUR (without maintenance) |
Includes: 1 Year maintenance
UK: 5.706$
USA: 4.120$ |
| Concurrent license |
~10-12 K EUR (without maintenance) |
Includes: 1 Year maintenance
UK: 10,985$
USA: 7.930$ |
| Maintenance fee (1Year) |
~20% |
USA: 2.220$ (floating)
UK: 2.900$ (floating) |
* excluding taxes
2.15. What rest out of scope?
Compatibility, reliability and quality issues were mainly out of scope, because of limited evaluation period.
3. Conclusions
3.1. QuickTest Pro
Pros:
Mercury Interactive support services are known to be one of the best on the market (of Software Quality related software).
QTP has better learning curve, VBScript as language used for script development is easier than both Java and VB.Net. QTP has some unique features that are not present in other tools, including Rational Functional Tester (say, Recovery Scenarios). QTP’s API is easier and more intuitive. Some other features are just more advanced (say, Logging and Data Table editor). Usage of VBScript brings QTP full compatibility with Microsoft component technologies (COM, ActiveX, OLE Automation, etc.). QTP has OLE Automation interface, that allows user to write batch scripts (on VBScript), that can be run from command line.
Contras:
VBScript as scripting language is not enough scalable and flexible, and is less feature-rich than Java and VB.Net. The fact that Functional Tester can use 2 languages in its scripts is also not to an advantage of QTP. IDE is very weak, comparing to both Eclipse and VisualStudio. Debugging of functions, contained in external files (libraries) is not supported, and, again, it limits QTP tests scalability. Some of features are implemented worse than in Functional Tester (say, Object Repository and Verification Points). There are rather severe bugs, leading to data corruption. QTP uses binary proprietary formats for storing its assets, and (together with mediocre reliability) it’s rather dangerous and is not a good practice. QTP is much more expensive than Functional Tester (AFAIK, it’s the most expensive solution on the market).
3.2. Rational Functional Tester
Pros:
Rational Functional Test enjoys Java or VB.Net as script languages and Eclipse or Visual Studio as an IDE. Both language and IDE are much more feature-rich, flexible and extensible. Test Object model and API are more flexible as well. Some features are better implemented in Functional Test than in QTP. During evaluation we experienced no problems with tool stability (except for performance issues that are rather common for java applications). Functional Tester is significantly (nearly 2 times!) less expensive, and does not require maintenance contract for access to vendor online resources (such as forums, technical articles etc.).
Contras:
Java is more difficult language than VBScript and requires more qualification from testers. Rational Functional Tester lacks OLE Automation interface, it makes it more tricky (though, not impossible) to run scripts in unattended mode. Some features of QTP are absent in Functional Tester (Recovery Scenario Manager) and others are less powerful in Functional Tester (Datapool editor).
|
