Posted by
Amit Kulkarni [Admin]
| Posted in
application
,
approach
,
compare
,
exploratory
,
justifiable
,
justification
,
topic
| Posted on
9/13/2009 09:51:00 PM
I like to share my thoughts on what I feel. When I interact with my fellow testers and after listening to them after a long discussion.
Today, I just want to discuss on something which I feel needs some serious thoughts. We get a new app to test but we try to compare it with the older app. Is this okay? I am not saying we can not have the comparion but the best approach would be to have the older app as your baseline for new test for your newer app. A tester obviously needs some scenarios, cases, user scenarios and practical approach to the app. But it would be much better if it stays the baseline and we do not get serious with it....we see that the older app has this feature but newer ones does not have one so does it mean that the newer app is not good? So if you start doing comparison you do not get a chance to explore the newer app to its fullest extent.
I as a tester do exploratory testing - and see that some of the newer app does not have the feature which I was using quite frequently in the older app. Does it mean that I should not use the newer app? Not really!! If we start thinking like this then there is no point of having new products in the markets where in the company know that the older app is much robust. But development is all about coming up with new features, attracting to new users and try to get the existing customers/users to use your app which in itself which increases your potential (as a company) in the market. Companies are not wasting their money when they come up with newer app (or rather to be specific new product which have more features than the old ones) os it helps them to gauge what the users are looking out for.
This topic is kind of debate where in I have shared my thoughts on this. Others would have their own say on this which quite justifiable giving their expertise in their respective areas. This blog I just wanted to share my thoughts so I have come up with this. Its always nice to hear from others and their justification for their approaches.
Lets chip in and have your say on this.
Best Regards,
Amit
testing is my passion!!!
http://bugteaser.blogspot.com
I am not sure what you mean by old app and new app. Does it mean it is the same application with old and newer versions?
Parimala Shankaraiah
http://curioustester.blogspot.com
Pari,
I am referring to old app as - one which is already stable, robust and used by many people in the market versus the newer version of the same app but by different company (competitor) so the features may vary.
Do you believe in comparing the new features in newer app as that of the older app (which is quite stable not to mention)?
Best Regards,
Amit
testing is my passion!!!
http://bugteaser.blogspot.com
Try to find and answer yourself, for what number of iterations you have downgraded (for lower version) or upgraded (for higher version) an installed software application, on the computer you are using.
Also ask yourself, have you chosen the other vendor's software application by uninstalling or also installing along with existing vendor's computer software application.
If one sees the need, change may be the one solution or not. Like, you have changed the buckle position of your waist belt (if you tied the belt around the waist and did tie it from past 10 years). If your waist size has increased, then the buckle position might want new position (version) or you can discard the waist belt and be without it. And the same if, you had reduced the waist size (downgrade to earlier version) of yours.
If a particular computer software application is taken as successful by the vendor who had put it into the market, may the vendor go for next version of it. And may be even when not successful too, vendor may go for higher version of it for retaining the people who the vendor has.
So, you said older app is robust. You should research on count of people who upgraded the Operating System as well as who downgraded the Operating System, if such scenario or contexts have taken place. If you find any thing here ask yourself with what you have written in http://bugteaser.blogspot.com/2009/09/is-comparison-valid-while-comparing-new.html
Or try to look out which users have upgraded their mobile phone handset to the latest model in the market and the count of users who have chosen the older model then the current make they have from the manufacturer or other manufacturer. Once you do this, find what made them to take such decision?
Though newer releases or models or versions have (or not have) features which were not available in the older releases or models or versions, people yet purchase the older models or makes or versions of the manufacturers or vendors, why? Older makes still attracts the users than the latest makes, why?
Try to figure out why new models or makes or versions or releases have been withdrawn from the market without noise, while still the earlier available are getting the users. And, find the reasons.
Go around your colleagues, friends and known people to you and ask them, have they upgraded to Standard version of Gmail or continuing to use the basic HTML, if they used GMail. Do the same for Yahoomail too.
Do not /just/ compare the older version with latest version. Instead install older version on one machine and latest version on other machine. Learn using both of these.
What ever it might be, the good or bad aspects /for you/, will have newer versions of it. And may be the newer version can have earlier version properties or features with much more complexity or reduced complexity /for you/.
Whatever you have said makes and the perfect way of looking at the things - which you have elaborately explained.
Concern over here though. Will give an example:
There is a product X (for more than 5 yrs now)
Some company ABC come up with the product Y to compete X.
Assuming that I was asked to test product Y. Do I still need to test product X, as product Y is new in the market but does the same functionality as that of X.
In the above case while testing product Y, do I still have to do comparison? Or I can just take product X as my baseline and explore product Y.
Best Regards,
Amit
testing is my passion!!!
http://bugteaser.blogspot.com
>> You did not say who is your employer or does the product X is from you or your employer.
>> Looks like you are very near to word confusion and yet to fall into the word 'confused'. And looks like there are lots of dilemmas in your thoughts you are trying to put here.
In the above case while testing product Y, do I still have to do comparison? Or I can just take product X as my baseline and explore product Y.
What you mean while saying the word 'comparison'. Comparison is not TESTING. It is like you are trying to find who is taller or shorter. But you are not learning and looking for details that give and makes you to learn much more that you might not get with comparison:
* why they are taller or shorter and why they look to be taller or shorter,
* for how long they /look or/ have been taller or shorter,
* for how long they /look or/ will be taller or shorter,
* for whom they look as taller or shorter and why,
* for whom they look taller or shorter and when,
* for whom they look as taller or shorter and how,
* from where I am looking at them,
* whether I am seeing there shadow in different luminosity of light,
* whether they are from different geographical place or planet or period,
* whether I see that they are standing or I am thinking that they are standing,
* whether they look taller or shorter whey they sit,
* whether they look taller or shorter why they run,
* whether they look taller or shorter when they jump,
* whether they look taller or shorter when they sing or scream or shout or yawn or open their mouth or their eyes or spread their arms or stretch their body or lift their leg,
* whether you are seeing them inside the water or any other medium with adequate luminous light or not at all,
* whether you are seeing them inside any other medium which looks like liquid,
* what is the density of medium they are in,
* what kind of light you are using,
* what color of light you are using,
* whether your eyes has any sight problem,
* am I wearing any spectacles, contact lens or have another object near my eyes that makes me to see and understand something as anything and anything as something and nothing as a thing and a thing as nothing,
* what kind of measurement you are using the tallness or shortness,
* from where you are measuring the tallness or shortness,
* what kind of surface, medium, geographical time, geographical place, evolution period, you are standing, while looking at them to measure.
You keep on testing with many more questions by learning. Might be with comparison you would have had few questions, but would it have been as learning (TESTING)? Each question gives questions, but comparison does give another comparisons and for how long? If you are able to get many such comparisons, they you are not comparing you are learning from your questions.
Since NASA launched Voyager 1 space probe, does it mean that any other scientific research body should not launch or study or work on launch of their probes? If others tried to launch their space probe, will they just compare to Voyager 1? Or will they learn Voyager 1 details and other details that were obtained during that mission and their context? Learning (Testing) gives comparison details too, but comparison might not give the learning that you will be missing if you did comparison at all.
If your kidney is being transplanted to other person, will they just compare the sex of the other person to yours? Won't they learn other person's chemistry, biochemistry and other needed details? Since the person who is receiving your kidney is new to your kidney, is there no need for physicians, surgeons, biochemists, and other staff to learn the recipient and donor? Does people who operate donor and recipient, will take donor physiology as a baseline? Or do they take the recipient physiology as a baseline? They don't assume, but might they have plans and skills to execute.
Ok, why do they have to compare? You answer this question for yourself. And, why they want to learn; answer this question yourself. What will you gain without comparing? What will you loose without comparing? What will you gain by learning? What will you loose by learning?
Don't underestimate anything saying it is 'old', 'new', 'alpha', 'beta' or whatever words you know or you come across. If you do, you are in big trouble and in the ruin which will not be a COMPARISON but learning. If some other says the same word just comparing what you did, learning is underestimated.
Lessons for me from Rapid Software Testing:
Consistency (“this agrees with that”)
an important theme in oracles
* History: The present version of the system is consistent with past versions of it.
* Image: The system is consistent with an image that the organization wants to project.
* Comparable Products: The system is consistent with comparable systems.
* Claims: The system is consistent with what important people say it’s supposed to be.
* Users’ Expectations: The system is consistent with what users want.
* Product: Each element of the system is consistent with comparable elements in the same system.
* Purpose: The system is consistent with its purposes, both explicit and implicit.
* Statutes: The system is consistent with applicable laws.
* Familiarity: The system is not consistent with the pattern of any familiar problem.
Consistency heuristics rely on the quality of your models of the product and its context.
Don't know what it brings for you. For me it is a lessons, which I am learning.
In the above case while testing product Y, do I still have to do comparison? Or I can just take product X as my baseline and explore product Y.
If I were in your context, I will learn the product X. I don't take product X as baseline. I will learn and test product Y. I will try to get details as much as possible from X as well as from Y. Instead of evaluating and providing proof, will provide the details what was required learning and using X and Y. I will know what my mission is and clarify my mission and what they need from me. I wish to see the cost and value for each and everything, which is indeed a LEARNING and not comparison.
* I see them taller or shorter when they are dead or alive,
* whether I am alive or dead when I seeing them as a taller or shorter,
* whether they are floating in dead sea,
Comparison is more or like Zombie process, though you learn it is still dead. And learning is like alive and exploring.
Hi Amit.
Comparison Testing is a heuristic method. It doesn't provide a guaranteed coverage or formal explanation but it does give you an indication that something (possibly) is wrong. As many other methods in tester's arsenal Comparison Testing has its niche:
- when no or very limited formal requirements available
- when applying it quickly reveals areas of possible problems.
Important thing to mention: results, received through Comparison Testing do not indicate a specific defect. Investigation using other methods is required in order to describe the problem.
With the example you used, if newer product (Y) is supposed to produce exactly the same results as older product (X) in a form like reports, tables, etc. then text-file comparison testing will help to find out quickly if results do not match - and that will give an idea where manual investigation should start.
By the way, in case of the large volume of report-capture-and-comparison-test tasks automation gives a great benefit performing all the routine jobs. I have a few posts in my blog dedicated to text comparison and visual reporting in a way similar to WDIFF utility.
Thank you,
Albert Gareev
I really appreciate your thoughts and to make it very simple here is a simple example.
Lets assume that I have to test "Search Engines"
For instance:
Google
Bing
As we all know "Google" is there for quite some time but "Bing" has just been launched by MS.
So if people start saying that one feature which is there in "Google" is not there in "Bing".
In this case we can explore "Google" first and learn how it works. Once we are done with that then we will move onto "Bing".
As a user, Can I still cry that one feature which is present in "Google" (for which I am used to) its not present in "Bing". I think I do not have to.
I see people are comparing "Google" and "Bing" which I feel not appropriate. "Google" is Google and "Bing" is Bing. If "Google" serve the purpose for which I am using it then I will stick to it....but if I feel "Bing" some additional features then I can use "Bing" that time...but ofcourse no comparison.
Best Regards,
Amit
testing is my passion!!!
http://bugteaser.blogspot.com
Hi Amit,
>> I see people are comparing "Google" and "Bing" which I feel not appropriate. "Google" is Google and "Bing" is Bing. If "Google" serve the purpose for which I am using it then I will stick to it....but if I feel "Bing" some additional features then I can use "Bing" that time...but ofcourse no comparison.
>>
Don't you think you shifted the focus?
First of all, what is inappropriate in comparison of the product you're about to buy? Because when you use google search service you BUY it. Not the last thing to mention is that google is not an "old app" even if bing was introduced later.
Second, comparison testing with search engine example is even more applicable. Performance, relevance, caching, localization...
Whichever combination fits better your personal needs and fulfilled better by one search engine in comparison to another.
Thank you,
Albert Gareev