Can BETA-test sites be trusted to provide 100% independent and objective reviews ?

[DR]

The purpose of BETA-testers is to specifically help manufacturers make products better.

The principle of BETA testing is a good thing, and manufacturers should do more of it.

However: -

BETA-testers provide manufacturers with recommendations on how to make a product sell better, make more money and also how a manufacturer can save money.

Do you entirely trust BETA-testers to provide you with 100% independent and objective comparative buying advice on the products they BETA-test?

Do you entirely trust BETA-testers to provide you with 100% independent and objective comparative buying advice on the competing alternatives of products they BETA-test?

Would it make a difference if the primary source of income for a publication came from the manufacturer for which it was BETA-testing?

If you found out you’d been hoodwinked into believing that buying advice was independent and objective - when it was not - how would you feel?

How would you feel about the ethics of a manufacturer which encouraged its BETA-testers to publish reviews and buying advice of its products, and its competitor’s products?

This is a very important subject and your opinion is very important.

It could well affect the honesty and quality of buying advice and editorial you read in the future.

Before you post your thoughts, please vote in the poll above

[Moby-Dick]

good question , but what if tester were testing not only the product , but its competitors as well ,would that give a more objective opinion.

should testing only be done by independant testers who have nothign to gain by quality of the test ?

[PD HEXUS]

Originally Posted by Moby-Dick

good question...

indeed

Originally Posted by Moby-Dick

...but what if tester were testing not only the product , but its competitors as well ,would that give a more objective opinion.

how do you think BETA-testers can be trusted to provide any independent and objective comparative buying advice on products they've specifically tried to help manufacturers make better?

Originally Posted by Moby-Dick

should testing only be done by independant testers who have nothign to gain by quality of the test ?

i've thought it through, and can't see how, ethically, it can be done any other way...

but that's one of the points of the poll and this thread Moby - what's your take ?

cheers,

PD

[directhex]

you should never *review* beta hard/software. things are all too subject to change. but i don't see any harm in offering previews, based upon experience with a beta product

what's important, though, is not to let the previewer also be the reviewer - it's far too easy to get emotive if the beta and final don't match, and slant scores.

[Moby-Dick]

If you seperate QA testing and public beta testing - then you might have the best of both worlds ?

If the QA phase was carried out in house , you'd know that any informatino published as a result of that testing would potentially be slewed.
Even if a beta tester had nothing to gain , I dont think they'd be able to sit on the fence when it came to attempting to improve 2 competing products. So reviews / advice based on that testing should carry informatino to that effect ?

[PD HEXUS]

Originally Posted by directhex

you should never *review* beta hard/software. things are all too subject to change.

agreed - and over the weekend this was a point well made by HEXUS.lifestyle Editor Bob Crabtree.

Originally Posted by directhex

but i don't see any harm in offering previews, based upon experience with a beta product

agreed.

and that's not the same as BETA-testers specifically offering editorial that's dressed up as independent and objective comparative buying advice, on the very products they've BETA-tested to specifically try to help manufacturers make better.

Originally Posted by directhex

what's important, though, is not to let the previewer also be the reviewer - it's far too easy to get emotive if the beta and final don't match, and slant scores.

interesting points.

do you think that BETA-testers posing as 'reviewers' are even more likely to "slant scores"?

and would it make a difference if the primary source of income for a publication came from the manufacturer for which it was BETA-testing?

[Moby-Dick]

and would it make a difference if the primary source of income for a publication came from the manufacturer for which it was BETA-testing?

I dont belive it would automatically discredit anything they produce - but you would have to bear it in mind.

[eldren]

*notes all the orange and red in this thread*

It's a rather biased question to ask, from the phrasing used ("masquerade" has negative connetations, for example, as does the "hoodwinked" question) and the 3 no answers vs. 1 yes answer. But maybe that's just me.

Either way, in my opinion, no, it's not okay. But there's a difference between [p]reviewing a beta product, and being a beta-tester. The former case is entirely allowable, I feel, so long as the reviewer has nothing to gain from doing so, but for giving his audience a look at what's to come.

Of course, if a reviewer never told anyone about being a beta-tester, no one would ever know, in the same way that if a company pays off a reviewer, no one would ever know. In both cases, there's opportunity for bias and downright corruption (Of course, accepting money for a review is already pretty corrupt). And I'd be happier, as a reader, knowing that the opportunity doesn't exist in the first place.

(Side note, I voted option 3 by accident, saw it was 3: 3 votes, and 4: 4 votes, clicked back, set 4, and tried again, this time with it being 3: 3 votes and 4: 5 votes. So not too sure if I voted twice, or if it was actually changed.)

[Shorty]

I think it's too subjective question to actually select a poll option.

A beta tester can still be a reviewer as long as they have the integrity and publish an honest review rather being slanted into publishing what makes the manufacturer happier.

An astounding number of writers and reviewers of Windows 2003 server were beta-testers. That's the very nature of the product.

The difference between the reviews & reviewers is those that felt they had to give a favourable review because it's Microsoft and they beta tested it.. and those who were completely honest and beta tested it.

It's more down to the publications integrity & reviewer honesty than a hard & fast rule of yes/no.

[kalniel]

Since when was reviewing all about

Originally Posted by Hexus

objective comparative buying advice

?

Beta testers will have access to a product, and presumably know a lot about it, thus may be the most informed to give out information. I'm glad just to get the information in a review.

Beta testers do not always work for the company of the product they are testing, and there is not always (or even usually in some cases) any incentive to falsely promote a product in a 'review'.

Finally Hexus itself is happy to give opinions during reviews - why should everyone else have to be objective when even Hexus isn't? I'm glad for subjective reviews sometimes - it's obvious enough if someone isn't trustworthy and then it's just the case that no-one listens to their reviews.

I'm disappointed hexus would chose to weigh such a discussion by using such language as 'masquerading as reviewers'. Just present to us the content, but also give us (factual) information about the reviewer so we can make our own minds up.

[PD HEXUS]

hi kalniel,

Originally Posted by kalniel

I'm disappointed hexus would chose to weigh such a discussion by using such language as 'masquerading as reviewers'. Just present to us the content, but also give us (factual) information about the reviewer so we can make our own minds up.

hm... ok, lets tweak the poll question a bit...

how about:

Can BETA-test sites be trusted to provide 100% independent and objective reviews ?

?

cheers,

PD

[eldren]

Shorty - You raise some good points, but the problem is establishing that a reviewer is honest and has integrity in the first place. Everyone represents themselves as being honest and accurate, since that's best for business, so as a reader, it becomes difficult to tell who really is honest, and who isn't.

As I see it, there could be three categories here:
1) Reviewers who say they don't/neglect to mention that they beta-test, but do
2) Reviewers who say they do beta-test, but assure everyone that it won't affect reviews
3) Reviewers who say they don't beta-test, and don't

For me, category 2 is potentially susceptible to corruption, which unfortunately puts anyone in that category in a bad light, since I'll never be 100% certain that the review is objective. Of course, I can't really tell the difference between 1 and 3 unless something gives it away (As has been the case with one or two other review sites, in terms of paid reviews).

[Shorty]

Eldren,

I agree with you wholeheartedly. Unless you know the publications "people" personally, there is no real assurance of honesty. That goes for any publication on any subject. They are may seem sincere but.. you know how it goes.

I still insist that whether someone beta tests a product is not a factor in whether they can fairly review it (im not a beta tester Id just to point out!). It's down to the person reviewing to provide an accurate analysis regardless of whether they tinkered with it in it's early days or now.

[geo]

Gosh, I've got the strongest feeling there is something y'all aren't telling us here. You didn't wake up this morning, stare at your cheerios, and "discover" this question. So, give --why this now?

Also, "100%" objective seems a bit like putting the thumb on the scale to me to get an answer, and the options weighted as well.

So, come on fellas, give out where you're coming from on this one. . .

[kalniel]

Thanks PD. Appreciate it.

[ikonia]

I've found pretty much with every opensource / public beta style application, the beta testers are %100 objective and honest, to the point where I have seen beta's pulled and a whole product revision dropped and re-wrote.

I would assume the same with companies like microsoft (I use microsoft as an easy example) simpley because by the time they do a "public" beta all the really serious problems should be gone, and kept internally, however if something bad is found I'd expect to read between the lines as companies will only release as positive spin on a major problem.

I think people are pretty open with beta's though as a general rule of thumb