Product Reviews With and Without Comparisons

There isn’t a single style of review that pleases everyone. Some people demand product comparisons while others don’t. 
 

Question: What do you prefer and why? 

Call me old fashioned (considering my age it’s OK) but I like the in depth style with comparisons, measurements, features explanations and subjective opinions. That style is educational and interesting. I may never buy a certain piece but if I needed something like it, I would gravitate to those that have been reviewed thoroughly. Absent a lot of hard data, a review can quickly be summed up as “ I liked it “ or not. Of course it takes time and resources to do it that way and readers who appreciate it and will pay to support it. I guess what I am describing is the old paid magazine model. I did say I was old.

Okay, as mentioned from other threads

I think it is fair and reasonable to ask for comparisons, just as it is fair and reasonable to present what findings and experiences are at hand without comparisons made - both styles can be interesting, informative, and entertaining

One expects relevant comparisons from a "professional reviewer" or "professional critic" *IF* they offer a ranking or hierarchy of some kind  based on whatever criteria. X is better than Y or different from Y because.

I tend to gravitate to reviews that do have critical comparisons because strengths and weaknesses are relative. It is interesting how one reviewer might stack it as compared to another. I will also look for polar opposite reviews for the same reason and especially the detractors. Paradoxically, that might be an example of positive bias - I look for the worst criticisms and make sure I have considered them.

Realistically no reviewer can cover all the comparisons anyway! Even if they could there is potential for subjective biases such that you would be wise to get a second (or more) opinion , even from doctors. Everybody has their style, approach, and ways of looking at facts and formulating opinions.

It also depends on where you derive "value" and reviews without comparisons are useful if done well.  I value for example the subjective review of say a food critic I have come to feel mirrors my own findings upon sampling the same cuisine and is able to communicate descriptively in a way that I find meaningful, if it conjures up that experience. There does not *have* to be a specific comparison made, like restaurant A serves better food than restaurant B, the comparison with other chefs or restaurants is implied in the absolute descriptions coming from someone experienced in tasting good cuisine . I use this to narrow down *my comparisons of choice* on what might otherwise be an endless list.

IMHO its good to choose to sample, compare and contrast, a variety of styles from different reviewers. The choice to get a second (review) opinion  may complement or contradict. Choices abound and that is a good thing!

Equipment don't live in a vacuum. If a reviewer chooses not to rate absolute performance (how it stands against the benchmarks) my view is that he should at least compare it with a couple of similarly priced products.

I would refrain from using any sort of classification rating (stars or number).

Arthur Salvatore has written a rather harsh but in my view lucid piece on his website criticising what he calls 'Audio Relativism':

http://www.high-endaudio.com/philos.html#Rel

5 hours ago, semente said:

my view is that he should at least compare it with a couple of similarly priced products.

Given how short and fallible audio memory can be, are you suggesting a reviewer should own a couple additional similarly priced products for each component he reviews, or ask manufacturers to be tangentially involved in a review for the purposes of comparing their products to the main subject of a review (at their own expense)?

1 hour ago, The Computer Audiophile said:

Given how short and fallible audio memory can be...

That is absurd. Do you mean to say that you can't recognize the voice of a family member or a friend on the other end of a phone line?

43 minutes ago, Tam Lin said:

That is absurd. Do you mean to say that you can't recognize the voice of a family member or a friend on the other end of a phone line?

Hello @Tam Lin, I don't mind you calling my statement absurd as I'm guessing you aren't familiar with the difference between echoic memory and pattern recognition. I certainly wasn't until @Jud educated me. I highly recommend reading some of his posts and linked articles / papers on the subject. 

4 hours ago, The Computer Audiophile said:

Given how short and fallible audio memory can be, are you suggesting a reviewer should own a couple additional similarly priced products for each component he reviews, or ask manufacturers to be tangentially involved in a review for the purposes of comparing their products to the main subject of a review (at their own expense)?

I'm trying to find usefulness in reviews. If that is what it takes then, yes, a couple of affordable benchmark products might be a good ideia. Maybe the reviewer can borrow/rent these, maybe buy them at accomodating prices. Doesn't have to be insultingly expensive gear which is unlikely to be better anyway.

For electronics, panel blind listening sessions would be another thing to explore.

And measurements should accompany every review.

Start with a technical description of the equipment, not the sales blabber but an objective assessment of the design and being the case innovations.

Proceed to an (observation-driven) objective listening report complemented by a (preference-driven) tasting session.

Compare objective listening with references and correlate with measurements.

Et voilà.

23 minutes ago, semente said:

 

I'm trying to find usefulness in reviews. If that is what it takes then, yes, a couple of affordable benchmark products might be a good ideia. Maybe the reviewer can borrow/rent these, maybe buy them at accomodating prices. Doesn't have to be insultingly expensive gear which is unlikely to be better anyway.

For electronics, panel blind listening sessions would be another thing to explore.

And measurements should accompany every review.

Start with a technical description of the equipment, not the sales blabber but an objective assessment of the design and being the case innovations.

Proceed to an (observation-driven) objective listening report complemented by a (preference-driven) tasting session.

Compare objective listening with references and correlate with measurements.

Et voilà.

Thanks for the information on how you think things should work. Specifics are always helpful when discussing this stuff. 

Let's take a look at products I'm currently listening to, in order to write a review, the Wilson Audio Alida CSC ($17,650 per pair) and Wilson Audio Mezzo CSC center channel ($21,250). I have eight Alidas on the wall and ceiling, and a single 200 lbs. center channel. 

You initially suggested that I'd need a couple similarly priced products for comparison. Your follow up seems to suggest there are "affordable benchmark products" that should be compared. I'm a bit confused about these seemingly incongruent suggestions. Are the "affordable benchmark products" in addition to "similarly priced products?"

Which products should I use to satisfy your suggested review procedure? 

Quote

Maybe the reviewer can borrow/rent these

I've never seen high end audio rentals, perhaps you have? Borrowing 8 surround speakers and a center channel would require at least two saintly people because I'd need a couple different options for comparison and they'd have to accept the risk of damage along with time the products can't be used, shipping, etc...

Quote

maybe buy them at accomodating prices.

Perhaps I'm on the wrong end of the high end audio publishing business, but I don't know of any person or business in this industry who can afford two sets of 8 speakers and 2 center channels, to be used for comparison in a review. Even at accommodation pricing, we are talking about a six figure expenditure. 

Quote

Doesn't have to be insultingly expensive gear which is unlikely to be better anyway.

Inexpensive gear is usually available for demo or money back guarantee listening periods and is often less stressful to take a chance on when purchasing. Perhaps this suggestion is about the gear that should be borrowed or purchased to be used in comparison. I can't imagine purchasing 16 speakers and 2 center channel speakers, for the review comparison, and not getting a barrage of comments about the selections being inadequate for comparison. 

Quote

For electronics, panel blind listening sessions would be another thing to explore.

Sounds like fun, once in a while. I did this at Schiit Audio. The company went through a lot of work to setup the listening session and had access to all the gear because it manufacturers everything that was used. Circling back to the previous suggestions, reviewers still need to get the gear for such a blind session. If borrowing it from a manufacturer, the introduction of a blind listening session is a variable that will make every manufacturer uncomfortable. Setting up a test is hard to do right and things out of their control can go wrong, and have nothing to do with the product or how it sounds. Plus, these products are about more than sound quality. I;m not sure I can blind test a company's ability to support its products. 

Quote

And measurements should accompany every review.

Certainly a laudable goal if you derive value from measurements. I do not. 

Quote

Start with a technical description of the equipment, not the sales blabber but an objective assessment of the design and being the case innovations.

Proceed to an (observation-driven) objective listening report complemented by a (preference-driven) tasting session.

Compare objective listening with references and correlate with measurements.

As someone who writes and enjoys creativity, I would lose all interest in writing if I used such a formula. I know other writers who'd love a formula that enables them to fill-in an outline created for them. Nothing wrong or right about either approach. 

I'm unsure what an "(observation-driven) objective listening report" could possibly be, but I respect that you know and others must as well. I'm out of the loop.

I'm also unsure about "Compare objective listening with..." Again, I'm out of the loop.

Since almost everything that gets reviewed is very good, reviews without comparisons are not helpful for a person trying to decide which of several competitors to buy or demo.  We already know that the product is good, so yawn

3 hours ago, The Computer Audiophile said:

I'm guessing you aren't familiar with the difference between echoic memory and pattern recognition.

49 minutes ago, The Computer Audiophile said:

 

Let's take a look at products I'm currently listening to, in order to write a review, the Wilson Audio Alida CSC ($17,650 per pair) and Wilson Audio Mezzo CSC center channel ($21,250). I have eight Alidas on the wall and ceiling, and a single 200 lbs. center channel. 

These points are straw men.  On the memory--we do not need a note by note dissection.  And if the differences are so close it's hard to remember which was better at what, then that's a perfectly acceptable answer.

On the products you're "currently listening to"--you've listed esoteric products, of course you do not have comparisons.  But that does not mean you don't have the ability to do comparisons on your DACs, sources, and other more common purchases

6 minutes ago, PeterG said:

These points are straw men.  On the memory--we do not need a note by note dissection.  And if the differences are so close it's hard to remember which was better at what, then that's a perfectly acceptable answer.

I'm seriously lost. 

Jud posted how echoic memory works, and you're asking for a comparison based on memory that actually doesn't work like many people think it works?  Please help me understand how this is a straw man. It's actually really life, how things work. 

6 minutes ago, PeterG said:

On the products you're "currently listening to"--you've listed esoteric products, of course you do not have comparisons.  But that does not mean you don't have the ability to do comparisons on your DACs, sources, and other more common purchases

I have a $40,000 DAC coming in for review. Please help me obtain at least two other DACs in the price range (per the suggestion), help me research the companies, pay for shipping, use your relationships to talk companies into sending a very expensive guinea pig for comparison purposes only, etc.... Then, do it for several other products, over 17 years, and see which manufacturers get tired of sending guinea pigs. 

Sound quality is but one piece of a product as well. 

7 minutes ago, PeterG said:

Since almost everything that gets reviewed is very good, reviews without comparisons are not helpful for a person trying to decide which of several competitors to buy or demo.  We already know that the product is good, so yawn

You previously said you appreciated two of my reviews for containing a reference to how the products hold up versus others. Thank you. However, your success rate is 50% because one didn't work out. Is this better or worse than guessing or better or worse than if my reviews had no such comparison?

I believe you are searching for something that no review can give you, whether it contains comparisons or not. Any review that tells you something is better than something else, must have tons of ancillary comments surrounding that determination because it's all relative to the situation and person. A review that says, for example, one DAC has more bass than another, tells you how the two DACs perform in a single system / room to one person's ears. If you have the same system and room, then you are off to a good start, never mind the same ears.

21 minutes ago, PeterG said:

Since almost everything that gets reviewed is very good

This is a topic that could use its own thread. One person's good is another's nightmare. I've turned down many products because I don't believe they are any good and I think the companies will screw over members of this community (and others). 

43 minutes ago, The Computer Audiophile said:

I'm seriously lost. 

 

Jud posted how echoic memory works, and you're asking for a comparison based on memory that actually doesn't work like many people think it works?  Please help me understand how this is a straw man. It's actually really life, how things work. 

I do not accept the whole echoic memory thing.  It is one of several canards that some (not necessarily Jud) use to explain why their $200 DAC is as good as a dCS.  I am able to remember if one product has better bass or highs, or mids, or imaging, or whatever, than another.  I have confidence that you do as well.

47 minutes ago, The Computer Audiophile said:

 

I have a $40,000 DAC coming in for review. Please help me obtain at least two other DACs in the price range (per the suggestion), help me research the companies, pay for shipping, use your relationships to talk companies into sending a very expensive guinea pig for comparison purposes only, etc.... Then, do it for several other products, over 17 years, and see which manufacturers get tired of sending guinea pigs. 

You have several DACs on hand (including a $30K EMM?) That's good enough for me.

49 minutes ago, The Computer Audiophile said:

You previously said you appreciated two of my reviews for containing a reference to how the products hold up versus others. Thank you. However, your success rate is 50% because one didn't work out. Is this better or worse than guessing or better or worse than if my reviews had no such comparison?

 

I believe you are searching for something that no review can give you, whether it contains comparisons or not. Any review that tells you something is better than something else, must have tons of ancillary comments surrounding that determination because it's all relative to the situation and person. A review that says, for example, one DAC has more bass than another, tells you how the two DACs perform in a single system / room to one person's ears. If you have the same system and room, then you are off to a good start, never mind the same ears.

I just mentioned two that I remember because they led to purchases.  I have appreciated a large number of your reviews.  I'm glad I tried the RAALs, they were a revelation, as I think you wrote, and I did not mind eating the restocking fee after deciding they were too revelatory for me.

49 minutes ago, The Computer Audiophile said:

This is a topic that could use its own thread. One person's good is another's nightmare. I've turned down many products because I don't believe they are any good and I think the companies will screw over members of this community (and others). 

This is exactly what I mean--bad products do not get reviewed.  The challenge for a consumer is separating the great from the very good

I am confused as to why it appears you no longer like the idea of comparisons.  Strange, especially after years of doing them. 

1 minute ago, PeterG said:

I do not accept the whole echoic memory thing.

Understood. No sense in us going back and forth about it then :~)

2 minutes ago, PeterG said:

You have several DACs on hand (including a $30K EMM?) That's good enough for me.

I'm happy you're OK with that :~)

4 minutes ago, PeterG said:

deciding they were too revelatory for me.

I love that phrase. 

4 minutes ago, PeterG said:

This is exactly what I mean--bad products do not get reviewed.  The challenge for a consumer is separating the great from the very good

I'm with you on this. It is a challenge to which I don't see a solution. It's tough to infer than unreviewed products aren't great or that frequently reviewed products are great. For me it comes down to people. If I connect with a writer or a person's style and I believe them to be honest, then I'm onboard. I'll take what makes sense to me, from an article, and leave the rest. 

I believe the Stereophile recommended components list, with classes of components (A, B, C...), is the biggest disservice to the industry and consumers, there ever was. Ranking components reviewed over many years by many writers under many different circumstances, is a fool's errand. It's no different than the US News college rankings. They are pure gold for the publication and pure heroin for young adults seeking more information and believing what they are reading. 

15 minutes ago, PeterG said:

I am confused as to why it appears you no longer like the idea of comparisons.  Strange, especially after years of doing them. 

I can see why you wrote that. I'm just pushing back at a part of renewing that is fraught with issues and seen as a gold standard when I see it very differently. I love that some writers here on AS like to compare things and readers love to read the comparisons. I like chocolate and they like vanilla. 

If I compare something I do it very differently from a stereotypical A/B. I listen to music all day every day, and spend many hours with every component about which I write. When I write, I sit down and the words come out. If a comparison comes out, and I'm feeling it / it just seems correct, then I go with it. In the review I posted this morning, I say the new Rendu is the best one ever. I suppose this could be seen as a comparison with all those that came before it, however I'm sure this isn't good enough for many people (not you specifically).

2 hours ago, Jud said:

The difficulty here, for me, is that I don’t see how echoic memory gets better when listening blind. So how deep a comparison can you make in 4 to 10 seconds? Can you switch that fast? Even if you can, if the brief comparison passage gets louder, then the volume drops as you bring in the second unit, making it sound worse. Vice versa if the passage gets softer.

 

There’s really no wonder that blind listening tests seldom if ever find consistent differences. There are some blind tests that do, such as carbon fiber violins vs. Stradivarius. But lo and behold, the carbon fiber violins that are preferred are louder. Mystery solved.

I was thinking of week-long listening in listener's system against references.

16 hours ago, Tam Lin said:

That is absurd. Do you mean to say that you can't recognize the voice of a family member or a friend on the other end of a phone line?

15 hours ago, The Computer Audiophile said:

Hello @Tam Lin, I don't mind you calling my statement absurd as I'm guessing you aren't familiar with the difference between echoic memory and pattern recognition.

very basically memory can be viewed

As types

    Explicit memory

    Implicit memory

As stages

   Sensory memory (echoic for sound, iconic for vision)

   Short-term memory

   Long-term memory

As processes

   Encoding

   Storage

   Retrieval

Unsurprisingly, memory for "sound systems" has not been well studied AFAIK so we draw from other models like music recognition/memory, voice recognition/memory, familiar sounds recognition/memory. Either way, Echoic sensory memory only lasts seconds, so is not relevant to long term memory recall.

Long term memory involved in familiar voices has been well studied and generally seen as a type of explicit episodic memory event involving pattern recognition based on various different cues or voice characteristics. One study showed that just two words were sufficient to recognize a familiar voice. Interestingly, the cues involved are not set in concrete and do not show a linear relationship - any one of a number of cues can be operational at different times.

Like other perceptual patterns like facial recognition/memory, higher cognitive processing is involved sorting the different cues that may be on offer - top down processing.

Remember (<--pun), lots of other stuff can interfere with recall as everyone knows. One interesting thing from voice studies is that hearing similar sounding voices can actually impair the ability to recognize/recall the familiar voice in question (voice mimicry studies). The top down processing gets muddled. I find this easy to understand how ABX perceptual tests can get quickly confusing. Perhaps why I hate eyeglasses testing done by optometrists. Even the order that voices are presented can affect the perceived similarity or differences between them. It seems the perceptual set of cues used for the first voice can potentially interfere with what is available for the next.

Suffice to say, it is an entirely different proposition to recognize a very familiar voice presented to you in real time than to recall the sound of a DAC heard months ago.

9 hours ago, Jud said:

 

There have actually been experiments done where Asian subjects unfamiliar with Western music have been asked to discern musical patterns that Westerners who've been exposed to these patterns all their lives discern readily. It was found that even after a week of 8-hour days of listening, the Asian subjects did not improve in their ability to find the patterns. That's why I said training oneself to achieve pattern recognition is likely a matter of hundreds of hours at least.

Surely everything sounds the same...

12 hours ago, Jud said:

The difficulty here, for me, is that I don’t see how echoic memory gets better when listening blind. So how deep a comparison can you make in 4 to 10 seconds? Can you switch that fast? Even if you can, if the brief comparison passage gets louder, then the volume drops as you bring in the second unit, making it sound worse. Vice versa if the passage gets softer.

 

There’s really no wonder that blind listening tests seldom if ever find consistent differences. There are some blind tests that do, such as carbon fiber violins vs. Stradivarius. But lo and behold, the carbon fiber violins that are preferred are louder. Mystery solved.

It's not about echoic memory but identifying issues with the playback. Remember your listening test at the AP demo?