Great discussion on how reliability data works and where it can fall short on being useful

Fightnfire said:
That's basically it, plus it struggles with confirmation bias and the average consumer wants to defend what they have purchased, which is why the more a vehicle sells the more popular it is and the more positive ratings it gets as shown in the video. Plus, they seem to reward the vanilla and hold on to issues long solved by specific manufacturers. They also ignore recalls...
Click to expand...

I would disagree with both statements. Those statements are the definition of confirmation biases. You don't agree with CR results; therefore, their methodology and results must be wrong.

I understand they are not the only source for reliability, but they have the greatest amount of data than any other source. It's like looking at an Amazon product with 5k reviews vs one with less than a 100. And the fact that they disregard a brand or model due to lack of data is a good thing. That means insufficient data won't change the numbers.

All they do is send out surveys with very simple questions: Did you need to have something repaired in the last year? What system was it? That's pretty much it. Can respondents lie? Of course, but are we going to say more Subaru respondents lied than Ram respondents? And if you drill down into their data, they reveal that only 1% of respondents have "powertrain" issues, 3% have "infotainment" issues, etc. Meaning the brands are really that close together by actual amounts.

Do I wish they drilled down deeper into specific model configurations? Yep, but then they may not have enough results to truly say the Ford 5.0 is more reliable than the 3.5 when there are so many more 3.5s than 5.0s out there. So, everyone would tear them apart for it.

And watching CR results vary over the years makes me think it is more legit, not less. The fact that Honda was always the most reliable brand, then Toyota, and now it's Subaru.

I don't care as much for their reviews because they don't have the same values that I do or needs that I do, but I find it hard to argue with their reliability rankings. As long as you understand the difference between #1 and #10 may not be that far apart.
 
testerdahl said:
I’ve had some offline conversations with them over this exact issue.

My big complaint is there scoring system and lack of detailed information. On the scoring system, trucks always rank with the a score of 50/100. They really only have one car that gets close to 100. I’m ask them what’s the point of a 100 point scale if only one vehicle comes close?? They also seemingly only like full-size sedans that haven’t been updated in a decade and are about to be discontinued. How is that helpful?

For detailed information, I want to know what engines people are reporting having issues with. They tell me they don’t have that information. I call BS. That would be a requirement for me on any survey I’d send out. I’d want make, model, year, engine, transmission, and then ask questions on a host of areas.
Click to expand...

Yes to all, especially the engine data and only liking 10 year old sedans. Plus, as I mentioned earlier, there's a reason some of the cars show up near the top.

I think trucks struggle with them because they don't have testers that understand what a truck is. That seems to cause them to over value the one with the best ride, or really hurt if it doesn't. As truck people we know that's not apples to apples. Which trim?
 
It seems like people are conflating reliability ratings and vehicle reviews. They apply the same methodology for reliability to all vehicles within a category equally.

With reviews that’s their subjective opinions… that’s just useful data to help you make a decision or not… their job isn’t to confirm anyone’s prior opinions or experiences.
 
mwl001 said:
It seems like people are conflating reliability ratings and vehicle reviews. They apply the same methodology for reliability to all vehicles within a category equally.

With reviews that’s their subjective opinions… that’s just useful data to help you make a decision or not… their job isn’t to confirm anyone’s prior opinions or experiences.
Click to expand...

Maybe but not in my case, I'm talking about a review where they compare a Bighorn to a LTZ and then a Limited 1500 and the limited gets the best overall marks and the reasons they list as why are interior options and ride quality. Well no shit. Then they carry that review into their overall reviews of RAM, GM and Ford. Suddenly, Fords have the best ride. The data is limited and skewed.

Like Tim mentioned they also don't give any data on power trains. But they will make comments in the reviews about driving performance.

I remember when we decided we needed a new vacuum for the house and had narrowed it down to two or three options. It was between the Dyson the Shark and I don't remember the last brand. That was a bit before I got heavy into the truck world and I walked away after that whole process thinking how incomplete it was and the things they gave the Dyson high marks for ended up not mattering one bit and definitely didn't justify more than double the price. We returned it, thank you Costco, and went with the shark. Much happier.
 
Fightnfire said:
Maybe but not in my case, I'm talking about a review where they compare a Bighorn to a LTZ and then a Limited 1500 and the limited gets the best overall marks and the reasons they list as why are interior options and ride quality. Well no shit. Then they carry that review into their overall reviews of RAM, GM and Ford. Suddenly, Fords have the best ride. The data is limited and skewed.

Like Tim mentioned they also don't give any data on power trains. But they will make comments in the reviews about driving performance.

I remember when we decided we needed a new vacuum for the house and had narrowed it down to two or three options. It was between the Dyson the Shark and I don't remember the last brand. That was a bit before I got heavy into the truck world and I walked away after that whole process thinking how incomplete it was and the things they gave the Dyson high marks for ended up not mattering one bit and definitely didn't justify more than double the price. We returned it, thank you Costco, and went with the shark. Much happier.
Click to expand...
That’s a review not an assessment or evaluation of reliability… like I said feel free to disagree or write your own review but there’s nothing useful in pointing that out. Just find a publication you agree with and read that one. Reliability data is based on surveys not editorial decisions and it’s the closest thing to actual reliability out there.
 
It's an assessment or evaluation of performance across the product line. It was not a singular item review.

Oh, and if you're talking about the trucks it was a comparison review across the three and if they don't understand the differences in those three trims then you have to wonder about their understanding of the product line in general including when they compare their reliability rankings and provide no power train data.
 
Last edited:
Also, I'm not trying to hurt your feelings about CR I've already stated it's a data point amongst many. I just wish there was more transparency in their rankings.
 
Fightnfire said:
It's an assessment or evaluation of performance across the product line. It was not a singular item review.

Oh, and if you're talking about the trucks it was a comparison review across the three and if they don't understand the differences in those three trims then you have to wonder about their understanding of the product line in general including when they compare their reliability rankings and provide no power train data.
Click to expand...
Who does provide power train reliability data? I’m just trying to understand who is doing a better job on reliability, even though they’re not perfect. I honestly don’t care how or why they rank subjective things like ride or interior. I’m an adult I make those decisions for myself AND they’re entitled to their own opinions and methods.
 
mwl001 said:
Fair enough. I just think there’s a difference between lack of transparency or the data not there to be able to share with members.
Click to expand...

Yeah that's where you and I differ and what I take issue with... with the amount of pickup trucks sold in this country every year there's no way they don't have the data available to break it down by powertrain. For some reason they're choosing not to and I can't help but think that behavior trickles into their other reliability rankings.
 
Last edited:
Fightnfire said:
Yeah that's where you and I differ and what I take issue with... with the amount of pickup trucks sold in this country every year there's no way they don't have the data available to break it down by powertrain. For some reason they're choosing not to and I can't help but think that behavior trickles into their other reliability rankings.
Click to expand...
They don’t have data on all the pickups sold every year, only on the trucks members buy who also fill out the survey. If I remember the video correctly they don’t collect enough data on Stellantis vehicles to include model-level data in the rankings. I can’t prove or disprove a negative but the only reason for them to withhold useful data would be that they are being paid not to.
 
mwl001 said:
They don’t have data on all the pickups sold every year, only on the trucks members buy who also fill out the survey.
Click to expand...

I'm glad you said that because that's been the point I've been trying to make all along, that's where the bias comes in.

If you look at the type of people who are willing to purchase a membership to consumer reports you're going to see a personality type. It's also reflective of the type of vehicles those people buy. Hence why some vehicles are so well represented and others are not...which further skews the results.

It's a bit of a self-fulfilling prophecy, with bias.

To clarify, I'm not accusing CR of being intentionally biased or deceptive. It's just built in to their process and works as a single point of data but it shouldn't be the only point. I do think they're very quick, too quick, to hand out recommendations with not nearly enough data or actual use.
 
Last edited:
Fightnfire said:
I'm glad you said that because that's been the point I've been trying to make all along, that's where the bias comes in.

If you look at the type of people who are willing to purchase a membership to consumer reports you're going to see a personality type. It's also reflective of the type of vehicles those people buy. Hence why some vehicles are so well represented and others are not...which further skews the results.

It's a bit of a self-fulfilling prophecy, with bias.

To clarify, I'm not accusing CR of being intentionally biased or deceptive. It's just built in to their process and works as a single point of data but it shouldn't be the only point. I do think they're very quick, too quick, to hand out recommendations with not nearly enough data or actual use.
Click to expand...
Alex explained the same point in the video where CR is just a self fulfilling prophecy with less and less data available each year as they loose subscribers.

CR is a take it with a heavy grain of salt type of review when it comes to anything with an engine. Alex, Tim and @Fightnfire just explained why that is. There's never only one source of truth our there, read up and built your opinion over time, fact check everything and keep your biases in mind.
 
You must log in or register to reply here.
Back
Top