How Does CR Get Its Reliability Information?

Where Is the Data From?
Consumer Reports obtains its reliability data from an Annual Questionnaire that is sent to members of CR.org and Consumer Reports magazine. In all, we received responses on over 500,000 vehicles in our latest survey, detailing 2000 to 2018 models.


How Is the Survey Conducted?
The Consumer Reports National Research Center conducts the survey each year. In the questionnaire, we ask members to note any problems with their cars that occurred in the past 12 months. They are asked to identify problems that they considered serious (because of cost, failure, safety, or downtime). We ask them to include problems covered by warranty, but not the ones resulting from accident damage or due solely to recall. Respondents check off problems from a list of trouble areas, ranging from the engine and transmission to climate system, brakes, electrical system, and power accessories. They also tell us specifically what their experiences were to help us understand precisely what problems they are having. (See the full list of trouble spots below.)

How Current Is the Data?
All our reliability information is completely updated annually. We begin sending out each year's survey in the spring. By late summer, we have collected and organized responses, and we complete our analysis and update the information in print and online by late October.

Scale of Car Reliability Data

How Many Cars Do You Have Information on Overall?
CR's Annual Questionnaire is one of the largest scientific surveys conducted in the United States. Our latest survey, which was sent to members of Consumer Reports magazine and to CR.org members, gave us feedback on their experiences with over 500,000 vehicles. This high number of responses allows CR to provide the most comprehensive reliability information available to consumers.

How Many Samples Do You Have of Each Model?
A typical model has about 200 to 400 samples for each model year. When we have smaller sample sizes than this on vehicles, we use brand history and the reliability of similar models that may share major components in calculating our predictions. Since 2015, we use an online questionnaire exclusively instead of our previous mix of electronic and paper ballots from members. That change shrank our respondent pool, but the Internet-only survey allows us to ask more in-depth questions and solicit detailed comments about problems.

What Effect Does Having a Larger Sample Size for Some Vehicles Compared With Others Have on the Validity of the Reliability Data?
Given an appropriate sample, the more data you have, the more statistical confidence you have in your information. A larger sample will always give more accurate information than a smaller sample (assuming, of course, that the data are valid and collected from an appropriate source).

When we have small sample sizes on vehicles, we use brand history and the reliability of similar models that may share major components. This gives us the ability to predict reliability of brand new vehicles or ones that have been recently redesigned.

What Types of Problems Are Reflected?

Are All Automotive Problems Included?
Respondents to our survey are asked to identify problems they have experienced in a 12-month period in any of 17 trouble spots. We do not publish scores for advanced safety systems and air bags since some problem rates in the area are almost universally very low.

What Do the Trouble Areas Cover?
Our Reliability History charts cover problems in any of 17 trouble areas. Here's a look at what's covered in each of those areas:

ENGINE MAJOR: Engine rebuild or replacement, cylinder head, head gasket, turbocharger or supercharger, timing chain or belt.

ENGINE MINOR: Accessory belts and pulleys, engine computer, engine mounts, engine knock or ping, oil leaks.

ENGINE COOLING: Radiator, cooling fan, water pump, thermostat, antifreeze leaks, overheating.

TRANSMISSION (and clutch) MAJOR: Transmission rebuild or replacement, torque converter, premature clutch replacement.

TRANSMISSION (and clutch) MINOR: Gear selector and linkage, transmission computer, transmission sensor or solenoid, clutch adjustment, rough shifting, slipping transmission, leaks.

DRIVE SYSTEM: Driveshaft or axle, CV joint, differential, transfer case, four-wheel-drive/all-wheel-drive components, driveline vibration, electrical failure, traction control, electronic stability control (ESC).

FUEL SYSTEM/EMISSIONS: Sensors (O2 or oxygen sensor), emission-control devices (includes EGR), engine computer, fuel-injection system, fuel cap, fuel gauge/sender, fuel pump, fuel leaks, stalling or hesitation.

ELECTRICAL SYSTEM: Alternator, starter, hybrid battery and related systems, regular battery, battery cables, engine harness, coil, ignition switch, electronic ignition, distributor or rotor failure, spark plugs and wires failure.

CLIMATE SYSTEM: A/C compressor, blower (fan) motor, condenser, evaporator, heater system, automatic climate system, electrical failure, refrigerant leakage.

SUSPENSION/STEERING: Shocks or struts, ball joints, tie rods, wheel bearings, alignment, steering linkage (includes rack and pinion), power steering (pumps and hoses, leaks), steering wheel vibration, wheel balance, springs or torsion bars, bushings, electronic or air suspension.

BRAKES: Antilock system (ABS), parking brake, master cylinder, calipers, rotors, pulsation or vibration, squeaking, brake failure, premature wear.

EXHAUST: Muffler, pipes, catalytic converter, exhaust manifold, heat shields, leaks.

PAINT/TRIM: Paint (fading, chalking, peeling or cracking), loose trim or moldings, rust.

BODY INTEGRITY (noises & leaks): Squeaks, rattles, wind noises, loose or cracked seals, and/or weather stripping, air and water leaks.

BODY HARDWARE: Windows, locks and latches, doors or sliding doors, tailgate, trunk or hatch, mirrors, seat controls (movement and temperature), seat belts, sunroof, convertible top, glass defect.

POWER EQUIPMENT AND ACCESSORIES: Cruise control, clock, warning lights, body control module, keyless entry, wiper motor or washer, tire pressure monitor, interior or exterior lights, horn, gauges, 12V power plug, USB port, alarm or security system, remote engine start, heated or cooled seats.

IN-CAR ELECTRONICS: CD or DVD players, radio, speakers, in-dash GPS, communication system (e.g., OnStar), display screen freezes or goes blank, phone pairing (e.g. Bluetooth), voice control commands, steering wheel controls, portable music device interface (e.g., iPod/MP3 player), backup or other camera/sensors.

Are All Problems Considered Equally Serious?
Problems with the engine-major, engine cooling, transmission-major, and drive system are more likely to take a car out of service and to be more expensive to repair than the other problem areas. Consequently, we weigh these areas more heavily in our calculations of model year Overall Reliability Verdict. Problems such as broken trim and in-car electronics have a much smaller weight. Problems in any area can be an expense and a bother, though, so we report them all in the Reliability History charts.

What Do the Different CR Reliability Ratings Mean?

What Different Reliability Scores Does CR publish?
Consumer Reports uses the data from its Annual Questionnaire to compile detailed Reliability Histories on several hundred makes and models of cars, minivans, pickups, and sport-utility vehicles, covering the 2000 to 2018 model years. For each model that we have sufficient data, the Reliability History Chart shows you whether the model has had more or fewer problems than the average model of that year in each of 17 trouble spots. That information can be a big help when inspecting and purchasing a used car. The Overall Reliability Verdict summarizes the 17 trouble spots for each model year and compares that to the average of all vehicles in the same model year. We use these Reliability scores to identify lists of Reliable Used Cars and Used Cars to Avoid.

For new models that are currently available, our Predicted Reliability rating is based on the model's recent history, provided the model hasn't been significantly redesigned for the current model year. Online, Predicted Reliability is presented in the new car model Overview and Ratings comparison pages, in car type pages under Reliability and Ratings in the Vehicle Overall Ratings comparison. It is also incorporated into the Reliability History charts as the New Car Prediction in print publications.

We also present Predicted Reliability in more detail in our graphs. This presentation, with bar graphs where we used to show the score as a percentage difference between each model's overall reliability and the average reliability of all models, is scored on a 0 to 100-point scale. We group models by vehicle type (for example, midsized cars or minivans), for ease of comparing models that are direct-market competitors.

What Are the Reliability History Charts?
The chart for an individual model year will tell you where a model's strengths and weaknesses have been. Scores are based on the percentage of survey respondents who reported problems for that trouble spot, compared with the average of all vehicles for that year. Models with problem rates close to the mean receive a are not necessarily problem-free, but they had relatively few problems compared with the average model.

In particular, within each trouble spot and within each model year, we create equal-width intervals for the [
The Overall Reliability Verdict summarizes a model's overall reliability over all 17 trouble spots. Because problems with the engine major, cooling, transmission-major, and drive system can be serious and more expensive to repair, our calculation gives extra weight to problems in these areas. The Reliability scores show whether the model had more or fewer problems overall than the average model of that year.

What Is Predicted Reliability?
The Predicted Reliability, also called New Car Prediction, forecasts how well a new model that is currently on sale is likely to hold up based on its recent history. For this Rating, we average a model's Overall Reliability score for the newest three years, provided the vehicle did not change significantly in that time and hasn't been redesigned for the current model year. Over the years, we have found that several years of data are a better predictor than the most recent model year alone. One or two years of data may be used if the model was redesigned within that three-year time frame, or if there were insufficient data for some years.

We will make a prediction for a brand new or redesigned model, or a model with insufficient data, based on the manufacturer’s track record, history of the previous generation, or similar models that shared the same components. Of course, this is only a prediction, and these scores are not a guarantee of the reliability of any individual car. However, buying a car that has an above-average score for Predicted Reliability will reduce the likelihood of having significant problems with your car.

You can find our Predicted Reliability for new cars in many of Consumer Reports' auto publications, including the April Annual Auto Issue, CR monthly road tests, our special new-car publications and online at CR.org.

How Do You Decide on Reliable Used Cars and Used Cars to Avoid?
Reliable Used Cars are specific models with above average overall reliability, based on the Overall Reliability score for that model. We also compile a Best of the best list, CR Good Bets, which are models that have had consistently better-than-average reliability for multiple years, and performed well in CR's tests when they were new. Note that just because a model is not listed as a Reliable Used Car or a CR Good Bet does not mean that it is necessarily unreliable—it may be the case that we do not have sufficient data to assess its reliability, or that we do not have a recent enough road test.

How Does the Reliability Rating Impact Recommendations?

CR-tested vehicles are ranked by the Consumer Reports Overall Score. The Reliability Rating and the Road Test Score are major components. Crash Safety, crash avoidance technologies, and owner satisfaction are also factors. The vehicles with the highest Overall Score that met CR’s criteria in their respective categories are recommended.

How Accurate Is CR's Reliability Information?

Is This a Scientific Survey?
There are generally two criteria that social scientists use to evaluate the quality of a survey: its validity and its reliability. Validity refers to whether the survey actually measures what it says it does. Reliability refers to whether the information generated by the survey would be repeated if the survey were to be conducted again.

We have strong evidence that our survey is both valid and reliable. The questions in the survey are designed professionally by experts in CR's National Research Center, in consultation with our automotive engineers and statisticians. Members of our survey team have advanced degrees and many years of experience in conducting all sorts of consumer surveys. The survey uses an aided response technique that leads respondents through well-defined specific items and gives each respondent the same perspective in answering the questions. The data we report tracks well with other sources of repair and reliability information available on the market. From year to year, our members' reports of their problem experiences are fairly consistent; when there is a difference on a particular model, we can often attribute it to known issues with a particular component of a car.

Is the Survey Based on a Representative Sample?
A sample is considered to be representative of a population if the relevant characteristics of the population are reflected in the sample. So, considering the population of interest is critical in evaluating the quality of a sample. Our survey sample is drawn from the population of CR.org and Consumer Reports magazine members. While all members are invited to participate in the survey, participation is voluntary, and there is always the possibility that those who respond are unique in some particular way. For example, members have sometimes questioned whether those who respond are those who have a complaint to make about their cars.

One reason for this is that our survey is an omnibus survey asking members not only about their cars, but about a dozen other products, about major services they have used (such as insurance, hotels, and health plans). Many members return surveys reporting that they had no problems at all with their cars in the past year. This is true for all makes and models of cars. So, owners with complaints about their cars are not the only ones who return the survey.

Any survey has some sort of sampling frame that limits the people being surveyed. We choose our members as our sampling frame. On average, CR members tend to be more educated and affluent than the general population. With the growth of Consumer Reports online, a wider demographic range of individuals has been surveyed in recent years. However, our reliability questions do not ask respondents about their attitudes or opinions about the reliability of their cars, where one might expect different groups of individuals to have different perspectives. Instead, we ask for factual information about whether specifically defined problems occurred; these types of questions are less sensitive to the nature of the characteristics of the sample itself.

Further, our results track well with other sources of reliability information available on the market.

Is the Survey Biased Toward Japanese Cars?
In our survey of CR members, Japanese vehicles are popular. Also, many Japanese models have had relatively low rates of problems in our survey. But the fact that we received responses on more than 300 makes and models from nearly all domestic, European, and Korean manufacturers shows that our members do not exclusively favor Japanese vehicles and that they buy a wide range of vehicles of all makes and models.

Unlike some other magazines or surveys, we do not take advertisements from any outside manufacturer, so we have no vested interests in the outcome of our survey. We have no agenda other than communicate accurate results of our survey. We do not consider country of origin in our analyses leading to our reliability ratings.

Some Japanese models in our survey have scored below average in reliability, for example the Mazda CX-3 and Subaru WRX, and some American models for example, the Chevrolet Impala and Ford Taurus have scored above average. Those findings provide evidence against pro-Japanese bias on the part of our members.

European luxury brands have made recent progress in CR's latest surveys. The Audi Q5 and Q7 led with the best predicted reliability scores in the luxury SUVs segment.

Your Survey Results Do Not Match With My Experience. Is Your Survey Wrong?
Even in the most unreliable models, some individual car owners are lucky and experience few or no problems during the 12 months covered by the survey. For example, in one of the worst models in our recent surveys, about 58 percent of the owners reported problems in at least one trouble area over the previous 12 months; of course, this means that about 42 percent of owners reported no problems. Your neighbor or friend might be one of those lucky owners. Of course, the opposite can happen as well—even in a model that tends to be quite reliable, there is an occasional "lemon."

Since the Average Number of Problems Is Small for Most Models, Is Consumer Reports Overemphasizing Differences That May Not Be Important?
Beyond statistical significance, we believe these differences are also meaningful to car buyers. We think that car buyers should expect a new car to be entirely problem-free in its first months or years of service. While the difference between a and a may be small, a pattern of several less-than-perfect trouble spots in a brand new car should be cause for concern and does not bode well for a model's long-term reliability. We have not yet seen a single model in our survey that is entirely problem-free. More than that, the Tesla Model X, which had the worst new car prediction score in the latest survey, is predicted to be about 12 times more likely to have a problem than the best, the Audi Q3. Those differences among models are important for car buyers to consider in choosing a car. We present these scores for trouble spots primarily to allow consumers to compare the relative incidence of problems among models. While there are no guarantees, you can improve your odds of buying a reliable car if you choose a model that has had a lower rate of problems in the past.

Some People Maintain Their Cars Differently From Others. How Does This Affect the Ratings?
The biggest growing area of concern in late-model cars is the in-car electronics: infotainment, entertainment, communication, and navigation systems. Other problem areas include noises and leaks (squeaks, rattles, and leaks), and power equipment (body control module, gauges, and warning lights). Maintenance does not affect those items. Problems in those areas might more likely reflect the inherent design or quality.

How Do You Account for Mileage Differences?
Vehicles with higher mileage will most likely experience more problems than vehicles of the same age with lower mileage. We adjust our analyses to minimize differences among models due to varying mileage. Our data are mileage-standardized by dividing cars of each model into groups of high, average, and low mileage, and employing the statistical technique of direct standardization.