A Taste for Claims: Conducting Sensory Claim Substantiation Surveys

Non-Comparative vs. Comparative Sensory Claims

Non-comparative claims are those that make a statement about the sensory attributes of a product without comparing it to other brands, such as “dry-touch sunscreen”. In contrast, comparative claims are statements that make a comparison to other brand(s) in that product category, like “softer on sensitive skin than the leading brands.”

Comparative claims break down further into two sub-categories: parity claims and superiority claims. Superiority claims state that the product is better than a competitor’s product, i.e., “our sugar substitute tastes more like sugar than Brand X.”ⁱⁱⁱ A parity claim may be an equality claim or an unsurpassed claim. An equality claim states that the product is at least as good as competitive products (“leaves hair just as soft as the name brand”), while an unsurpassed claim states that the competing product is not better or greater than the advertised product in some way (“nobody beats our odor-fighting power”).^iv

Hedonic and Perception Claims

Within each category of claims, there are two main types of sensory claims: hedonic claims and perception claims.

Hedonic claims are emotional, subjective, and experiential in nature, relating to overall or attribute-specific liking and preferences. “Great taste” is an example of a non-comparative hedonic claim, and “tastes as good as Brand X” or “preferred over Brand X” are examples of comparative hedonic claims.

Perception claims, on the other hand, are analytical in nature, relating to the perceived intensity of specific sensory attributes. “Savory” or “crispy” are examples of non-comparative perception claims, while “tastes as sweet as Brand X” or “more vanilla flavor than Brand X” are examples of comparative perception claims. More examples can be found in Figure 1 above.

Any of these claims may be challenged, particularly in a competitive marketplace. The Federal Trade Commission Act requires that all advertisers have a reasonable basis for their claims prior to introducing those claims to the marketplace.^v Without this type of evidence, advertisers may be vulnerable in a lawsuit, regulatory action, or private dispute resolution, putting them at risk of losing substantial money and having to withdraw advertisements.^vi To withstand these potential challenges, advertisers should substantiate their sensory claims with reliable, valid, and relevant scientific research.

In the context of sensory claims, substantiation research may take the form of sensory evaluation, which provides information on how a product’s consumers perceive its taste, smell, feel, or other sensory attributes. This piece provides information about how to substantiate sensory claims in a scientifically sound way that can stand up to legal and regulatory challenges.

Types of Sensory Data to Substantiate a Claim

To substantiate a sensory claim, advertisers must gather data, generally through surveys. There are three main types of data that can support sensory claims: consumer data, trained sensory panel data, and analytical instrument data. Consumer data is the most broadly used and can support both hedonic and perception claims. Analytical data is typically used to support perception claims.

Consumer Data

Consumer data is collected from consumers who represent the relevant consumer population for the advertised product category, and who have not been specially trained in sensory perception. For example, if an advertiser is claiming that “mothers prefer us,” it can interview a group of mothers who regularly purchase products in the right category. In some cases, data from specialized groups of consumers is required, as with a claim about the preferences of dermatologists.

A relatively large sample size of untrained consumers is typically required for this type of survey, which may ask about opinions, feelings, and preferences.^vii In addition, researchers cannot repeat a survey using the same sample set and the same set of consumers in a short period of time, because once a consumer has already evaluated a product, their opinions may be influenced by the first survey. Consumer data has many applications in advertising claim substantiation, including:

• Prior to claim substantiation, identifying which sensory attributes to highlight in consumer perception claims for a product
• Substantiating a comparative superiority claim about preferences (such as “seven out of ten consumers prefer Brand X to Brand Y”)
• Substantiating a claim that the product has been improved in some way (i.e., “now with improved formula” or “new and improved formula”)
• Substantiating a perception claim (“now meatier,” “even juicier,” etc.) by having consumers rate the intensity of the specific sensory characteristic that the ad will highlight

When working with consumers, researchers should use sensory terms or attributes that can be easily understood and evaluated by those consumers.

Trained Sensory Panel Data

Also known as descriptive analysis data or trained panel data, trained sensory panel data is collected from a small sample of trained descriptive sensory panelists rather than from ordinary consumers. Descriptive panels are highly trained and more like an analytical tool or instrument in that they describe the presence and intensity of sensory attributes. Panelists are extensively trained and evaluate the samples two or three times, meaning duplicate or triplicate measures are collected from the panel on the same sample set to ensure consistency, reliability, and ability to discriminate among samples. A smaller sample size, typically 8-12 trained panelists, is sufficient.^viii

The objective of this type of research is to identify and quantify the relevant sensory attributes (including appearance, flavor, and texture attributes) in a product; there are no measures of preference or liking. Trained sensory panel research might be used to:

• Compare a product’s sensory attributes to its own previous formulations or to competitors (for example: “now sweeter” or “sweeter than Brand X”)
• Identify sensory attributes to highlight in advertising (like “sweet, fruity, and floral”)
• Identify overall perceived differences in sensory characteristics among products
• Measure specific differences in flavor and texture among products
• Measure perceived intensity of sensory characteristics over time (i.e., “first they’re sour, then they’re sweet” or “the burn that sneaks up on you”)

Analytical Instrument Data

Analytical instrument data, or laboratory instrument data, can also be used to compare the intensity of specific sensory attributes, such as texture, flavor, or visual attributes. However, although instruments are highly objective and can provide reliable, valid data analyzed in triplicate, instrument results do not always translate to consumer perception or acceptance.

An instrument may pick up subtle differences between products or prototypes that would not be detected by consumers. Thus, a small difference in flavor detected by an instrument may not be noticeable or meaningful to consumers. To ensure that your research can be successfully used as substantiation for your claim, you may need to provide sensory testing alongside or instead of testing with laboratory instruments.

Best Practices in Sensory Research Design and Testing

The research design for a claim substantiation survey is dictated by the claim itself, so you must have an idea of what you would like to communicate prior to designing the research. If you do not know, there are preliminary research steps that can be taken to identify potential sensory claims.

Regardless of how you decide to test your advertising claim, it is vital to follow established best practices for sensory research. A survey that cannot stand up to a challenge by a competitor or a regulator is just like no substantiation at all (but more expensive). This section describes several basics of reliable design for sensory claim substantiation surveys.

1. The Aim of the Claim (Objective)

The message communicated by your sensory claim determines the type of sensory data required, the appropriate test methods, the questions that will be asked, and the relevant group of people to ask (if applicable). A single survey can test multiple claims, in some cases, if designed properly. If needed, preliminary testing can be done with either a group of consumers or a trained sensory panel to identify which product characteristics should be the focus of the claim.

2. Who to Interview

The survey respondents, or survey universe, must be users of the product category being tested. Otherwise, the findings may not be relevant. For example, if a claim study on coffee finds that 75% of respondents prefer Product A over Product B, but only 20% of the respondents are coffee drinkers, the results are not necessarily representative of the preferences of the coffee drinker population.

It is also important to recruit respondents who are representative of the consumer population of that product category; if they are not, the survey could be irrelevant and worth very little in court or with the National Advertising Division (NAD) of the Better Business Bureau (a private forum for deciding advertising conflicts).^ix Hedonic claims are typically tested with untrained consumers, while perception claims can be tested with untrained consumers, a trained panel, or laboratory instruments. Additionally, the respondents ought to be representative of real-life buyers of the product in terms of age, gender, geography, and any other demographic that might be relevant. The four major geographic regions in the United States should be represented if the product is or will be sold nationally.

Researchers may have reasons to exclude certain respondents. For example, respondents who have dietary restrictions, food allergies, or food sensitivities may be unable to evaluate certain foods, and those with a cold or allergy symptoms may not be able to taste or smell accurately. When testing alcohol, researchers should exclude anyone who arrives already inebriated or who has conditions or takes medications that limit alcohol consumption.

3. The Relevant Time Frame

Researchers should qualify consumers for surveys by asking about purchases within a time frame that is relevant for the product or product category at issue. However, what timeframe is relevant may depend on the category. Consumption or purchase frequency can vary across categories; for example, many foods are purchased frequently, while bottles of liquor may be more occasional purchases. The product’s usage or purchase frequency data, if available, can help identify the appropriate timeframe.

4. Data Collection Methods

There are two main data collection methods: Central Location Testing (CLT) and Home Use Testing (HUT). The method used the claim.^x In a CLT, the products are assessed in a centralized location, which gives the researchers more control over product preparation and evaluation. This can be useful in situations where some aspects of serving require special care, such as checking respondents’ ages for an alcohol test or serving a beverage at a specific temperature. However, the tradeoff is that this is typically not the real-life context in which the product would be consumed. If the product is or will be sold in many geographic locations, researchers may need to do CLTs in multiple cities.

In an HUT, the products are evaluated in the consumer’s home, so there is less control over product preparation and evaluation. This is sometimes necessary, depending on the type of product and length of the survey. For example, a preference test for pillows may be conducted as an HUT if consumers will need to sleep on the pillows and try them out for a longer period of time. HUTs may also be a good choice for a product that is intended for preparation in the consumer’s home kitchen, such as a baking mix.

5. Product Sampling, Preparation, and Serving

All samples in a test must be prepared and served in “reasonably realistic conditions” and in a manner that minimizes biases.^xi Additional parameters are described below.

Considerations for Product Sampling and Preparation

• Location and expiration

Comparable products should be purchased or procured at the same stores and have similar expiration or best-by dates, if possible, when they have similar shelf lives.

• Size and form

Comparable products should have similar sizes and forms. For example, if a product is sold in both bottles and cans, the products should be either all canned or all bottled, depending on the claim being tested. For comparative claims, testing should be conducted against samples of the brand or the brands that are mentioned in the ad.

• Representation

Products should be sampled from multiple batches that are representative of products available to consumers for purchase, if possible.

• Handling and storage

Researchers should handle and store the products in a way that imitates the typical supply chain protocol for those products.

• Preparation

Researchers and staff should follow the on-package preparation instructions, if any, to ensure that respondents experience the product as the manufacturer intended.

Considerations for Product Serving Procedures

• Sample size and visual appearance

The size and appearance of the samples served should be consistent. Panelists may rely on visual cues, so an inconsistency might give them incorrect ideas about the product, biasing the results.

• Sample serving temperatures

The temperature of the samples should be consistent, and researchers should provide clear instructions about cold storage, cooking, and food warmers. Temperature may affect the perceived flavor and texture of products.

• Serving containers, carriers, and contexts

In a multi-product sensory test, researchers should take steps to ensure that the respondents and the research staff they work with do not know which sample is which. To achieve this “blinding,” samples must have an unbranded identifier, such as a three-digit code, rather than brand names, and be served to respondents in the same way across all samples.

The serving containers and any edible serving medium, as well as the context, should be selected with careful consideration of realistic consumer conditions for the type of product, while minimizing any effects on the perception of the product. For instance, if one sample is served in a white mug, all samples should be served in the same white mugs. If a butter sample is served on a piece of bread, all samples should be served on a piece of bread.

• Palate cleansing and expectoration

To minimize carryover flavor, respondents’ palates must be cleansed, typically with unsalted crackers and a specified type of water. However, the study’s palate-cleansing procedure should be designed with the product in mind.

Certain types of products, such as alcoholic drinks, are sometimes not swallowed but rather expectorated after tasting, because alcohol may accumulate in the bloodstream over time. Additionally, alcohol and other types of products, such as spicy foods or sauces (chemical heat), may have “mouthburn” or an otherwise lingering or irritating effect on the palate and may require additional rest in between products or a milk rinse.^xii

Test Methodology

Two test designs are typically used in product sensory evaluation: sequential monadic design and comparative test design.

Sequential Monadic Design

In a sequential monadic design, the products are evaluated one at a time, with the respondent completing the evaluation of one product before moving on to the next product. The serving sequences are randomized and counterbalanced, meaning that respondents do not evaluate all the products in the same order. This is to balance for psychological and physiological biases, such as first order bias, in which the first product may be liked or rated differently from subsequent products; carryover bias, in which a product affects the ratings of the subsequent product; and sensory and emotional task fatigue, in which the ratings of the products evaluated towards the end are affected.

A sequential monadic design works well in surveys designed to provide numerical measures based on rating scales that respondents fill out. This kind of survey might include questions about how much respondents like a product (acceptance) or questions about the intensity of a sensory attribute (such as saltiness). This type of design can also be used when measuring which of two choices is preferred (binary preference questions), although the researcher must ask the preference question after respondents evaluate both products.

Comparative Test Design

In a comparative test design, respondents evaluate all products to be tested at the same time. A comparative test design is useful for direct preference ratings, in which the assessors are asked about their overall preference or their preference for specific attributes. However, it is not ideal if there are more than two products, because it is more difficult to control the sensory interaction between them.

For example, a product that has a strong aftertaste may carry over into the evaluation of the other product if the assessor is going back and forth between the products. This is controlled for in sequential monadic designs with counterbalanced serving sequences. Another limitation of a comparative test design is that preference questions cannot be combined with sequential monadic-type questions, such as acceptance and intensity ratings, so this method provides less information overall.

Questionnaire Design

Surveys to substantiate sensory claims should be as simple as possible, only including questions necessary to measure the claim. As the ASTM guide states, “In a claims test, more confidence will be placed in data obtained from first-asked questions.” Position the key question first so that it is not biased by respondents’ reactions to other questions, if any, in the survey. Start with questions about overall acceptance first, then ask questions about specific attributes afterwards.^xiii

This is important because questions asked earlier can bias the responses to later questions, which is called order bias. For example, respondents who are rating multiple aspects of a product may rate attributes asked about later in the survey more highly if they have already given a positive rating to an earlier attribute.^xiv In addition, responses that are not relevant to the advertising claim being researched might bias or otherwise limit the usefulness of the responses that do matter to the claims research.

There are several types of possible questions in a consumer test, as shown below.

• Preference: whether consumers prefer one product or another
• Ranking: how consumers rank multiple products in overall preference or according to specific attributes
• Acceptance: whether consumers like the product(s)
• Attributes: how consumers perceive individual aspects of a product(s)

Acceptance questions give absolute measures, while binary preference questions give measures of one product relative to another. Because of this, acceptance questions provide more information than binary preference questions. For example, if both samples are disliked, acceptance measures will reveal this, whereas preference measures will only show which product is preferred, but not that both products are disliked.^xv

Attribute questions also provide deeper insights than binary preferences, diving into:

• Attribute acceptance: whether consumers like a specific attribute of the product
• Attribute diagnostic: whether consumers believe the attribute is at a satisfactory level
• Attribute intensity: how strong a particular flavor, smell, or other sensory quality is

When researchers ask trained panelists to evaluate products, questionnaires often only have one type of question—attribute intensity. In this questionnaire design, typically called a descriptive analysis ballot, trained panelists evaluate the presence and intensity of specific product attributes, such as firmness or chocolate flavor.

As with all questionnaires, questions in sensory claims research should not be worded in a way that suggests there is a correct or desired outcome. Any response options offered should not lead a respondent toward or away from any particular response or prompt the respondent to consider options they might not otherwise have considered.^xvi

6. Data Analysis and Interpretation of Results

Once the research is complete, researchers should ensure that their analysis of the data and interpretation of the results is statistically sound and appropriately executed. For example, a superiority preference claim is supported if a statistically significant proportion of the respondents prefer the advertiser’s product. Or, if an advertiser wants to claim that a product “reduces odor,” they would have to demonstrate that the perceived intensity of relevant odor is lower by a statistically significant amount when the product is used versus when the product is not used.

In sensory tests, statistical confidence in the results—how certain researchers can be that the results are not simply due to guessing or chance—should typically be above 95%. The ASTM guide for sensory claim substantiation provides tables showing thresholds for calculating statistical significance when measuring certain types of sensory claims.^xvii

The analysis that researchers should use depends on the questions asked.

Preference

Preference questions (overall or attribute) provide categorical data and are typically analyzed with a test of proportions, such as a binomial test of proportions. This can determine whether there is a statistically significant proportion of respondents who prefer the advertiser’s product.

Acceptance

Acceptance questions (overall or attribute) provide numerical data and are typically analyzed with a parametric statistical method, such as a t-test or Analysis of Variance (ANOVA) to determine 1) whether there is a statistically significant difference in liking and 2) overall product acceptability or acceptability of specific attributes.

Attribute Acceptance, Diagnostic, and Intensity

Attribute intensity questions provide numerical data and are typically analyzed with a t-test or Analysis of Variance (ANOVA). This can determine whether there is a statistically significant difference among samples in the perception of specific attributes.

Best Practices: A Synopsis

Keep the following guidelines in mind while designing and conducting surveys for sensory claim substantiation research.

• Ensure that the research is designed in a way that will yield results relevant to the claim^xviii
• Replicate the conditions of the real-world marketplace as much as possible^xix
• Interview people who represent buyers of the product,^xx and come from all geographic regions where the product is or will be offered^xxi
• Interview a large enough sample of those people to generate meaningful results^xxii
• Incorporate controls, which can remove survey “noise” to create more accurate measures, into studies when they are applicable^xxiii
• Provide a reliable research report analyzing the results: it should be unbiased and explain the study’s findings, as well as how they apply to the proposed claim^xxiv

Putting It All Together

With completed sensory research in hand, you can start assembling or challenging advertising claims. However, remember that the research must clearly substantiate the specific claim being made. Relying on research that is not quite a match for the claim can lead to lost lawsuits and regulatory proceedings.

If there is a challenge, however, a research report can serve as evidence that the claim is substantiated. Litigation survey experts can provide research reports that can be used as evidence before courts, the NAD, or the FTC, and also offer expert witness testimony if needed. With relevant and well-designed research backing up sensory claims, you can move forward with confidence.

Partnering with Claim Substantiation Experts

The litigation surveys and consumer science team at IMS Legal Strategies specializes in helping businesses and law firm clients understand what consumers are thinking. Our in-house experts provide reliable, thoughtfully designed consumer surveys and analysis for advertising claim substantiation, as evidence in trademark and false advertising disputes, and to help businesses research their markets. All our claim substantiation surveys are litigation-ready, so you can move forward with confidence.

With a dedicated team led by academic experts who also have real-world experience as business leaders, we understand both the theory of reliable survey design and the practical concerns you may face. Please reach out to request a consultation.

References

ⁱStandard Guide for Sensory Claim Substantiation. ASTM E1958-19a, ASTM International, 2019. See also Standard Guide for Serving Protocol for Sensory Evaluation of Foods and Beverages. ASTM E1871-17, ASTM International, 2017.

ⁱⁱStandard Guide for Sensory Claim Substantiation. ASTM E1958-19a, ASTM International, 2019, p. 2.

ⁱⁱⁱ“Types of Claims.” Advertising Claim Substantiation Handbook, edited by Chris Cole, ABA Publishing, 2017, p. 43.

^ivStandard Guide for Sensory Claim Substantiation. ASTM E1958-19a, ASTM International, 2019, p. 3.

^v“FTC Policy Regarding Advertising Substantiation.” Federal Trade Commission, November 23, 1984, https://www.ftc.gov/public-sta....

^vi“Sources of Advertising Law.” Advertising Claim Substantiation Handbook, edited by David Zetoony, ABA Publishing, 2017, pp. 1-21.

^viiLawless, Harry T, and Hildegarde Heymann. Sensory Evaluation of Food Principles and Practices. Springer, 2nd ed., 2010. See also Standard Guide for Sensory Claim Substantiation. ASTM E1958-19a, ASTM International, 2019.

^viiiLawless, Harry T, and Hildegarde Heymann. Sensory Evaluation of Food Principles and Practices. Springer, 2nd ed., 2010. See also Standard Guide for Sensory Claim Substantiation. ASTM E1958-19a, ASTM International, 2019.

^ix“Failure to define the universe properly can have devastating consequences on the evidentiary weight given to a survey and can even lead to its exclusion from evidence in extreme cases.” Barber, William G. and Giulio E. Yaquinto. “The Universe.” Trademark and Deceptive Advertising Surveys Law, Science, and Design, edited by Shari Seidman Diamond and Jerre B. Swann, 2nd ed., ABA Publishing, 2022, pp. 31-32. See also Diamond, Shari Seidman. “Reference Guide on Survey Research.” Reference Manual on Scientific Evidence, 3rd ed., National Academies Press, 2011, pp. 377-378. See also “Implied Claims and Communication Surveys.” Advertising Claim Substantiation Handbook, edited by Norman Simon, ABA Publishing, 2017, p. 36.

^xStandard Guide for Sensory Claim Substantiation. ASTM E1958-19a, ASTM International, 2019, pp. 8-9.

^xiStandard Guide for Sensory Claim Substantiation. ASTM E1958-19a, ASTM International, 2019, pp. 7-8. See also Standard Guide for Serving Protocol for Sensory Evaluation of Foods and Beverages. ASTM E1871-17, ASTM International, 2017.

^xiiStandard Guide for Sensory Evaluation of Beverages Containing Alcohol. ASTM E1879-17, ASTM International, 2017, p. 4. See also See also Standard Guide for Serving Protocol for Sensory Evaluation of Foods and Beverages. ASTM E1871-17, ASTM International, 2017, pp. 2-3.

^xiiiLawless, Harry T, and Hildegarde Heymann. Sensory Evaluation of Food Principles and Practices. Springer, 2nd ed., 2010.

^xivLawless, Harry T, and Hildegarde Heymann. Sensory Evaluation of Food Principles and Practices. Springer, 2nd ed., 2010.

^xvStandard Guide for Sensory Claim Substantiation. ASTM E1958-19a, ASTM International, 2019, pp. 10-11. See also Standard Guide for Two-Sample Acceptance and Preference Testing with Consumers. ASTM E2943-15, ASTM International, 2015.

^xvi“Implied Claims and Communication Surveys.” Advertising Claim Substantiation Handbook, edited by Norman Simon, ABA Publishing, 2017, p. 28. See also Diamond, Shari Seidman. “Reference Guide on Survey Research.” Reference Manual on Scientific Evidence, 3rd ed., National Academies Press, 2011, pp. 390, 392.

^xviiStandard Guide for Sensory Claim Substantiation. ASTM E1958-19a, ASTM International, 2019, pp. 20-23.

^xviii“Implied Claims and Communication Surveys.” Advertising Claim Substantiation Handbook, edited by Norman Simon, ABA Publishing, 2017, p. 30.

^xixMcCarthy, J. Thomas. “§32:163: Survey methodology—Approximating market conditions.” McCarthy on Trademarks and Unfair Competition, 5th ed., Thomson Reuters, 2021, pp. 32-456 to 32-457.

^xx“Implied Claims and Communication Surveys.” Advertising Claim Substantiation Handbook, edited by Norman Simon, ABA Publishing, 2017, p. 27.

^xxiDiamond, Shari Seidman. “Reference Guide on Survey Research.” Reference Manual on Scientific Evidence, 3rd ed., National Academies Press, 2011, pp. 377-379.

^xxii“Implied Claims and Communication Surveys.” Advertising Claim Substantiation Handbook, edited by Norman Simon, ABA Publishing, 2017, p. 27.

^xxiii“Implied Claims and Communication Surveys.” Advertising Claim Substantiation Handbook, edited by Norman Simon, ABA Publishing, 2017, pp. 27-28.

^xxivBernstein, David H. and Bruce P. Keller. “Survey Evidence in False Advertising Cases.” Trademark and Deceptive Advertising Surveys: Law, Science, and Design, 2nd ed., edited by Shari Seidman Diamond and Jerre B. Swann, ABA Publishing, 2022, p. 208 (survey data should be accurately reported and analyzed in accordance with accepted statistical principles).