Perception and Reality: A National Evaluation of Social Norms Marketing Interventions to Reduce College Student's Heavy Alcohol Use by Henry Wechsler, Toben Nelson, Jae Eun Lee, Mark Seibring, Catherine Lewis and Richard Keeling. (Journal of Studies on Alcohol, 2003, 64:484-494) evaluates the effectiveness of social norms marketing campaigns by comparing schools in the College Alcohol Survey (CAS) that met the authors' criteria for a social norms marketing campaign with schools that were determined to not have a campaign. When data was analyzed from the 1997, 1999 and 2001 CAS surveys, these two groups of schools did not differ from each other on seven measures of alcohol use. As a result, the authors concluded that the study "does not provide evidence to support the effectiveness of social norms marketing programs, as currently utilized…" and they urged "college administrators and health educators to base their prevention programs on scientific evidence instead of the perception of promise." This is an important study that merits attention and careful scrutiny. Its announcement was planned with the assistance of a public relations firm and was preceded by great secrecy in the form of an embargoed press release that when released was reported in almost of the major national media outlets.

The results of the study hinge on the authors' definition of a social norms marketing program. These were identified in two ways: administrators were asked if their school had "ever conducted a social-norms campaign to decrease alcohol use and related-problems," and an index of student exposure to these campaigns was created based on students' reporting that their school provided information on student drinking rates, and/or that they saw posters or signs, read announcements or articles in the student newspaper, or received mailings and handouts.

A number of experts, including Michael Haines, social norms co-founder H. Wesley Perkins and William DeJong have questioned the methodology of the study. They point out that the definition used for a social norms campaign is not strict enough to ensure that a valid campaign was conducted. In fact, with 50% of CAS study administrators reporting that they had such a campaign, it is entirely likely that the measure is too broad and overestimates the number of accurate, well-designed campaigns. In addition, the measures that were used to identify student exposure to social norms marketing do not guarantee in any way that what the students saw meet the minimum requirements of a bona-fide social norms effort.

The authors make a number of criticisms of the social norms literature that would also apply to their own study. For example, they mention the low response rate in some social norms studies, yet the CAS response rates for the 120 schools ranged from 22-86%. With only 215 surveys distributed on each campus this computes to an "N" of 47-185 per campus, independent of campus size. This is hardly an adequate sample size, especially in schools with large enrollments that were found to be most typical of schools reporting social norms campaigns. The authors also present an incomplete review of the literature, draw incorrect conclusions from the studies cited, and make unsupported criticisms of social norms theory.

To provide an example of how a school with a weak campaign could have been included in the study, consider a hypothetical case of a campus with a first-year orientation presentation that includes a discussion of campus drinking norms. The campus also has posters or signs, announcements in the student newspaper, and mailings or handouts on alcohol prevention with unspecified content. Although this campus does not have anything else relating to social norms other than the orientation workshop, it would qualify as having a social norms campaign in Wechsler's study even though the program is too limited in scope to have an impact on overall drinking rates. In another example provided by William DeJong in this issue, a campaign that violated the basic premises of social norms media construction would also have been included in the study.

Imagine that instead of asking about social norms campaigns the study had asked administrators if they "have policies and disciplinary approaches to decreasing alcohol use and related problems on campus" and then asked students if their school had policies, if they were enforced, and if they had received brochures or information about them. These schools would constitute the "policy group" in this imaginary study and would have been compared with the schools that did not report having policies. Determining if schools have policies and if their students know about them is in no way a valid measure of the effectiveness of policy enforcement and other strategies recommended by Wechsler and his colleagues. Yet this was the methodology chosen to evaluate the social norms approach.

A more valid measure of the effectiveness of social norms marketing campaigns could have been conducted with the CAS data set. The study's authors could have asked a panel of social norms experts to rate the quality of the campaigns on those campuses that reported having social norms programs. Then, those campuses rated as having excellent programs could have been compared with other campuses in the study. This would have created a more valid evaluation of the effectiveness of social norms marketing campaigns, properly executed.

In summary, it is necessary for social norms practitioners to be familiar with this study and understand its limitations. This is especially important given the large amount of uncritical attention that it has received. Our job is not only to design, implement and evaluate social norms interventions that are theoretically sound and faithful to the principles of social norms, but also to educate key stakeholders and audiences on our campuses to understand criticisms and answer questions. In this regard, Henry Wechsler, Richard Keeling, and their colleagues have once again provided us with an excellent opportunity to do our homework.