Survey says…?

by Jen

The results are in, and they are… ambiguous. Ok, just kidding. There are some clear implications but they require a bit of unpacking, so unfortunately I can’t lead off with a straightforward headline like “All hierarchical-individualists like weaponry metaphors!” If you’ll read along, however, you’ll see that the results are, in fact, positive and, I think, illuminating. In case you’ve forgotten (or never been here), I created an informational piece about vaccines to test out the design strategies I’ve been developing. The accompanying survey garnered 70 complete responses. (A lot more people viewed it and started the survey, but only 70 provided quality answers to the open ended items, so those are the ones I am working with). Right off the bat I have to admit I’m disappointed in that number; I tried my best to get this survey in front of as many people, and as wide a variety, as possible. Alas, without funding to offer compensation for respondents or to pay for services that promote surveys online or via telephone, my options were fairly limited and within the scope of a 9-month thesis project I have to accept these constraints.

The survey collected respondents’ demographic information as well as their ‘cultural worldview,’ (you can read the basics here and here if you need a refresher), using the screening questions described by Kahan in his conception of cultural cognition (fig. 3 in the paper). Respondents indicated the level of their “disagreement” or “agreement” with each item on a Likert response measure in order to place each respondent on a map at the intersection of two scales: hierarchical-egalitarian and individualist-communitarian. Each respondent then answered a handful of questions about vaccination to assess factual knowledge about vaccination as well as attitudes toward vaccination. After viewing the informational piece, the respondents were asked the same questions about attitudes (to determine if the piece had any effect on attitudes) as well as prompted to share what they found most and least appealing about the piece, and to identify which arguments for vaccination were most appealing or resonant to them.

respondents worldviews-01-01

As you can see, I did not get very many responses from those with hierarchical values. On the one hand, this is disappointing because I designed the vaccine piece to test whether certain ways of framing information appealed to each of the four quadrants of viewers, so I need people in those quadrants to make this study valid. On the other hand, I had previously identified that much of the vaccine “controversy” unfolds along the individualist-communitarian line, because most of the pro-vaccination messaging we see is designed by, and for, egalitarian-communitarian values and those with more individualist (whether hierarchical or egalitarian) values are left to worry about individual risk with little attention paid to their concerns. There is still a good spread of responses from individualist to communitarian, so despite my disappointment in the overall cross section, I decided to forge ahead with the data analysis to find out what I could.

The first numbers I wanted to see were about attitudes. I asked the same six questions before and after respondents viewed the vaccine piece to see if viewing it would have any affect. Each item was a Likert scale measure:

“For each of the following items, please indicate where your opinion lies on a spectrum between the two statements.”

  • Vaccines are safe — Vaccines are not safe
  • I would definitely have (or already had) my children vaccinated — I would definitely not (or have already opted not to) have my children vaccinated
  • Childhood vaccines do not cause autism — Childhood vaccines can cause autism
  • Everyone should be required to vaccinate — Vaccination should be completely optional
  • Relying on natural immunity instead of getting vaccinated is risky — Natural immunity is better than vaccinating
  • Vaccination is a good idea — Vaccination is not worth the risk

Before viewing the piece, the average attitude toward vaccines was overall slightly positive. Those scoring individualist tended to feel less positive, while the communitarian responses were a little more positive- but all within a range that wasn’t statistically significant. There was not a statistically significant correlation between the change in responses and cultural worldview either. That is, people were predisposed to feel a certain way about vaccines and while that attitude may have changed slightly after viewing the piece, the extent of that change itself was not tied to any worldview: individualists and communitarians alike tended to adjust their attitudes by roughly the same amount. The item was scored as a sum of the six questions (five-point Likert measures), so a person feeling completely neutral would score 0, a person with more negative attitudes would score -1 to -12 and a person with more positive attitudes +1 to +12.

The average score before viewing the piece was +6.9 and after viewing was +7.4. Nice to see a positive change, to be sure, but it’s not a significant change at all (statistically or otherwise). Generally, most people’s scores did not change by much: some increased (and a small handful decreased) by one or two points (for example, someone who was neutral about the autism link before may have changed their reply to be one step closer to the statement that vaccines do not cause autism), and a very small number showed significant increases. One person’s score increased by 5 points and another by 8 points (both very strong egalitarian-communitarians), which would correspond to someone choosing more positive answers on most, if not all six, of the questions… but the overall positive trend resulted from a larger number of people with small (+1 or +2) changes- perhaps on just one or two of the questions. Generally, with a change this small, we really can’t assume anything- some people may have felt no change in attitude at all and simply changed their answer by one or two steps in either direction because they didn’t realize which one they had selected five minutes earlier, before viewing the piece.

I certainly don’t think one should take very much away from these results; I included this item out of sheer curiosity but it’s silly to think a single viewing of a single artifact could have a significant effect (and certainly not lasting) on a person’s attitudes. Without any correlation to respondents’ cultural worldview, this information doesn’t offer much; however, the overall positive trend (however small) does suggest that the piece wasn’t detrimental or polarizing, which is a good thing.

On to the arguments themselves. As you hopefully already know, the vaccine piece included an opening conversation about seat belts as a metaphor for vaccination, followed by a chart that illustrated how anti-vaccination movements over time have contributed to increases in preventable disease, and concluded with a handful of arguments for vaccination that were framed for different worldviews and presented under the guise of three questions.

questions

The “why should I” arguments included information framed in a way that I thought would be most compelling for those with communitarian (and to an extent, egalitarian) values. They focused on community responsibility and interdependence. (Clicking on each item took the viewer to more information about each statement):

should i

The “why would I” arguments included information framed in a way that I thought would be most compelling for those with individualist values. They focused on the benefits to the individual and followed a “me against the world” theme through comparisons to training and protection:

would i

The “why wouldn’t I” arguments included information that I didn’t frame for a particular worldview; instead I hypothesized that this information would be fairly neutral in its attempts to debunk some commonly held myths about vaccination. I used plain, straightforward language, statistics, and a neutral tone.

wouldnt i

And at the end the survey I asked respondents to identify which of the arguments (of these nine) they found most appealing or resonant. Here are the results:

worldview scatter preferred arguments-01

At first glance, this graph looks like it shows a lot of nothing, or at least a lot of randomness. But let’s take a closer look. First, notice that no one chose the very last argument that debunks the myth about overloading immune systems, so it’s not included in the graph at all. Next, let’s consider what might be expected. In a way, I might have been hoping to see a whole bunch of blue on the left and red on the right (that is, the arguments that  thought were individualist should have appealed to the individualists, and the arguments I thought were communitarian should have appealed to the communitarians). That’s definitely not the case; there is certainly a good amount of variability in the preference for different arguments. In hindsight, I have to admit that a) if that had really happened, it would have been a bit disconcerting- people are certainly not that one dimensional, and b) this result is a little more interesting (and I think, encouraging) because it suggests that there is potential for people with different cultural worldviews to find the same sort of messages compelling (even if for different reasons, which I’ll expand on below).

Just to be thorough, though, I decided to map these results purely on a horizontal individualist-communitarian scale:

horizontal scale-02-02And it would seem this graph does illustrate a little bit of a trend. Not a strong one, but it appears the individualist (why would I) arguments are more popular toward the individualist end of the scale, and vice versa. So, my original hypothesis isn’t completely shattered- there’s definitely something going on here- and it seems in general the ‘individualist arguments’ were more popular with ‘individualists’ and so on. We can better understand what’s really going on here by investigating the actual text responses.

But, before I get to that, how about the green items? These were the arguments that I thought would be fairly neutral, that basically provided evidence in the form of statistics that debunk the myth of an autism/vaccine link and dispel the notion that vaccines cause severe reactions. It sure looks like those arguments were preferred by the communitarians, because all but one of the people who chose those arguments as their favorite fell on the communitarian side of the map. I’m not entirely sure what to make of this result but it’s interesting. I wonder if it doesn’t have a little bit to do with some confirmation bias- the folks who are already fairly pro-vaccination (probably communitarians) may have mentally highlighted this information, in its neutrality, for supporting their previously held beliefs, while those who were anti-vaccine (or more accurately, fairly neutral on the subject overall, most likely individualists) had a correspondingly lukewarm reaction to this type of neutral language and wouldn’t have chosen it as the most appealing argument. It could also just mean that some of the people who scored as communitarians just preferred the statistics over more specifically communitarian language (which then leads me to wonder why more individualists didn’t also feel that way). I may explore this result a bit further in the future.

Onto the open ended responses. I asked the respondents to identify which part(s) of the vaccine piece appealed to them most, or resonated most with them, and why, as well as the reverse, (which part(s) appealed to them least). Before I talk about the general findings from that info, let’s look at the first graph again. I was curious about all those people who scored as individualists but claimed to prefer a communitarian (why should I) argument, as well as the communitarians who preferred what I thought were individualist (why would I) arguments. What’s going on there? Is this proof that I, as a designer trying my hand at this stuff for the first time, simply did a bad job of framing information specifically for what I thought were communitarian and individualist arguments? Or is some of the underlying theory, based on so much social science, flawed in its application to this kind of communication design?

Or… maybe these folks are choosing arguments I didn’t expect them to choose, but for reasons I would expect them to give. Consider the following responses:

worldview scatter preferred arguments HIGHLIGHTED POINTS-03

Respondent A, who scored on the individualist end of the spectrum, claimed the arguments about herd immunity and community responsibility were most appealing, but in the open ended response, also stated:

the issues here seem to be about overall societal importance; I see this as an individual choice but it also made me think about the overall effect on society.

Perhaps this respondent is saying, as an individualist, the individualist arguments may resonate more but the communitarian arguments actually achieved some level of convincing beyond his or her current views. The “I never thought about it that way before” sentiment is echoed throughout the text responses at a significant level, and it may explain some of the ‘unexpected’ points on the graph.

Take respondent B, who also scored as an individualist, (though less individualist than respondent A). He/she also identified a communitarian argument as the most resonant, and in the open ended section described the most appealing part of the whole piece as being prompted to

reflect on my views on society’s good vs. individual freedom

And to explain why, said:

it forced me to take the greatest time to consider my answers. I spend quite a deal of time trying to develop and revise my own personal, internally-consistent political philosophy.

Thank you, respondent B, for being so thoughtful. And explaining why you, as an individualist, ended up finding a communitarian argument most appealing.

Similarly, respondent C, a communitarian who chose an individualist argument, said this:

vaccination is a safe procedure and the outcome benefits us all, as individuals and as members of society

and

clearly, to get vaccinated is an individual responsibility, but not in every case- sometimes you pay a high price, or things make them unavailable to a great portion of the society, government should be responsible for making the vaccination process available to all the population.

This type of response suggests that this person does, indeed, share communitarian values and sees vaccination as a community responsibility, but perhaps found the argument about arming one’s immune system with weapons to fight disease as a good case to make to a government whose job it is to defend its citizens.

Respondent D, also a communitarian who chose the argument about arming one’s immune system, wrote:

good points that I liked: societal obligation vs. individual choice, allocation of risk and which to minimize i.e. getting vaccinated creates a tiny risk for my child but significantly reduces societal risk whereas without vaccines I PERCEIVE to have all risk eliminated for my child but society may take on greater risk…

I am highlighting these handful of examples to illustrate what I’ve learned: sure, individualists do seem to prefer the arguments I expected to be compelling for individualists, and communitarians do seem to prefer the arguments I expected to be compelling for communitarians, but that effect is not dramatic. Perhaps more importantly, there is evidence to suggest that two people with different values may both find the same message appealing, but for different reasons- and those reasons are consistent with their values. Individualists like respondents A and B may have found the information framed for communitarian values appealing, but both expressed their reasoning through language consistent with individualist values, and the same is true for C and D for communitarian values.

Of course, this may not be the case for every single respondent (and likely isn’t), but my analysis of the open text responses supports this general interpretation. What’s more, it’s encouraging to see that a science communicator can look for ways to frame information with a few carefully crafted metaphors and creativity in voice and narrative, and expect to reach audiences with different values who may find overlapping arguments resonant, even if they do so for different reasons. At this point, the major take away may be that it’s more important to avoid alienating a particular audience than it is to meticulously engineer a message for a specific cultural worldview, which may be where this particular piece about vaccines was most successful. As I continue to consider how I can put this knowledge in the hands of science communicators, it is reassuring to think that a solid understanding of cultural cognition and a willingness to include all four ends of the value scales may be all it takes to get some ideas through the psychological door of otherwise resistant audiences.

Finally, I explored the open text responses for common themes among what people found most and least appealing. As a quick and completely non-scientifically-valid exercise, I threw them all into wordle, because hey, why not.

The responses from those with individualist values:

individualists-what resonated most

The responses from those with communitarian values:

communitarians-what resonated most

Of course these are mostly meaningless but I did get a good chuckle out of the difference in word frequency among the two groups.

As for the meaningful results, I analyzed the text for the most commonly cited examples of what respondents found “most appealing” and “least appealing.” Most people elaborated on what they found most compelling and resonant, but several did not answer the same question about “least appealing,” or did so with little elaboration, so there are fewer responses for the negative aspects of the piece.

The most common items cited as most appealing (with at least five mentions) were:

  • providing multiple viewpoints (25 mentions)
  • the seat-belt metaphor (12 mentions)
  • the timeline depicting anti-vaccination movements (7 mentions)
  • the old anti-vaccination propaganda and painting (6 mentions)
  • presenting facts and statistics (6 mentions)
  • the conversational tone (5 mentions)
  • the interactivity (5 mentions)

The most common items cited as least appealing (with at least three mentions) were:

  • the pace of the opening conversation (12 mentions)
  • the seat-belt metaphor (4 mentions)
  • not enough statistics (3 mentions)

most appealing least appealing

You can see that while most of the positives cited by respondents relate to content, most of the negatives refer more to the form of the piece itself, (which admittedly could stand to be refined a bit more, I agree the pacing is slow in the opening segment, among other things).

Interestingly, while many cited the seat-belt metaphor as a strength of the piece, some seemed to hate it. Here’s what those four people had to say:

these two issues seem to disparate in terms of societal importance

I was actually one of those people who was in a car accident and I was saved by not wearing a seat belt

seat belts are a bad argument and not a consistent parallel: someone who is afraid of being caught in a burning car because of a seat belt can carry a pocket knife while still benefiting from seat belts

I guess honestly I feel like seat belts are more important (for adults) than vaccinations

The first two responses were from people who scored very strongly individualist and the second two scored strongly communitarian, and all four were strongly egalitarian, so I can’t draw any conclusions about cultural cognition’s effect here; there will always be a few people who simply don’t experience information the way a majority of others do. However, it seems like their dislike of the metaphor stems either from unusual personal experiences or strongly held opinions about seat belts. Since my choice of seat belt metaphor hinged on the assumption that most people would agree that they are a good idea and wise for both individual and community benefit, it makes sense that anyone who doesn’t enter the conversation with that philosophy may dislike the entire comparison.

The most compelling implication seems to be the power of making someone see a subject in new light, of presenting information in multiple forms or from multiple perspectives, and ultimately helping someone experience that “I never thought of it that way before” moment.

Consider the following quotes, each of the following comes from different respondents who scored as communitarian:

On the concept of vaccination as a right

I had never thought about them that way

On the concept of vaccinating children as an individual choice to avoid risk,

I have never really thought about it in that light

On the myth of an autism-vaccine link,

I hadn’t paid much attention to the autism debate, it was satisfying to have a clear answer

On the risk that comes from others who may chose not to vaccinate,

I hadn’t considered that aspect before as much as the others

On the seat-belt metaphor,

it’s a good common sense argument that made me stop and think about the risks of not vaccinating

made me stop and think about tradeoffs between different types of risk on a problem I didn’t already have a sensitivity on

And, each of these from different respondents who scored as individualist:

On the seat belt metaphor,

it made me think more about what the government’s role in vaccination could/should be

On the timeline depicting past anti-vaccination sentiment

it was the one part of the entire presentation I never thought about before

I never thought people had the same fears decades ago

On the concept of vaccination as a responsibility

it made me stop and think about it in a new way

********

All in all, I consider these results encouraging. While I initially expected to see a more drastic split in the types of arguments that resonated at each end of the cultural worldview spectrum, hindsight suggests that would have been a disappointing result and would have implied the need to carefully integrate distinct narratives and metaphors any time we might wish to communicate with large and varied audiences. I was able to design a piece that did just that for the purposes of a study, but in the wider world where no single communication artifact operates outside a larger ecology of information exchange, it becomes exponentially more challenging to engineer a way to unify those disparate messages into one piece.

Thankfully, these results suggest that it may not be necessary, or even ideal, to frame information uniquely for each cultural worldview, but instead just to prioritize the inclusion of multiple perspectives and avoidance of polarizing language. This implication aligns perfectly with the significance of pluralistic advocacy in my design strategy; providing multiple perspectives that affirm, in one manner or another, the values of each cultural worldview, provides opportunities for individuals to align with the messages that resonate most, regardless of whether it is due to those messages’ affirmation of cultural identity, or because they provide an opportunity for welcome reflection and adjustment of views on a subject they had previously only considered in a different light. I’m really looking forward to bringing this perspective forward into the development of design tools that will be useful for science communicators who want to put these strategies to work in the world.

Advertisements