About the Ontario Evaluation Study

Since 2004, Project Porchlight has been doing what no energy conservation campaign has done before: reaching Canadians at the grassroots level. Delivered through a comprehensive social marketing campaign, Project Porchlight makes energy conservation accessible, relevant, and personal by giving regular people the power to take action in their own communities. Action that matters.

But how effective has this exercise in community engagement really been at influencing change in individual attitudes towards energy conservation? To better understand the impact Project Porchlight has on individual attitudes, ONECHANGE engaged Innovative Research Group (INNOVATIVE) to conduct a scientific evaluation of their campaign in Ontario.

Volunteers' Survey:
Conduct an online survey of Project Porchlight volunteers to better understand their general perceptions of the organization and campaign to determine what messages and tactics will work best at mobilizing volunteers for future programs/campaigns and how to best attract new volunteers to broaden the grassroots network.

The results of the online volunteer survey showed that the experience of volunteering for Project Porchlight had a strong impact. Results include the following summaries:


Volunteer Satisfaction:

• 93% of volunteers surveyed describe their experience with Project Porchlight as positive.
• Among those who volunteered at a Project Porchlight event, almost everyone enjoyed participating (96%) and found them well organized (72%).
• 80% agree that Project Porchlight staff are friendly and knowledgeable
• When asked if there was anything Project Porchlight could have done to make the volunteer experience easier or more positive, a third would like to see improvements in logistics, organization and communication (30%) while close to half (40%) said no improvements were necessary.
• 40% of volunteers returned at least once to deliver more bulbs showing their enthusiasm for the campaign and their enjoyment of the volunteer experience.

Impact on Attitudes and Behaviour:

• 80% volunteers say they’ve made lifestyle changes that reduce energy use since their Project Porchlight experience.
• 59% of respondents say they’re confident on where to take CFLs for proper disposal. Of those who say they’re confident they know where to dispose of CFLs, 81% are either extremely or very likely to take CFLs to proper disposal site.
  
  
  Volunteer Support:
  • 88% of volunteers think their community would react positively if Project Porchlight mobilized its volunteers to affect change in environmental policy
  • In terms of volunteer retention, 72% say they're either extremely or very likely to volunteer again with Project Porchlight. Furthermore, 92% say they're either extremely or very likely to volunteer for another One Change initiative in the future aside from Project Porchlight.
  • 72% would like Project Porchlight to contact them about news and events in the future.
  • 33% of respondents say they’re likely to make a charitable donation to Project Porchlight if asked.

  
  
  Consumers Survey

  • This study was conducted by telephone among 600 Oakville, Ontario residents between November 14 and November 20, 2007.
  • The pre-test (n=600) margin of error is +/- 4.0%, 19 times out of 20.

  
  
  Key Findings:

  • Project Porchlight favourables doubled in the drop off treatment and more than doubled with conversation.
  • One Change grew by smaller but still statistically significant amounts in both treatments.
  • The control group also declined significantly on how likely it is they will use a CFL bulb while both treatments groups remained statistically stable.
  • The control group declined significantly again on how often then think about electricity consumption while both treatments groups remained statistically stable.
  • The control group was a significant 12 points more likely to agree without government action, no progress is likely while the treatment groups held stable.
  • The control group showed a marginal decline in the number of people who felt CFL are a good idea but both treatment groups remained statistically stable. Conversation trumps other groups in three more cases
  • Conservation gets people talking. The conversation group demonstrated a significant increase in the number of people who say they have talked about electricity at least a few times in the past month
  • While the control and drop off groups saw a small but significant decrease in the number of people reporting energy-efficient products can make a major difference in protecting the environment, the conservation group avoided the decline.
  • The conversation group also was able to demonstrate a significant improvement in the number of people saying energy star appliances make at least a minor difference while the other groups narrowly avoid a significant decline in those reporting a major difference. No change where no change makes sense
  • With 90% agreement in both treatments groups in the first wave to the idea that simple changes can make a big differences, there was little room for improvement on this measure.

  
  
  General Overview:
  

• This study clearly shows that on a majority of measures that could move, the people included in the Project Porchlight program ended up with more favourable views than people in the control group.
• The doorstop conversation is clearly a formidable brand building tool.
• Both the doorstop conversation and simply distributing the bulbs re-enforced existing favourable opinions during a period in which they were sliding among the control group.
• One question that remains outstanding is what happened in the control group? The relative decline of these participants was not unique to Oakville. Learning why Canadians who are focused on conservation retreat from initially positive attitudes is an important topic to cover in future studies. However, while there is always room for improvement, Project Porchlight clearly is an effective vehicle for avoiding these declines in the future.
  
  
  Jason Lockhart and Greg Lyle, Innovative Research Group June 2008
  
  
  
  Control vs. treatment groups
  
  
  The control group received no exposure to the Project Porchlight campaign, while the treatment group received exposure to the campaign. This design allows us to determine whether the Project Porchlight campaign is actually a causal factor in changing individual attitudes. For example, when we introduce the Project Porchlight campaign to the treatment group but not to the control group, we expect in some cases to observe a shift in attitudes occurring in the treatment group at a significantly greater rate than in the control group.