Why do we evaluate?

When we evaluate communication activities we basically pursue the same objectives as with the evaluation of any other programme or project activity: we would like to measure the effectiveness and impact of our communication activities and tools including programme/project communication, media outreach strategy and internal communication. The basic question you should ask yourself is whether you are providing the right communication means and products to the right people through the right channel at the right moment? If the answer to all of these questions is yes, it is very likely that you are having an impact in terms of communication.

Impact and communication

However, impact assessment of communication is not an easy task. Often it is difficult to measure cause and effect and sometimes the impact can only be measured a long time after you have finished your activity. You should also stay realistic, since "changing public opinion at large" seems like a great objective, but often it is impossible to reach within the framework of a European Territorial Cooperation programme. So setting realistic but not trivial objectives from the very start of your programming lifecycle is important.

Impact itself can be measured at various levels. Here are some examples:

1) Communication output: is the actual product of a communication activity, what is delivered to the target audience (e.g. conferences held, press releases distributed..)

However, these figures reveal nothing about the quality, reach, impact or utility of your communication activities.

2) Outgrowth: refers to how the target public demands and receives the output produced (response rates which can be measured e.g. by media clippings, attendance figures, web hits...)

Provides information on the availability of information and who potentially received it, but no information on how much of the information was retained or led to further impact.

3) Outcome I awareness/knowledge: impact can be assessed in terms of knowledge gains that can be traced back to communication activities

The problem is that the term awareness raising is often too vague.

4) Outcome II Attitude/perception: (Positive) perception of a project or your programme and EU funding in general

The difficulty is that often evaluations of attitudes lack clear objectives and changes cannot be traced back to communication measures.

5) Outcome III Behaviour: For example influence on regional policy (regulations..)

This is most obvious when things have gone wrong (protests, complaints...), however, behavioural communication impacts can hardly be isolated validly.

Legal background

At the beginning of the programming period all programmes had to draft a communication plan including a detailed communication strategy. According to Commission Regulation (EC) 1828/2006, article 2 the communication plan shall include (...) "an indication of how the information and publicity measures are to be evaluated in terms of visibility and awareness of operational programmes and of the role played by the Community." Further in the same regulation, article 4 it says "The means used for implementing monitoring and evaluating the communication plan shall be proportional to the information and publicity measures identified in the communication plan." The annual activity report of each programme due on 30 June 2011 has to assess the effectiveness of the communication strategy and propose possible amendments to the communication plan if necessary. The report shall contain a chapter evaluating the results of the information and publicity measures in terms of visibility and awareness of the operational programme and the role played by the Commission including achievements, examples of good practice and significant events. Communication also has to be evaluated in detail for the final report of each programme.

Some practical examples

Evaluating your communication activities starts with regular monitoring. throughout the whole programme lifecycle. For example, during each event you should keep track of the list of participants and contacts, prepare feedback questionnaires or conduct a brief online survey afterwards. To monitor online tools can be done by calculating the website hits or the number of friends and feedback you get of your social media activities. Other examples include the drafting of a diffusion list with the number of publications disseminated to each target group. Or you may even think about getting more detailed feedback through surveys or focus groups. 

Media monitoring is another important part of the evaluation of communication and should be part of your overall media outreach strategy. This can be done quite easily through the setting up of a press clipping archive including screenshots of the websites that mention your programme. If you cover a large programming area (e.g. transnational programmes), you may also think about hiring a professional media monitoring service, although this can be costly. Your archive should also consist of radio and TV clippings and communication products of your projects.

Tools

See the practical fiches developed by DG Regional Policy for evaluation of communication on Inforegio.

For more general information about evaluation you may also refer to our special evaluation section.