Impact indicators

Specific sets of impact metrics on the same theme are combined into a single indicator. Impact scores – out of 42 - are calculated based on a set of rules which takes these combinations into account. A higher score means the project is carrying out more activities related to the theme of the indicator, and is, therefore, more likely to have a higher positive impact in this area.

For example, the first indicator described below is that of “activeness”. Three impact metrics are combined to formulate this indicator. A high score means that the project is carrying out more activities related to “activeness”.

It is possible that the project may receive recommendations for indicators the project is not really interested in, or focused on. For example, the project may not have considered its economic productivity, and may not be interested in improving it. That’s OK. No project is obligated to take action on any of the recommendations MICS proposes.

society
governance
the economy
science and technology
the environment

Society

To measure the impact of citizen science on society, we initially considered two indicators: activeness and involvement.

Activeness

Definition:  Activeness is the level of cognitive engagement; where “active” indicates that the participant requires full cognitive engagement during participation, and “passive” suggests there is no engagement beyond setup. 

Questions: 

  • How much responsibility is offered to the participants? 
  • Are the participants satisfied with the process of participation in the project? 
  • Are participants aware they are contributing to a research project? 

Why we use it:  

Activeness is included in the ECSA characteristics of citizen science and explained in the explanation notes of this document. 

“Participants need to be aware of the contribution and participate actively and intentionally, as this is necessary for cases where the information that participants produced is not directly used by the project, but only as a secondary use of data (e.g. reusing images that people share on a social networking site). We recommend transparency about roles and expectations…. it is highly recommended to be open and transparent about choices that were made about the roles of participants. The project owner has responsibility for communicating that the participants are contributing to research.” 

Activeness is further discussed in the associated peer-reviewed publication, “Contours of citizen science: a vignette study”. 

Indicator formula

All inidcator scores are calcualted by summing the weights of the answer options selected. Weights for the activeness indicator are given below:

Question number Question text Answers Weights
Society 4 How much responsibility is offered to the participants? (with options depending on interests, availability and knowledge). Not much 0
       Something in the middle 8
A lot 16
I don't know 4
Society 8 Are the participants satisfied with the process of participation in the project? Yes, and it has been measured 12
       Yes, but it has not been measured 10
No 0
I don't know 4
Society 12 Are participants aware they are contributing to a research project? Yes 14
     No 0
I don't know 4
Recommendation score thresholds
Recommendations Lower score Higher score
The activeness of participants within a project is an important aspect of citizen science. Efforts should be made to make participants aware they are contributing to a research project through clear communication channels, and to offer them the opportunity to be responsible for their activities. If the project has not measured their degree of satisfaction in the process, it might want to consider to consider investigating this further using this paper as a starting point. 0 12
The activeness of participants within a project is an important aspect of citizen science. Activeness depends on participants being aware that they are contributing to a project, having a lot of responsibility in the project, and being satisfied with the process of participation. This project should ensure that all aspects of activeness have been considered. 13 33
The activeness of participants within a project is an important aspect of citizen science, and this project has made great efforts to ensure participants are aware they are contributing to a research project, have responsibility in the project, and are satisfied with the process of participation. Great job! 34 42

Involvement

Definition: Involvement is the degree of participation in different stages of a process. 

Questions: 

  • In which phases of the project do participants play a role? 
  • Are participants offered multiple project activities which they can take part in? 
  • Are participants offered different levels of involvement in each project activity depending on their interest, availability and knowledge? 

Why we use it: 

Involvement is included in the ECSA characteristics of citizen science. 

“Research involving citizen science can take many forms, and the roles of the participants can include, for example: identifying a research question, collecting or analysing data to support or refute a hypothesis; monitoring environmental or health conditions for management or policy outcomes; and creation of generic data within a domain to support a wide range of research questions.” 

Involvement is further discussed in the associated peer-reviewed publication, “Contours of citizen science: a vignette study” as well as Kieslinger’s paper, “ The Challenge of Evaluation: An Open Framework for Evaluating Citizen Science Activities”. 

Return to top

Governance

To measure the impact of citizen science on governance, we initially considered two indicators: policy and the sustainable development goals. 

Policy

Definition: Policy is a deliberate system of guidelines to guide decisions and achieve rational outcomes. A policy is a statement of intent and is implemented as a procedure or protocol. 

Questions: 

  • Does the project's data or findings lead to the enforcement of existing regulations or policies? 
  • Which policy frameworks does the project consider? 
  • Does the project explicitly inform any governmental policy process? 
  • Does the project have any explicit impact on external organisational policy? 

Why we use it: 

Policy is addressed in the ECSA characteristics of citizen science where it is covered under “Links to decision-making”. 

“Citizen science projects may include an intervention into the current state of affairs, such as local decision making. This might happen in activities that fall under banners such as participatory action research, community science, or addressing environmental injustice.” 

In the explanation notes of this document, it is also noted that “Citizen science can be used in cases where the participants are concerned with an issue and want to actively change the situation, be it concern over public health, medical support to a group of patients, or addressing a pollution issue.” 

Sustainable development goals

Definition: The Sustainable Development Goals (SDGs) or Global Goals are a collection of 17 interlinked global goals designed to be a "blueprint to achieve a better and more sustainable future for all". The SDGs were set up in 2015 by the United Nations General Assembly and are intended to be achieved by 2030. 

Questions: 

  • Is the project team aware of what the Sustainable Development Goals (SDGs) are? 
  • Is the project at all related to the Sustainable Development Goals (SDGs)? 
  • Which of the following Sustainable Development Goals is the project related to? 
  • Does the project include data which match a specific indicator of a Sustainable Development Goal? 
  • Does the project contribute data to the official reporting for a Sustainable Development Goal indicator? 

Why we use it: 

Aside from the SDGs becoming the latest buzzword in citizen science (see Fritz et al., 2019, for example) and MICS’ commitment to considering the SDGs in our project Grant Agreement, the United Nations’ 2030 Agenda for Sustainable Development is an ambitious plan for “people, planet and prosperity”, aimed at achieving a sustainable future for all; what’s not to like about it?! 

Return to top

Economy

To measure the impact of citizen science on economy, we initially considered two indicators: economic productivity and financial sustainability.

Economic productivity

Definition: Economic productivity measures output per unit of input, such as labour, capital, or any other resource. It is often calculated for the economy as a ratio of gross domestic product (GDP) to hours worked.

Questions: 

  • Does the project explicitly improve economic productivity through diversification, technological upgrading or innovation?

Why we use it: 

Economic productivity relates to SDG 8.2 Diversify, innovate and upgrade for economic productivity: Achieve higher levels of economic productivity through diversification, technological upgrading and innovation, including through a focus on high-value added and labour-intensive sectors

Financial sustainability

Definition: Financial sustainability is the assessment that a project will have sufficient funds to meet all its resource and financial obligations, whether the fund continues or not.

Questions: 

  • Does the project have explicit plans to sustain its activities after the current funding received has ended?
  • Does the project have an explicit exploitation plan?
  • Does the project require recurring investments in technology (for example, software licences or app/platform maintenance) that affect its long-term sustainability?
  • What is the estimated, approximate cost per observation (in £, € or $) (observations as defined by the project)?

Why we use it: 

Citizen science is frequently hailed as being an inexpensive alternative to traditional science. Palmer et al. note, “with its relatively low cost centred on non-recurring investments, citizen science is inherently more scalable than traditional tools”. But financial sustainability is more than simply using cheap equipment - it’s also about planning ahead for the exploitation of outputs, for example - which is why this indicator is a little broader than Palmer’s original definition.

Return to top

Science and technology

To measure the impact of citizen science on science and technology, we initially considered two indicators: scientific productivity and interdisciplinary science.

Scientific productivity

Definition: Scientific productivity refers to the productivity of scientists in their research performance. In other words, the term concerns how much output scientists produce within a certain time period, or compared to the inputs that are utilized for the research.

Questions: 

  • How many publications, indexed by Google Scholar, resulted from the project?
  • How many citations have the publications produced by the project received, in total (according to Google scholar)?
  • What is the highest impact-factor (or impact-index) of the publications produced by the project? (Just Google: "[journal name]" impact factor
  • Has the project supported student's dissertations or theses?
  • How are the project outcomes shared?

Why we use it: 

Scientific productivity is included in the ECSA characteristics of citizen science under the heading of data and knowledge generation.

“Citizen science, scientific, academic, and policy-oriented research can include different forms of data and knowledge generation, including novel data generation, creation of new analyses, or production of new knowledge in written and other forms. The knowledge produced in such projects should aspire to disciplinary standards, such as appropriate data quality and quality assurance, the peer review of project publications and materials, or policy-relevant evidence that is fit for decision-making.”

Kieslinger et al. also note the importance of citizen scientists participating in publications, or having their engagement recognised.

Interdisciplinary science

Definition: Interdisciplinary science is the collaborative process of integrating knowledge/expertise from trained individuals of two or more disciplines.

Questions: 

  • Which disciplines are the focus of the project's research?
  • Does the project explicitly promote interdisciplinary ways of working?

Why we use it: 

There is evidence that interdisciplinarity is statistically significantly and positively associated with research impact (Okamura, 2019).

Return to top

Environment

To measure the impact of citizen science on environment, we initially considered two indicators: environmental footprint and environmental awareness.

Environmental footprint

Definition: The environmental (or ecological) footprint measures how fast we consume resources and generate waste compared to how fast nature can absorb our waste and generate resources.

Questions: 

  • Does the project take measures to decrease its material footprint?
  • Does the project take measure to reduce its polluting emissions?
  • Does the project have a procurement policy that is sustainable?
  • Do the project activities include pro-environmental actions g. litter picking?

Why we use it: 

Environmental footprint relates to SDG 9.4 Upgrade all industries and infrastructures for sustainability: By 2030, upgrade infrastructure and retrofit industries to make them sustainable, with increased resource-use efficiency and greater adoption of clean and environmentally sound technologies and industrial processes, with all countries taking action in accordance with their respective capabilities.

It also relates to SDG 12.2 Sustainable management and use of natural resources and SDG 12.7 Promote sustainable public procurement practices.

Moreover, even if a citizen-science project does not aim to have a positive impact on the environment, there are still ways in which the environmental footprint can be addressed; thus, this indicator is relevant to all citizen-science projects.

Environmental awareness

Definition: Environmental awareness can be broadly defined as the attitude regarding environmental consequences of human behaviour (Ham et al., 2016)

Questions: 

  • Does the project explicitly disseminate information on sustainable development or lifestyles?
  • Does the project educate participants on environmental challenges?
  • Does the project explicitly contribute to a higher awareness of, or positive attitude towards, the natural environment, on this planet or others?
  • Does the project lead to an increased stewardship of the natural environment among participants?

Why we use it: 

Environmental awareness relates to SDG 12.8 Promote universal understanding of sustainable lifestyles and SDG 13.3 Build knowledge and capacity to meet climate change.

Return to top

This project has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 824711.

Newsletter

Enter your email address above to receive updates on the MICS project. Please see our privacy policy for more information.

Creative Commons Licence
MICS science is open science. All the information on this website is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.