Is it a Lottery?
Improving Gender Fairness in Research Funding
Around 55 people from six European countries (Austria, Czech Republic, Germany, Netherlands, Sweden and Switzerland) attended the first GRANteD Stakeholder Conference that took place on February 24th 2020 in Vienna at Sensengasse 1 (Haus der Forschung). Find the Program here
Openings
Klement Tockner, president FWF,
stated in his opening words that “Diversity is the key driver for excellence” and introduced new instruments by which the FWF aims to improve the funding system. He also emphasized that the FWF is happy to be part of the GRANteD Project as a case study in order to know what works successfully and what has to be improved. “We are in the process of learning and we hope that gender equality is no longer a topic in 20 years because it is a fact.”
Wolfgang Polt, Director of Policies Institute of JOANNEUM RESEARCH (the institute that coordinates the GRANteD project),
stressed that while there has been quite some progress in addressing gender inequalities both on the level on individual researchers and institutions, structural differences still remain which have to be the subject of critical investigation and policy action. These often concern societal behaviour and norms, which are hard to tackle and change. GRANteD wants to shed light on these issues and brings together a very well qualified team of researchers to do so.
Helene Schiffbänker (JOANNEUM RESEARCH) and Peter van den Besselaar (Teresa Mom Consultancy) introducing the GRANteD project Framework and the Aims of Granted
Helene Schiffbänker, the coordinator, introduced the GRANteD project, presenting differences in success rates as a starting point for the analysis as well as some hypotheses explaining why success rates may differ. Here she focused on the need to control for past performance of applicants, look at potential gender stereotypes of reviewers and dynamics within panels as well as taking imbalances in the science system before applying into account. She stressed that GRANteD aims to identify if bias in grant allocation exists and if so, to find out which factors explain it. Therefore the GRANteD consortium analyses complex existing data sets on funding as well as new data generated when closely collaborating with selected research funding organisations (RFOs). As it is important to understand “what happens in ongoing grant allocations processes”, the GRANteD consortium will study panel decisions in more detail and will work closely with selected RFOs as well as with the GRANteD Stakeholder Committee and the GRANteD Scientific Advisory Board.
Find the presentation here.
One question raised by the audience was whether GRANteD would implement changes and evaluate those. This is not planned but the consortium will take this valid point into consideration.
Peter van den Besselaar provided some preliminary analysis of a case study that already started before the Granted project, but will be fully analyzed within the Granted project: the ERC Starting Grants case. He described the rich set of data that were obtained from the applicants and from the ERC, and the additionally collected data. Using these he showed for the applicants with a Dutch residence that the gender distribution is very different from the gender distribution among the potential applicants. This suggests that a large part of the gender differences may occur much earlier than in the grant selection process. After that he showed that – when controlling for past performance and several other variables, gender still has an effect on the scoring of the applications in about 50% of the panels, mostly in favor of men, but in various panels in favor of women. These panel differences show that there is a need for systematically studying the panel structure, panel procedures and panel practices, as that may help to understand the conditions under which gender bias may occur or not.
Questions from the audience suggested (i) that studying the panel level was seen as important, and (ii) that there is demand for strong methodologies to do so.
Keynote 1
Ulrike Bischler (Volkswagen Foundation)
Ulrike Bischler shared Volkswagen Foundation’s experiences with randomized selection process (lottery approach) for allocating research grants. Starting from common issues in peer review such as subjective influences (cognitive bias), group/social dynamics, limited panel sizes, risk aversion, which can have negative effects on merit assessment in grant allocation she then went on summarizing the experiences with the lottery approach. The Volkswagen Foundation reported that the newly used approach for allocating grants did not decrease the quality of applications. Initial internal skepticism vanished (jury members even became active ambassadors for the randomization strategy) and an increase in attention towards the Volkswagen Foundation could be observed within the scientific community. Currently an ongoing evaluation is in place to analyze the impacts of the lottery approach.
Elevator Pitches 1
Channah Herschberg from the Dutch Research Council NWO introduced the Dutch position paper ‘Room for everyone’s talent’ aiming to foster a broader, more inclusive understanding of talent. You can find the positions paper here.
Tina Olteanu from the Austrian FWF presented an alternative approach for selecting grantees within the program ‘1000 Ideas’
Interactice Session 1 Worldcafé
Discussions on peer review dis-/advantages, challenges, current state and future perspective
Table 1 – Marlene Hock: What are the strengths/advantages of existing peer review processes?
For most of the attendees, it was hard to formulate strengths or advantages of the existing peer review system; the discussion was ambivalent: if advantages were raised in the discussion, other participants perceived them as disadvantages. However several qualities of the process were mentioned and discussed. It was stated, that, if the reviewer takes the requirement and standards seriously and contributes effort to it, the review process works as good feedback for improving the work/publication. In this sense, peer review creates a dialogue, provides exchange of experiences and knowledge and lifts the quality of research. Sometimes even additional recommendations can be received in the process. Additionally the peer review process was described as an established, institutionalized system that is robust and ideally works quick and efficient. Since peer review is free from political interference, it supports the autonomy of research. In sum, participants agreed that the basic idea of a quality control by (more or less experienced) colleagues is good but the system still needs improvement.
Table 2 – Peter Van den Besselaar: What are disadvantages/challenges?
The following issues were raised and discussed as challenges within the peer review process:
-
- Not as objective as it pretends to be
- Quality check for peer reviewers
- Power plays in panels
- Enough time for reviewers
- Overburdening
- Review Criteria
- Finding reviewers
- Too much peer reviewed (peer review inflation)
- Superficial reviewers
- Peer review sometimes confused with censorship
- Quality of peer reviews only known after collected
- Interdisciplinary Project/Papers often fail
- Bias is given although not firstly evaluate the scientific idea but evaluate the person
- How to judge potential innovation/creativity?
Table 3 – Liisa Husu: Which measures are currently in place that contribute to fairer peer review processes?
The following measures were identified to be currently in place in (some) RFOs to contribute to a fairer peer review process (not including general policy statements on gender equality and fairness by many RFOs):
-
- Codes of conduct for peer review
- Policy principle according to which the success rates of women and men should correspond their application rate, based on the RFO principle that talent is equally divided among women and men in the applicant pool.
- Briefing the panel chairs
- Enlarging the pool of potential reviewers (not only full professors)
- Quota or target for gender balance in panels or among reviewers. It is reported by many RFOs that it more difficult to recruit female reviewers due to scarcity of women in the professoriate and due to strong international competition for women reviewers (because of increased gender equality actions in many countries). Women decline more often reviewer tasks than men do. Measures include setting up a reviewer database explicitly including female reviewers or asking panelists to always suggest both female and male reviewers.
- Applying non-blind peer review – enhances accountability of reviewers
- Clear review criteria communicated to applicants and reviewers, against which applications evaluated
- Academic age used instead of biological age in assessing the CV
- Gender dimension in the research content to be addressed in funding applications and evaluation
- Gender observers in panels, or diversity/gender quality officers present in meetings
- Gender bias training of peer reviewers, may include IAT test (diverse views on the impact)
- Increased transparency, e.g. providing peer reviewers access to other peer review reports
- National network of RFOs around gender equality and gender mainstreaming
- Quality monitoring of reviewers, small part of a funding programme reserved for monitoring
- Regular statistical monitoring of success rates by gender of PI, disciplinary area, academic age, funding instrument, and gender composition of research teams. This information is openly published and accessible on annual reports and RFO website
- The measures are often included in the gender equality plans or gender equality strategies of the RFOs
- Gender equality reviews commissioned by RFOs
Table 4 – Helene Schiffbänker: The future of Peer Review: how to advance and redesign it?
It was discussed that open reviewing or crowd reviewing could be options to enhance the peer reviewing system, yet being discussed controversially: While these forms bring more transparency on the one hand they might be less favored by top scientists who are afraid that their ideas get stolen. The further professionalization of reviewers was another main issue discussed: specific briefings, constant training, (better) payment or maybe having reviewing as a specific career track were discussed. On a more general level the competitive approach to allocate research money was discussed as too far developed, demanding too much time for proposal writing to too little time for following the researchers’ curiosity.
After lunch, the conference’s focus shifted from grant selection mechanism (lottery, peer review) to the gender dimension in research funding, addressed by gender stereotypes in the science system.
Keynote 2: Ruth Van Veelen (Utrecht University)
“Academics as superheroes: How the exceptionally masculine occupational stereotype in academia affects women (and men)”
In her Keynote Ruth van Veelen explained the different gender gaps (leadership, rewards, resources) in Dutch academia, refuting the common “women don’t ask” explanation. She showed that on opposite, women often negotiate more than their male colleagues do, as women are more often in precarious positions and get less promotable tasks. Van Veelen explains that the joint reason for existing gender gaps in academia still lies within the stereotypical perception of the researcher as agentic, brilliant and genius, which suggest “less chances to fit-in” for (femal) researchers who don’t rise to this expectations .
Finde the presentation here.
One participant stated that the ‘superstar model’ may have a material basis, and that it might not only be a social construct that can/should be changed.
Elevator Pitches 2
Existing tools for addressing gender bias
First Karin Dögl from the Vienna Business Agency presented a recently developed e-learning tool for tackling gender bias as well as other forms of bias, stressing the challenge that the online-tool addresses a target group with very divergent awareness levels.
Sandra Hesping from the Technical University in Eindhoven presented the recently implemented Irene Curie Fellowship Program, which opens positions only for women within the first six month – and opens for both sexes after this period – an approach that gained a broad international media coverage.
Alexia Bumbaris from the Austrian Research Promotion Agency (FFG) presented experiences with the implementation of gender criteria in the FFG funding schemes, that are implemented within all funding programmes since 2011.
Find the presentation here.
Interactive Session 2 Worldcafé
Continuing the discussion focusing on genderbias in peer review
Table 1 – Torger Möller & Stefan Hornbostel: Where does gender bias occur? What are the sources of gender bias in peer review?
It was discussed at which different stages gender bias can occur in the grant allocation process. In this respect, not only the research funding organizations (RFO) but also the research performing organizations (RPO) were taken into account. The following table gives an overview of the different stages of the process and possible sources of gender bias:
| Stages of the process | Potential sources of gender bias |
| 1. Researchers as potential applicants (RPO) | – self-selection
– academic socialization |
| 2. Applicants (RPO) | – application support structures
– working conditions (position, age, tenure) |
| 3. Application (RFO) | – application formats (early career or excellence grants)
– requirements (mobility) |
| 4. Review (RFO) | – different assessment of “academic personality” and care work
– (non-)blind peer review and interviews |
| 5. Decision (RFO) | – extension for parenthood
– “old boys”-networks and nepotism |
Table 2 – Peter van den Besselaar: What measures for addressing gender bias in research funding are currently in place? What are their strengths and weaknesses?
-
- Neutral moderator
- Gender Bias trainings for staff
- Gender Bias briefings for reviewer
- Gender criteria in selection process for all projects
- Women finding programs
- Check lists
- Gender neutral language (software check)
- Women-only programs
- Gender aspects in the topic
- Structured scoring dimensions
- Observers
Table 3 – Helene Schiffbänker: What are your experiences with gender bias trainings? What are their strengths and weaknesses – under which conditions are they working well?
Participants on this table discussed that it is crucial to link theoretical explanations to personal experiences.
Another discussion was about the implicit association test (IAT) which was practiced by serveral participants (trainers as well as trainees). Both reported that this testing implies risks as participants sometimes become helpless, insecure how to further behave or even angry. It was stressed that it is important to let participants know that stereotypes are no problem as such, but you need to be aware to have them and it takes time to change them, like changing habits always is an effort.
Table 4 – Liisa Husu: Future ideas how to tackle gender bias in research funding
This table was engaged with a lively discussion of various topics on future ideas to tackle gender bias in research funding, not only limited to RFOs but also taking up more broadly the culture and values of academia and early gender equality training in RPOs. Key issues discussed included:
-
- Discussion of the need for change of culture of academia – from competition to cooperation vs. competition seen as inevitable
- Setting may matter: many measures to diminish/remove gender bias have not yet been tested in different national and organizational contexts – there is a need to test good practices in new settings (e.g. gender observers, gender quota of panels, reviewers, decision-making boards that is applied only in some contexts)
- Strong support for putting gender equality training early on in curricula and researcher training in RPOs, integrate gender equality training in RPO and RFO leadership training
- Panel chairs should go through gender training, or panels could have “neutral” chairs with gender expertise; role of panel chairs was considered crucial
- Use dilemmas and ambiguities around gender equality among actors effectively in gender training
- Other, new criteria that were taken up were outreach, care responsibilities, intersectoral mobility, diversity
- More generally, valuing care in and outside of academia was taken up – academia needs to overcome “carelessness” as a basic value
- More double-blind evaluations were suggested but many feasibility problems are included
- Intersectoral mobility (between academy-industry-government sector research) should be considered in evaluation – sector-specific merits tend often to be overlooked
- Measures to get dropped-out researchers back in were suggested which could bring in new skills, perspectives and more women
Impressions
