Skip to main content
SearchLoginLogin or Signup

1. Evidence Based HRM

Published onSep 19, 2022
1. Evidence Based HRM
·

Abstract

How can practitioners learn from organizational problems related to people in the workplace and systematically use the best research evidence in the field of human resource management, in order to develop effective HR policies and practices in their organizations?

Key Theories
Evidence-based management
Decision-making as bounded rationality

Evidence-based HRM Process
Understanding a problem
Formulating a question
Local and external evidence
Evaluating evidence: Validity, reliability and generalizability
Generating alternative solutions
Considerations for implementation Evaluation

Many of the problems and challenges that organizations face, involve people in workplaces. The performance, effort, ideas, and collaboration of people in organizations contribute to achieving organizational goals. Work matters for individuals as it provides meaning, satisfaction and income to support their lives and those they care for. Work and organizations contribute to the wealth of communities and the national economy. People who work in organizations can think, act and feel. They are not machines, but they make choices about every aspect of work behavior. The attitudes and behavior of people in workplaces may also cause headaches in organizations: resistance to change, demotivation, a lack of skilled employees, authoritarian supervisors and power politics can hinder productive and happy workplaces. All kinds of people-related issues need attention to keep organizations on track to meet their goals. The continuous change of societies, industries and organizations poses additional challenges to managing people. Managing people in workplaces is never complete, as there will continuously be new challenges and solutions to deal with. Decision processes about large and small interventions to keep organizations socially on track are at the center of this chapter. In particular, it explicates a process that leads to effective, evidence-based decisions about HR activities to solve and direct people-related issues in organizations.

The topic of interest for evidence-based HR activities is the employment relationship. This is an exchange relationship between an employee and an organization in which employees provide labor and employers reward employees for their contribution. Employment relations commence after a negotiated exchange between employees and employers that involves an employment contract as well as a less clearly defined social relational agreement. The nature of the exchange is influenced by the context of employment relations, which consists of organizational goals, peer and team processes, labor market dynamics, global competition, legislation, power relations, and a society’s culture, politics and wealth (Johns, 2006). These all influence employment relations in an ongoing dynamic interaction, which can become quite complex to manage. Human resource management is the sum of all strategy, policy, procedures and day-to-day acts that together aim to guide employment relations in organizations towards the goals of organizations, while ensuring alignment with various contextual conditions such as organization characteristics, industry dynamics, competition, labor markets, legal and institutional settings, and societal dynamics (Jackson et al., 2014). Zooming in, human resource management is made up of distinct human resource practices. Human resource (HR) practices are all the policies and procedures used for managing employment relations. The emphasis of the word ‘use’ accentuates that an HR practice is more than a written policy – it is what people in workplaces experience. HR practices are not the sole domain of the human resource management department. Managers, teams, project leaders and employees themselves all initiate, use and change HR practices. They can all be decision-makers in the choice to use an HR practice. All of these users will benefit from knowledge about effective HR practices for people in workplaces.

There is a large number of distinct HR practices with different purposes. Figure 1 gives an overview of HR practices along three stages of the employment relationship: entry, work and transition. In the entry phase, people develop from being an outsider to a member of an organization. After engaging in the employment relation, a period of productive work commences. Finally, at some point, the employment relationship may change or end, due to transitions in organizations, careers or life stages. HR practices usually serve to realize more specific outcomes compared to the entire human resource management system. Performance issues, diversity, labor shortages, organizational change and innovation are just some of the more specific domains that can strategically improve with targeted HR practices, as will be illustrated in the next chapters. For each outcome, there is a choice of HR practices that can all have some effect. The question is which practice to choose.

Consider, for example, recruitment, the domain of HR practices aimed at creating a pool of qualified applicants from which an organization can select their employees. Recruitment involves choices about who to recruit, where to recruit and how to recruit (Breaugh, 2013). Does the organization aim to recruit graduates or experienced professionals? At which educational levels? Locally, nationally, or globally? Via word-of-mouth, campus tours, or by using internet analytics? These are just a few examples of choices that should lead to finding qualified candidates who want to work in the organization. Factors like characteristics of the job, job requirements, budgets, norms and expectations of recruits about professions and organizations all influence and constrain choices about recruitment practices. The more decision-makers understand the recruitment aims, the means and characteristics of the organization and its context, and the effectiveness of various recruitment tools, the better these choices will be, thus resulting in a recruitment approach that will bring a good quality pool of applicants.

Of course, recruitment is just one example, and similar questions can be asked in the choice of any HR practice displayed in Figure 1. Choice, however, is not always the outcome of a rational decision-making process. Rational decision-making means considering all information and weighing it according to some criteria before taking a decision. Due to the involvement of different actors who sometimes have conflicting interest in the HR practice, power and organizational politics, and simply because it is impossible to have all the information to make a good decision, the choice for HR practices is one of bounded rationality; an effort to the best decision given an imperfect understanding of reality (Sahakian, 2020).

Bounded rationality is a fact, but there are strategies to add more rationality to decision-making. Unfortunately, decision-makers often neglect these strategies or are just not aware of them. In the rush of day-to-day business, decision-makers often solve problems in a haphazard way. Imagine that some superior raises a people-related problem (We have serious budget problems! Our talented staff are leaving! Our competition is innovating, and we are losing our customers!), and calls for immediate action. The superior may immediately suggest an intervention to quickly solve the problem. However, such quick-fix decisions are likely to produce HR practices based on personal experience, outdated management theories and management fads, for which it is not evident that they are really effective in solving the problem (Briner, 2007). Although the benefit of a quick solution is that it shows that someone is willing to take immediate action, there are also severe risks to the use of quick fixes.

The risks begin the moment a problem is noticed. The first step, exploring the problem and seeing what is truly the matter, is ignored in the rush to solve the problem. The risk of skipping this step, is that the choice of HR practices may not really tackle the underlying issue. Instead of solving a problem, quick fixes often lead to new problems that need a new quick fix and so on. To prevent quick fixes and be able to suggest more sustainable interventions, decision-makers should ask questions like ‘What is the underlying problem?’, ‘What are the affected outcomes?’ or ‘Which data was used to inform us about the size of the issue?’ Such questions will lead to knowledge that is needed to compare alternative solutions. Hence, quick-fixes are risky decisions that can do as much harm as doing nothing at all (Briner & Walshe, 2014). Even worse, in the rush of solving the next issue, a quick fix is seldom evaluated for its benefits, thus increasing the risk that the same faulty solution will be recycled a next time.

This book advocates the practice of evidence-based HRM as a method to support decision-making for effective HR practices by taking a better look at the problem and its causes and by taking the context of a problem and the actors involved in it into account. It is a decision-making process that starts with identifying a problem and making an effort to understand the problem and its underlying causes in the work context. Then it entails finding evidence about effective HR practices for such questions from theory, research and experts. By combining these sources of knowledge, an HR practice can be chosen and tailored to ensure that the problem is fixed in a viable manner. Evidence-based HRM proposes a method to take decisions in a more rational way, while simultaneously recognizing the importance of accounting for power and politics in organizations.

The structure of the remainder of the chapter is as follows. First, the background of evidence-based HRM theory is explained, followed by an explanation of all elements of an evidence-based HRM decision-making process. The process is illustrated by a hands-on example concerning the development of HR practices to ensure equal pay for equal work in a public government organization.

A diagram showing three columns with HR practices as they  relate to employment relationship phases, labeled, left-to-right, "Entry", "Work", and "Transition".

The "Entry" column has the following HR practices:
 - Labor relations,
 - Labor analytics,
 - Forecasting,
 - HR strategic goals,
 - Workforce planning,
 - Organizational design (roles, reporting lines, outsourcing)
Compensation policy,
 - Job analyses,
 - Job descriptions,
 - Employer branding,
 - Recruitment,
 - Job offering,
 - Negotiations,
 - Employment contract,
 - Selection,
 - Flexible labor contracting

The "Work" column has the following HR practices:
 - Cascading performance goals,
 - Setting tasks and goals,
 - Performance appraisal,
 - Onboarding,
 - Training and development,
 - Teamwork,
 - Information sharing,
 - Employee participation,
 - Employee involvement,
 - Career development,
 - Leadership,
 - Safe workplaces,
 - Health and wellbeing,
 - Flexible working,
 - Workplace design,
 - Job design,
 - Social climate,
 - Inducements for motivation,
 - Inclusion of vulnerable workers,
 - Preventing discrimination,
 - Work-family balance,
 - Management across countries,
 - Telework

Finally, the "Transition" column has the following HR practices:
 - Research motives of voluntary leavers,
 - Employability,
 - Transitions between jobs,
 - Flex pool,
 - Workforce reductions,
 - Mergers and acquisitions,
 - Cultural change,
 - Dismissal,
 - Life events,
 - Retirement
Figure 1

Scoping HR practices on employment relationship phases

Theory

The origins of evidence-based management

Evidence-based management advocates that practitioners do some research into the nature of a problem, gather information from different sources within the organization and from experts and research sources before they suggest an intervention (Barends et al., 2014). This call for using insights from scientific evidence by practitioners in organizational practice is not new. For decades, scientists have been blaming practitioners for ‘not using research insights in practice’ (Rynes et al., 2002) and, in reply, practitioners blame academics for ‘producing tons of research findings that are too hard to find and too far away from the daily needs of practitioners anyway’ (Jeffrey & Sutton, 2006). Blaming each other does not solve the dispute, and therefore the emphasis shifted towards strategies to close the gap between practice and research.

One strategy is that researchers make more of an effort to explicate the practical implications of their findings and to make these available for practitioners. Most scientific journals nowadays ask researchers to do so. Many research articles have a dedicated paragraph or box called ‘practical implications’ for practitioners. Another beneficial development is ‘open science’, which makes research publications easier to access for people outside academia. The past years have witnessed a tremendous increase in open access publications. In principle, research findings are available as a source of knowledge to make better decisions.

Another strategy to bridge the science-practitioner gap is co-creation. A scientific study in which practitioners are closely involved in the design and execution of the research in their work contexts leads to learning on both sides. Practitioners learn about the value of research findings, and researchers learn about the scope of practical implications (Rynes & Bartunek, 2017). This situation is still exceptional, and usually practitioners will need to find their way to scientific evidence on their own initiative. This may be challenging because there is an abundance of information that tends to be very technical and difficult to read, even despite all the initiatives to open up science for practice.

Evidence-based management developed because of the persistence of the research-practice gap. It took the debate one step further by actively helping practitioners find and use scientific evidence. Before explaining how evidence-based management can serve in this regard, its origins will be explained first. The roots of evidence-based management lie in the development of evidence-based medicine, which began a few decades earlier. The need for evidence-based medicine was called for by both patients and medical professionals. Patients expect that their medical staff always apply the most recent research findings to treat their illness. However, medical staff lacks the time to continuously keep up with all research findings. Even if doctors and nurses would find time to look for the latest research evidence on treatments, chances are that they become overwhelmed by the amount of insights produced by researchers. There is so much research evidence that it is difficult to find the right information for every question. Relying on a ‘quick-fix’, e.g. a treatment that worked ten years ago, is tempting. Luckily, physicians have sworn a medical oath in which they promised to keep their medical knowledge up to date. The connection between science and practice in medicine has always been strong, which explains why the concern about finding and using evidence is prominent here.

In the 90s, the Cochrane Institute developed a systematic approach for comparing research findings. The institute produces overviews of research evidence in an accessible way, so that medical staff can easily find the evidence of the most effective treatments for all kinds of health issues. Today, medical staff use the Cochrane library to gain easier and faster access to the best scientific evidence for treatments. In this way, the faith of patients in that their physicians use the most effective treatments to cure patients remains justified.

The use of research evidence by managers to inform management decisions is much less common than it is for medical staff. Nonetheless, applying research findings to organizations could lead to better ‘cures’ as well. However, comparable to the situation of medical staff, managers’ time is limited and often managers do not know where to look for the right advice needed in particular situations. Therefore, similar to the Cochrane Institute the ‘Center for Evidence-based Management’ was founded in 2010 to make management decisions more evidence-based (Center for Evidence-Based Management, 2010).

Although the mission of both institutes concerns the access and dissemination of research evidence to practitioners, to date the awareness and use of the Cochrane library amongst medical professionals is much wider than the awareness and use among managers of research evidence collected and disseminated by the ‘Center for Evidence-based Management’. One cause could be that unlike medical staff, managers have not taken an ‘oath’ to practice up-to date knowledge. Therefore, it may happen that managers rely on knowledge they obtained at college in the 1980s, and never seriously updated it afterwards. Nevertheless, since the field of evidence-based management is newer, it may just take a bit more time before practicing evidence-based management becomes part of the professional management culture. Training students and practitioners in evidence-based management principles and processes is a first step (Rynes et al., 2014).

Evidence-based HRM is a specification of evidence-based management principles to the domain of employment relationships. It is a conscientious, explicit, and judicious decision-making process to address important people-related issues in organizations by combining the best available research evidence with measurable data and professional knowledge available in organizations (Rousseau, 2006; Sackett et al., 1996). Core elements in this definition of EBHRM are, based on Briner et al. (2009):

  1. It is about decision-making by practitioners who consciously apply their expertise and judgment;

  2. they use evidence from the local context to which the decision applies;

  3. they critically evaluate the best available external research evidence;

  4. and they take perspectives of people who might be affected by the decision into account.

This means that evidence-based HRM is not about ‘applying best practice’. Best practice assumes that there is one best way of doing HR in all organizations (Delery & Doty, 1996). However, where equally effective HR practices are available to solve a problem, practitioners may evaluate the use of each alternative in light of means and requirements of the organization in question. For example, paying high rewards to motivate employees is not affordable for small business owners. However, the short communication lines in smaller organizations provide many other advantages to easily build a culture of trust and involvement that also motivates employees (Drummond & Stone, 2007). Hence, specific organizational characteristics (financial means, organization structure, the type of work) should lead to the selection of specific practices. It is also not about ‘benchmarking’, which essentially holds that practices are compared between organizations. Benchmarking leads to copying HR practices from successful competitors, without much consideration for the precise needs of the organization (Paauwe & Boselie, 2005). In practicing evidence-based HRM, practitioners evaluate a variety of options to solve specific problems in a specific context by taking research evidence and the organizational context into account.

Figure 2 visualizes the steps in an evidence-based HRM decision-making process. The following sections will explain decision-making theory and subsequently elaborate on the steps in the model in more detail.

A figure showing the evidence based HRM process. 

First, try to identify the problem.

Then, gain an understandig of the problem. Ask yourself:
- What is happening?
- What seems the underlying problem?
- What are the affected outcomes?
- Which data was used to understand the
nature and impact of the problem?
- What would improve in objective measures
if the problem was fixed?
- Who are involved in the problem?
- Which domain of HR practices

After that, formulate a focused question:
- Which people in the organization are the
target group for the HR practice?
- What kind of HR practice
- Which specific outcome?

Then, search for the evidence:

You should collect local evidence:
What do people in the organization know
about the problem and about interventions
that may help?
- Quantitative evidence: Administrative
data about the problem (causes,
outcome); Questionnaire data
- Qualitative evidence: What are views
of stakeholders? Experiential
knowledge of management & HR,
what has been tried before?

You should also collect external evidence:
What do we know about the science of
effective interventions for such problems in
general?
- Expert knowledge
- Theory
- Research publications (empirical
studies, systematic review, meta
analysis)

Then, evaluate the evidence:
- Check validity, reliability, generalizability

Generate alternatives:
- Find solutions to the problem that meet the local
and external evidence

Decide, Prepare and implement the solution:
- Decision making procedure, agreement by
MT and Works council, ethics

Finally, evaluate the HR practice:
- Track use and effect on the outcome
- Adjust

These steps are not always linear. You can and should go back and forth between the steps.
Figure 2

The evidence-based HRM decision-making process

Decision-making as bounded rationality

Decision-making theories consider the process of actions that lead decision-makers to take a decision. In this book, the interest concerns strategic decision-making. Strategic decisions involve using resources (time, people, money) and are supposed to lead to some substantial outcome that matters for (a part of) the organization (Eisenhardt & Zbaracki, 1992). In human resource management decisions, substantial outcomes can range from pure business results (profit, innovation, strategic change) to ensure that a business adapts to its environment (labor markets, employment relations, society) or to improve the work and lives of employees. The structure of the next chapters in the book follows these substantial outcomes, starting with the business case for HRM, followed by demands imposed on HRM by the organizational context, and finally the employee perspective on HRM.

Depending on the school of thought, there are different views on decision-making processes. The first is rational decision-making theory. Rational means that decision makers who engage in a decision-making process first have an idea about what they want to achieve with the decision and then use some methodology to gather appropriate information and weigh various alternatives before deciding on the best solution. In figure 2, the top-down steps from defining a problem to implementing a solution illustrate a rational decision-making process. Rational decision-making is likely to improve the quality of decisions.

The problem with rational decision-making is that people are not machines. With their limited knowledge and their preset cognitive processes, people are not able to process information neutrally, nor do they have the capacity to know everything. The extent to which people are able to take rational decisions is limited by their cognitive processes, including their understanding of the problem, their preferences and blind spots, and by their social context, including access to knowledge in others and their own position in organizational power, politics and conflicts (Eisenhardt & Zbaracki, 1992). Early management scientists already doubted the capacity for rational decision-making in organizations. In 1947, Herbert Simon therefore introduced the concept of bounded rationality (Cristofaro, 2017). According to bounded rationality theory, individuals are able to take rational decisions within the limits of their preferences, their social position and their understanding of the problem and the alternative solutions. Opposite to rationality is irrationality. Bounded rationality suggests that it is possible to take the best decision given an imperfect understanding of reality. Strategies for improving decisions under bounded rationality focus on explicating the decision-making process in a number of specified steps.

Evidence-based management proposes a strategy to improve the decision-making process because it proposes a sequence of diagnosing, understanding and gathering alternatives before jumping to a solution. It seems a sensible approach to human resource management to improve decisions about policies and practices about people in workplaces, which is flexible enough to incorporate new insights and changes along the way.

Research on the practice of evidence-based management shows that it is used in many management domains (Rynes & Bartunek, 2017). However, there are situations in which evidence-based management may be too time-consuming. One example is decision-making during crises, and another is decision making on minor, non-strategic daily issues. In both examples, evidence-based management can contribute to improving the quality of decisions in another way, by contributing to knowledge and developing evidence-based intuition. By using evidence-based HRM for strategic decision-making about people in organizations, it will contribute to building evidence-based knowledge in decision-makers (managers and human resource management professionals), which will benefit their understanding of solutions for minor issues. The knowledge developed by applying evidence-based HRM in non-crisis situations will also help decision-makers to develop their intuition for the quality of their decisions under time pressure like in crises (Pratt & Dane, 2007).

To conclude, evidence-based management and its application in human resource management provide a process to make better strategic decisions about people in workplaces under conditions of bounded rationality. The process is flexible and accounts for iteration, where information in each phase may require adjustment of the previous step. Ultimately, evidence-based HRM policies and practices will contribute to improved performance of organizations, better alignment of organizations with their contexts, and ultimately to the well-being of employees.

The Evidence-based HRM process

The steps in figure 2 describe an iterative decision-making process. There is a logical sequence of rational decisions in the process, beginning with identifying an issue, collecting evidence, generating alternative solutions and finally designing and implementing an HR practice. However, each step can reveal new information, which necessitates the project to move back to a previous step to maybe restate the issue or collect additional evidence. Issues such as changes that happen during the execution of an evidence-based HRM project, and simply not knowing everything at the start of the project, make that evidence-based HRM is characterized by bounded rationality. Some flexibility is required to move along the steps in figure 2 to develop the most effective HR practice.

Process part 1: Asking a focused question

The first steps in the process are exploring the problem and determining the problem statement. This involves asking questions such as ‘What is the problem?’, ‘What should be solved?’ and ‘Which outcomes are at stake?’ It also implies exploring the context of the problem, by asking critical questions about its relation to organizational values, stakeholder views and organizational politics. This exploration helps to decide if the problem deserves an evidence-based management process intervention. After deciding that a problem is worth starting an evidence-based HRM process, a few more questions should be answered to narrow down the problem. The first task here is to define the outcome as specifically as possible (Barends & Rousseau, 2018; Dietz et al., 2014). It can help to phrase an outcome in indicators that would improve after an intervention, such as ‘client reports of employee service levels’, or ‘employee health’, or ‘the quality of suggestions by employees. A problem statement also benefits from an initial idea of the domain of interventions. This may be achieved by checking if the intended outcome is more performance-oriented, or related to demands imposed by the organization’s context, or by concerns about employee well-being. An example of a focused problem statement could be: ‘How can learning and development improve the service provided to clients by frontline employees in the customer service department?’. This is a focused question, because it 1) describes which observable and measurable outcome will improve, 2) it indicates a specific human resource management domain, and 3) it specifies the targeted group of employees.

Example project “Task force equal pay”

In the slipstream of a larger project on modernizing the compensation policies of a public government organization, human resource analytics reveal a pay difference between men and women in the organization of about 1000 Euros per month. A deeper exploration reveals that about 700 Euros of this difference can be explained, because men are better represented in higher paid jobs. However, after controlling for job level and years of experience, an unexplained pay difference of about 300 Euros per month remains. It looks like there is pay inequality between men and women working in the same area of activity and in the same organization. Although pay inequalities are a persistent problem that occurs in many organizations around the globe (International Labour Organization, 2018), the senior management of the public service organization regards this unexplained pay inequality as problematic for several reasons. First, there is a risk of legal prosecution. Constitutional law prohibits pay inequality based on legal grounds for discrimination, of which gender is one. Second, senior management believes that a public service organization has a societal responsibility to treat all employees equally. Also given its human resource management strategy to advance the careers of minorities in the organization, they consider the pay gap a strategic employment issue in need of repair and prevention for the future. The management composes a task force that is asked to design sustainable HR practices for this problem. In a problem statement, the question was: “How can pay procedures prevent pay differences between women and men doing the same work at the same level in this public service organization?”

Process part 2: Collecting evidence

The next step is to explore the causes of the problem from multiple angles. The word ‘evidence’ needs some elaboration in order to understand what types of information can be used. There is a common understanding of the meaning of evidence. In courts, for example, evidence refers to information that can be checked or proven, evaluated and weighed, before reaching a verdict. In medicine, evidence holds a similar meaning. There is evidence for the effectiveness of a treatment when there is proof that there is a causal relationship between the treatment and a patient’s recovery. Hence, evidence concerns proof of the existence of causality: an empirically observable relationship that suggests a mechanism through which a cause leads to an effect. In a management context, evidence concerns an understanding about all the causes that lead to the current state of an outcome, and about causes that may improve the outcome. The importance of understanding causal evidence is that it shows where interventions will most likely improve an outcome.

Insight into causes of an outcome can be derived from multiple sources. Evidence-based management generally discerns four sources of evidence: organizational evidence, stakeholder evidence, experiential evidence and scientific evidence (Baba & HakemZadeh, 2012). Organizational evidence consists of all sorts of management information that is present in the organization. Stakeholder evidence consists of all the opinions and perceptions about the problem and its causes by everyone who is involved in the problem. These can be managers, customers, suppliers and of course employees. Experiential evidence is knowledge from previous problems and projects in the organization. Grouped together, organizational, stakeholder and experiential evidence make up the local evidence. Scientific evidence is what is known about the problem and its causes in science. Since this evidence is not tied to the organization itself, it is called external evidence.

Local evidence

Local evidence is systematically gathered data in a particular organizational setting with the aim to inform local decisions (Dietz et al., 2014; Rousseau, 2006). Local evidence helps to build a thorough understanding of the problem in question. The benefits of local evidence are threefold. First, by looking at the facts, decisionmakers will understand the details of the outcome. Moreover, if possible, the outcome can be linked to other data about the organization. Second, by exploring the views of stakeholders, decision-makers gain a better idea of the social dynamics that cause and sustain an outcome. Finally, it is important to look at what has been tried before to deal with similar problems in the organization, and at what has been learned then. Typically, collecting local evidence consists of some desk research (financial data, reports, questionnaires, outcome measures, administrative data), and some interviews with stakeholders (managers, employees, former project team members) and experiential experts connected to the organization (project leaders, HR manager, former consultants to the organization).

Example project “Task force equal pay” The taskforce for equal pay immediately realized that pay and pay inequality are sensitive topics in any organization. They would have to act carefully in exploring the true causes of the issue. It is one of these topics where strong opinions are held, and where there may be unintended victims if measures are implemented without careful consideration. The task force developed a project plan to collect local evidence from three sources. First, they used company records to link any potential variable in the personnel records to the current pay levels of all men and women in the organization. In doing so, the task force identified that the pay gap was larger in higher than in lower levels of the organizational hierarchy. Female managers appeared to receive systematically less pay compared to their male counterparts. This counted as the organizational evidence. Next, the task force sought advice with the legal department to understand how a court evaluates if a pay gap really exists. The legal advice constituted the experiential evidence. After this legal consult, the task force developed a systematic method to compare the history of pay decisions for a sample of same-level employees doing the same job. The method consisted of interviews with managers and employees, who were asked to recall how a pay decision was made. These interviews represented the stakeholder input. Among other things, findings revealed that women were less likely to negotiate about the salary that was initially offered by the organization compared to men. In addition, managers and HR professionals seemed to attach greater importance to the previous work experience of men compared to the experience of women, which resulted in higher initial offerings for men.

External evidence

External evidence is the evidence generated by systematic research of similar cases of cause-effect relationships. This type of evidence is not available in the organization itself, but is gathered from databases filled with research findings created by scientists. The amount of research in the domain of management and human resources is abundant. To prevent practitioners from relying on evidence from a single study or on new and exciting ‘breakthrough research’, EBHRM advocates that practitioners rely on a larger body of research to form a complete picture of external evidence. The types of publications that provide such overviews of research findings are so-called systematic reviews and meta-analyses. Both are systematic approaches that provide an overview of research evidence regarding a specific research question. As a meta-analysis is often part of a systematic review (but not necessarily so), first the characteristics of systematic reviews are explained before the additional features of meta-analyses are outlined.

Systematic reviews are characterized by a methodical approach to comparing research findings across different studies. Over the years, the procedures for creating systematic reviews have been refined to ensure information is more reliable and useful for practitioners (Moher et al., 2009). The approach generally involves the following steps: First, a search strategy is defined for finding research that is relevant to the research question. This may include decisions on search words, databases and journals citation indexes. Next, the abstracts of the research articles found in the search are checked to guarantee that the study indeed relates to the research question. The search strategy can be found in the ‘methods’ section of a systematic review article. Afterwards, the research articles resulting from this search are coded by different authors to make findings comparable and interpretable in case of different findings between studies. Apart from the variables relevant to the research question, the coding also involves the quality of the research, for example, characteristics of the research sample. This coding process helps to find patterns in conditions under which findings are perhaps stronger or absent. Then, the selected research articles are systematically compared. This can be done in a descriptive way that summarizes the approach and findings for each of the articles before synthesizing the overall findings from the studies, or in a quantitative way in which the research data of the original studies are combined into a new dataset that allows for statistically testing the average findings of all studies. The latter procedure is called a meta-analysis. A meta-analysis is a statistical procedure for examining the overall strength of findings across a number of studies. Thus, a meta-analysis is often a part of a systematic review. This is supposed to be the most stringent evidence for research questions. However, since qualitative studies and experimental studies are unsuitable for meta-analytical quantification, a systematic qualitative comparison of the available research still provides a powerful source for external evidence.

In a final step, the systematic review provides in its results and conclusion section insight into a) patterns across study results concerning the research question, b) potential sources of disagreement between studies, and c) new findings that have only come to light after looking at the aggregate results of the studies.

Overall, systematic reviews provide practitioners with the best available external evidence to inform their decisions on practical problems. However, practitioners need to be aware of possible biases caused for instance by the inclusion or exclusion of research such as the so-called ‘publication bias’. This bias is created through the circumstance that research gets published more easily if it confirms the existence of a relationship than if all hypotheses are rejected. A systematic review that only includes published research can therefore present a slightly misleading picture. Nevertheless, such potential biases are noticeable in the defined search criteria for the inclusion of research papers.

What if there is no systematic review? Although the number of systematic reviews and meta-analyses in the domain of human resource management is steadily growing, chances are that the precise research question in which a practitioner is interested has not yet been submitted to a systematic review of research. In such cases, practitioners can do their own ‘light version’ of a systematic review (Briner & Walshe, 2014). Figure 3 summarizes the steps of such a review (Briner & Walshe, 2014).

Figure 3 Steps in a do-it-yourself systematic literature review

  1. Define the research question in terms of a causal relationship.

  2. Determine which type of research is included in the search (quantitative, qualitative, case studies, experiments?)

  3. Determine where the research is likely to be found (which search engines)

  4. By going through all titles and abstracts of the search results, select only those research articles that fulfill the search criteria (and thus really relate to the research question).

  5. Evaluate the quality of the research in the selected studies.

  6. Synthesize the findings from the studies. Since the number of studies reviewed in a ‘light’ systematic review, this will be done by hand.

  7. Evaluate the findings in light of the limitations of the search.

Where to find external evidence? Apart from the question whether a research question has been addressed in a systematic review, another important concern for practitioners is how to access systematic reviews and research articles. Systematic reviews are published in academic journals whose access is often limited to scholars and people who have paid access through their institute’s library. Luckily, times are changing, and more and more journals provide open access to their contents. This means that content is downloadable from the internet without additional fees. In the meantime, initiatives like the ‘Center for Evidence-based Management’ help practitioners without institutional access to non-open access articles to gain access via membership arrangements.

Example project “Task force equal pay”

Members of the task force equal pay asked the support of a scientific researcher to help them provide an overview of scientifically proved causes that explain or prevent a gender pay gap, and potential solutions to reduce it. These are two examples of meta-analyses and large survey research that indicate some of the causes for a gender pay gap:

 Joshi, A. (2015). When can women close the gap? A meta-analytic test of sex differences in performance and rewards. Academy of Management Journal, 58(5), 1516–1545. https://doi.org/10.5465/amj.2013.0721

Auspurg, K., Hinz, T., & Sauer, C. (2017). Why should women get less? Evidence on the gender pay gap from multifactorial survey experiments. American Sociological Review, 82(1), 179–210. https://doi.org/10.1177/0003122416683393

These publications show that the pay gap exists to various extents, and that there are various causes like stereotypes and underrepresentation of women in certain jobs. Since the task force was particularly interested in finding out about effective interventions, the researcher suggested the following sources:

Castilla, E. J. (2015). Accounting for the gap: A Firm study manipulating organizational accountability and transparency in pay decisions. Organization Science, 26(2), 311–333. https://doi.org/10.1287/orsc.2014.0950

Leibbrandt, A., & List, J. A. (2015). Do women avoid salary negotiations? Evidence from a large-scale natural field experiment. Management Science, 61(9), 2016–2024. https://doi.org/10.1287/mnsc.2014.1994

A number of relevant findings followed from the external evidence. The three most important ones were 1) increase the transparency of pay decisions, 2) ensure consistent use of pay policies throughout the organization, 3) make managers accountable for the justification of pay decisions and 4) ensure equal representation of men and women in pay decision committees. These conditions level the ‘rules of the game’ for pay decisions for everyone and lead to a sustainable reduction of the gender pay gap.

Process part 3: Evaluating the evidence

Before, during and after collecting the data for local and external evidence, decisionmakers need to consider the quality of the evidence. Although Figure 2 suggests that assessing the evidence happens after collecting it, paying attention to what is used as evidence will facilitate this assessment considerably. The saying ‘garbage in, garbage out’ also applies to evidence-based management. Decision-makers can rely on a set of tools to evaluate the quality of evidence. Knowledge of these tools supports designing sound procedures for collecting local evidence and evaluating the quality and relevance of external evidence. Evaluating evidence involves an elementary understanding of research methodology. The text below presents the core indicators for evaluating evidence: validity, reliability, generalizability, and ethicality. Decision-makers with some training in research methods are skilled at finding good-quality evidence.

Validity

Validity means that the evidence is actually telling us something about the causal relationship between a cause and an effect. How can you be sure that the cause created the effect? First, the evidence should indicate that there is a relationship between a cause and an effect. This means that an HR practice (a cause) should relate to an observable change in the outcome (the effect), and when the HR practice is absent, such a change would not happen. Research evidence that involves experiments to test if a condition really leads to an outcome, or gathering data over a longer time (longitudinal research) more or less guarantees that there is temporal sequence between cause and effect. Contrary, things become less clear when the cause and effect are measured at one time. In this case, we cannot guarantee which variable is the cause and which one is the effect. When the evidence does not provide proof for causality, decision-makers should be careful in using such evidence. Preferably, practitioners look for additional evidence to find further proof for the existence of a cause-effect relationship.

Second, validity means that there are no alternative explanations for a causal relationship. To rule out alternative explanations, other variables that could account for the existence of a cause-effect relationship also need consideration. There are three strategies for decision-makers to rule out alternative explanations:

  1. Check the quality of measures by asking whether evidence is based on measures that really capture the meaning of constructs in the problem statement. This is the construct validity of the evidence. For example, there is a lot of research on employees who leave an organization on their own initiative (voluntary turnover). This is a difficult research topic, because it needs data from before employees leave (how did they like their job?) and afterwards, when they have left the organization. Much evidence from research on voluntary turnover uses reports of employees’ intention to quit while still in their current jobs, rather than their actual quit behavior (Griffeth et al., 2000). Measuring actual quit behavior is also complicated, because it matters whether employees quit of their own volition or on the initiative of their employer, for example in a redundancy. Intention to quit, voluntary turnover behavior and involuntary turnover are each different constructs, which are predicted by different causes (Hom et al., 2012). Checking information about construct validity is important, because if not, the research evidence is not relevant to the problem statement of interest. The quality of the measures is reported in the methods section of a research paper. When it comes to collecting local evidence, construct validity concerns the selection of data within the organization to gain knowledge about the problem, the quality of the questions asked to stakeholders, and the quality of the experience of the people who provide experiential evidence.

  2. Check the quality of research designs. The strongest evidence that rules out alternative explanations comes from longitudinal and (quasi-experimental) research designs. However, carefully conducting research to empirically observe causality is complex and although the amount of longitudinal and quasi-experimental research on HRM is steadily increasing, the majority of external evidence still depends on cross-sectional data. In-depth qualitative research on multiple cases is also relevant, because it can provide more insight into subtle social processes. A limitation of qualitative research is the small number of cases, which hinders its generalizability. In collecting local evidence, the quality of the research design lies in the decision who to interview. For example, managers will report differently on the causes leading to an outcome than employees. This is why evidence-based management suggests seeking local evidence from multiple stakeholders of a problem.

  3. Good theory. The best bet for cross-sectional studies to rule out alternative explanations is by providing good theoretical arguments that explain why a cause would lead to an effect (and not the other way around). There are good theories that explain relationships between HR practices and a variety of organizational outcomes. The next chapters each describe the essential theoretical paradigms in the field of human resource management. Good theory is valid when it is based on insights derived from many studies, because then it describes a generalizable mechanism that is valid in many situations. If there is time to do a thorough literature study, an up-to-date knowledge of theories about people in workplaces will improve decision-making.

Reliability

Reliability means that we can be largely certain that if we would repeat our research, we would find the same results. A quick examination of the reliability of the evidence is by checking if it is possible to verify the research method. This should provide information that enables replication of the procedure to collect the evidence. The research method should also provide information about the sample and the sampling procedure. Knowing the sample will help to understand if evidence is replicable. The evaluation of reliability involves two elements:

  1. Replicability of the evidence by asking if the evidence provides clear descriptions of the measures, the research procedure and the sampling of organizations and respondents. Read critically to check if this information is clear enough that it allows other researchers to replicate the research. Reliability means that we can trust that the findings reported in the evidence would appear similar if we did the same research again. Of course, a verifiable procedure does not automatically mean that the same findings will show when the research is repeated. Practitioners should therefore use multiple studies to find research evidence. If a causal effect has been reproduced frequently and across different settings, the evidence for a causal link is more robust. Systematic reviews and meta-analyses are good sources of evidence to see whether a causal relationship is stable across studies. In local evidence, keeping a log of what was used as evidence supports future replicability.

  2. Quality of the sample. Using a sample of people or organizations to get evidence about a population is a smart way of investing time and resources. However, whether or not research findings can be replicated also depends on the quality of the sample that was taken. Sample size and sampling procedures can both hamper the reliability of evidence. Sample size influences the precision with which you can say something about the population from which the sample was taken. More data means more information, and this makes the estimation more precise. The risk of extreme findings gets buffered in larger samples. Studies with larger samples are therefore preferred over those with smaller samples. However, a large sample may still be unreliable if the sample is not a good representation of the population from which it was selected. For example, if managerial effectiveness is measured by subordinates that were selected by the managers themselves, the evaluation of managerial effectiveness may be more favorable than when a random sample of all subordinates was taken. Hence, the reliability of the evidence depends on whom you ask. Preferably, evidence should be collected from a randomized sample of a clearly defined population. In local evidence, the rules of thumb that follow are that single interviews are less reliable than multiple interviews, and multiple perspectives are more reliable than a single perspective.

Generalizability

Generalizability deals with the question whether local or external evidence can be used to draw conclusions about people or organizations that were not included in the research that led to the evidence. Evidence repeated in multiple samples is more likely to generalize to new samples compared to evidence that is unique (‘breakthrough’). Therefore, it is safer to rely on evidence that was repeated in multiple samples. Still, even when practitioners carefully read systematic reviews and meta-analyses to collect external evidence, this does not hold that the evidence generalizes to their own research questions. The following questions should be taken into account:

  1. Does the evidence have boundary conditions? Boundary conditions are local conditions that may change a causal relationship. In research, we talk about moderators if we suggest that a certain condition will have an effect on the relationship between two variables. Meta-analyses usually mention a number of moderators (boundary conditions). Frequently reported boundary conditions in research articles are institutional characteristics (such as legislative environment, the state of the economy, country), organization characteristics (such as firm size, firm age, industry type), demographic characteristics (such as age, gender, job level, tenure, education level, employment state or contract) and research design characteristics (longitudinal, cross-sectional, experimental, qualitative).

  2. What if the generalization of evidence is unclear? If it is unclear whether the external evidence can be generalized or not, it is a good idea to examine if it is possible to replicate the evidence locally (Dietz et al., 2014). This can be done by taking a random sample of the target population of the organization and then repeating the research procedure as found in the external evidence to produce local evidence. When such a procedure is difficult to replicate, the practitioner can run focus groups with experts in the organization to get a good idea of how the evidence could be generalized to the local circumstances.

Ethicality

Working with evidence in the domain of Human Resource Management implies working with people. By applying evidence-based HRM, decision-makers have a responsibility towards people who contribute to generating the evidence and to those who are targeted by the HR practice that follows from the evidence. Academic and professional ethical guidelines should always guide the behavior of those using evidence-based HRM. Ethical guidelines exist to protect participants. Participation of stakeholders in providing evidence should by no means lead to any negative consequences for them.

Ethical guidelines include at least the principle of informed consent, the principle of honest information sharing, as well as the principle of data protection and privacy regulations. Informed consent means that participants explicitly agree that they participate in a research project, and that the information that they provide can be used as evidence. Open and honest information sharing involves that participants understand what the project entails and that they are informed about the outcomes (debriefed). Data protection and privacy regulations are meant to ensure that data are reliable, that data will be stored in a safe way and that participants’ input is anonymized in the research findings. A good practice is to have an ethics committee installed that evaluates if the consequences for participants are acceptable.

Example project “Task force equal pay”

Before setting out to collect local evidence, the task force checked the research literature to understand what kind of data they would need in their own organization. In particular, it was important to decide which variables to use from the employee records to understand the causes of the gender pay gap in the organization. For example, the research literature showed that performance evaluations of men and women can be biased, due to subtle processes of stereotyping the performance of men and women. The conversations to explore the problem hinted that this might also be the case in this organization. Likewise, the pay that employees received in a previous job also bears the risk of bias. Based on this knowledge, only facts that indicate the human capital of employees were included in the prediction, such as the level of education, the rank in the organization, and the years of experience in that rank. To be sure, the analyses controlled for having children and working part-time. This increased the validity of the organizational evidence.

To compare pay decisions for a selected position, the task force ensured a random selection of comparable male and female employees for whom the pay decision process was traced. To collect the information for each pay decision, structured interviews were performed with the hiring manager, the employee and the HR professional involved in the decision. This procedure guaranteed reliability of this part of the local evidence.

During the project, the task force reached out to other public organizations who had dealt with the same problem. By sharing experiences, they checked if their findings were generalizable beyond their own organization. This strengthened the importance of the findings.

Finally, all steps in the project that required collecting evidence from people were presented to an ethical review board. Because the task force hired help from a nearby university, ethical approval of the university was gained first. In other projects, the task force checked for ethicality with the employee representative board in the organization. This is a formal group of employees, who consult with senior management about employee policies. The employee representative board also approved the project before data collection began.

Process part 4: Generating alternatives

With good-quality evidence, it is possible to generate effective HR practices. This is a fun part of the process. People involved in the evidence-based HRM project can generate ideas for HR practices in a series of brainstorming sessions. To stimulate out-of-the box alternatives, the first round of brainstorming aims to generate as many ideas as possible. The second and following rounds of brainstorming evaluate each idea against the local and external evidence to find if it meets the expected causality. Sometimes, an idea can be adjusted and improved; other ideas do not meet the requirements and will be dropped. The final round of brainstorming looks towards the implementation. Local evidence provides knowledge about the context in which an HR practice will operate. To ensure effective use of the HR practice, the context should be taken into account in the choice for a practice. Eventually, just a few HR practices remain that are valid interventions to improve the targeted outcome.

Example project “Task force equal pay”

The overwhelming amount of evidence brought a challenge to the task force in scoping all alternative HR practices. To generate alternatives, the task force first wrote a report about the project findings, including the findings from local evidence and a listing of all the interventions that were found in the research literature review. They then sent the report to representatives of identified stakeholder groups, such as senior management, unit managers, the employee representative board, and to the human resource managers in the organization. Next, they organized three rounds of brainstorming where both the task force members and all representatives who received the report could contribute. After explaining the project and the findings, this group of about twenty people brainstormed about alternatives. A team coach facilitated the brainstorm sessions and summarized the findings between each session. Eventually there was consensus about the need to improve consistency, transparency and accountability in the design and execution of pay decisions. This was the input for the task force to make a decision about the design of the HR practices.

Process part 5: Decide, prepare and implement

After generating valid alternatives, there is a decision about the preferred HR practice. The next step is to prepare for implementation. It is likely that there are obstacles in the organizational culture, structure, and regulations or politics that need to be taken into account during the development, implementation and use of evidence-based HRM. Potential obstacles during implementation include the user-friendliness of the HR practice, the support and beliefs of stakeholders, legislation, conflicting policies, and behavior of key senior managers (Mirfakhar et al., 2018; Trullen et al., 2020). A good understanding of the organization and its context is a prerequisite for effective implementation. Local evidence can help to detect obstacles and indicate where potential resistance may exist in the organization. This knowledge should be used in preparing the implementation plan. Some practical guidelines for preparing implementation, based on research evidence (Mirfakhar et al., 2018):

  1. Easy to understand HR practices have a higher chance to be used.

  2. HR practices should align with other regulations and laws.

  3. Line managers should be trained and prepared in using the practice in their management of employees.

  4. Top management should communicate the importance of the HR practice.

  5. During implementation, stakeholders need support in learning to use the HR practice.

An organizational culture in which the use of evidence is considered good practice prevents many of the obstacles during implementation. Management can promote a culture for evidence-based management by stimulating learning and knowledge sharing in the organization. The following strategies can be used to advocate the advantages of evidence-based HRM in organizations (Jeffrey & Sutton, 2006; Rousseau, 2006):

  1. Raise awareness. It helps when there are advocates for evidence in the organization. People who demand evidence, share evidence and who are suspicious of ‘brand new’ ideas. The more people in an organization dare challenge the logic of plans for action, but instead examine and question it, the wider the awareness about evidence-based HRM will become.

  2. Showcase data. Promote the use of a company’s own data and experience. Run trials, do small experiments. This will contribute to building a culture in which managers learn to learn.

  3. Facilitate access to external evidence. Advocate a company subscription to research databases and train decision-makers in how to use these.

  4. Collaborate in research. Use the relationships with experts and knowledge institutions like universities to acquire and share knowledge. Evidence-based HR practices for important issues can be developed in co-creation between science and practice.

  5. Training. Organize training in evidence-based HRM. Ensure that the skills will transfer to practice by having participants work on their own questions. The more people are trained in EBHRM, the larger the support system in the organization that promotes access to knowledge.

  6. Curiosity. Senior managers are role models. By asking questions about the nature of problems, and by initiating the generation of local and external evidence, a climate of curiosity will cascade down the organization.

Example project “Task force equal pay”

An important obstacle faced by the task force, were differences in existing practice in pay decisions across organizational units. In a unit with high-skilled professionals, management feared losing the authority to make competitive reward offers to external recruits in a competitive labor market. The task force realized that they would need the buy-in of this management team as users of the HR practices to gain results. To overcome this obstacle, the director of the unit was invited to join the sessions in which ideas for HR practices were generated. This resulted in a sense of ownership in the unit during and after the implementation of the practices.

Process part 6: Evaluate and adjust the HR practice

Finally, yet importantly, the use of the implemented HR practices and the outcome need evaluation. Tracking the outcome will show if the HR practice leads to the expected effects in the outcome. Tracking can be quantitative, by using objective measures, or qualitative, by asking stakeholders about their views on the effectiveness of the HR practice. A structured approach to evaluation can be achieved by using an after action review (AAR). This is a structured debriefing process looking back at what happened, understanding why it happened, and what could be improved. In case the effect is other than expected, decision-makers may need to go back in the process and make adjustments.

Given the importance of the project, senior management and the employee representative board agreed to monitor the pay gap on a yearly base. As sources of evidence they agreed to 1) repeat the analysis of personnel files to compare gender pay differences, and 2) do random checks on the pay decision process for new hires. The findings are discussed between senior management and the employee representative board. The task force is dismantled; however, the human resource management department has taken over its role to monitor and evaluate the practices and improve these if needed.

HR practices are tools for managing people in organizations, aimed at changing or improving an outcome. Outcomes vary from organizational performance (the business case), to ensuring alignment with the organizational environment (the context of HRM) to employee health and well-being (the employee perspective). Some HR practices are more effective compared to others in achieving the desired outcome. Evidence-based HRM is a decision-making process to improve the choice and design of effective HR practices, under conditions of bounded rationality. Bounded rationality is the condition that it is impossible to know everything, and that unforeseeable change may occur. Evidence-based HRM proposes an iterative process, where decision-makers go back and forth in the process before reaching a decision. The evidence-based HRM decision-making process promotes a judicious evaluation of organizational and research evidence, while taking into account ethicality, to design effective HR practices. The process is sequential, starting with defining a problem, followed by collecting evidence, evaluating evidence, generating alternatives and deciding on an HR intervention, preparing for implementation and, finally, evaluating the implementation and its effects on the outcome. Good evidence is the key to effective decisions. The quality of evidence can be checked by evaluating validity, reliability, generalizability and ethicality. The success of the implementation will depend on how well an HR practice is designed and finds support by users in the organization. The chapter provides practical tips to guide effective designs of HR practices.

Comments
0
comment

No comments here

Why not start the discussion?