What is CEO’s research development process?

The Department of Labor’s Chief Evaluation Office oversees a standardized process to ensure our contracted research is trustworthy, credible, independent, and rigorous. CEO sponsors evaluations and research that:

  • Are primarily conducted by competitively-selected, external third-party contractors;
  • Undergo rigorous and multiple reviews to ensure the quality and objectivity of the research during and after the research development process; and
  • Are otherwise conducted in accordance with the established Department of Labor Evaluation Policy principles of rigor, relevance, transparency, independence, and ethics.
  • Once all technical and subject matter reviews have been completed and the evaluators have responded to reviewer comments, the research team finalizes the report, including preparing it to meet all Section 508 compliance requirements to make final reports accessible to all. Most studies are multi-year and may have multiple reports. Many projects also have short policy briefs that summarize the main findings.

What does CEO’s research review process include?

While different methodologies require different types of review and may vary, CEO studies generally include numerous review points to ensure quality:

  1. Independent contractor review of data collection plans: CEO contracts independent experts uninvolved with the research to conduct a thorough review of the Paperwork Reduction Act (PRA) Information Collection Request, including an assessment of whether the methodological and statistical aspects of the planned data collection are appropriate, rigorous, and likely to generate data that support the research questions.
  2. Office of Management and Budget (OMB) review: OMB reviews and approves data collection activities per PRA requirements.
  3. Institutional Review Board (IRB) review: IRBs composed of experts conduct reviews of plans for research to ensure the protection of human subjects involved in the research, ensuring CEO-sponsored research is held to the highest ethical standards.
  4. Technical Working Groups: during the research development process, plans for evaluation designs, data collection, and analyses are reviewed by expert peers from the academic, non-profit and private sectors, convened together in Technical Working Groups (TWGs).
  5. DOL agency and program officials: partner agency staff are consulted and briefed regularly during the research development process, and conduct a final review of deliverables prior to publication.
  6. CEO technical staff: CEO’s evaluation specialists and data analytics experts provide ongoing technical management and reviews before, during and through the research development, finalization and dissemination processes.
  7. Additional outside technical reviews: are conducted as needed, based on the methods or topic area of the research.

  8. Final independent expert review: CEO contracts independent experts uninvolved with the research to conduct reviews of all final deliverables that:

    1. Flag any statistical or methodological issues present in the deliverable, and options on how to correct the identified weakness or deficiency;
    2. Assess each claim, line by line, in a given report for whether it is supported by the data collected and analyzed, based on the methodology used, or by reference to external research; and
    3. Use a detailed methodological rubric (with hundreds of evaluative areas) to review the technical accuracy, data analyses, and reporting according to best practices in research methods in the social sciences, including economics, statistics, and others.

Once all technical and subject matter reviews have been completed and the evaluators have responded to reviewer comments, the research team finalizes the report, including preparing it to meet all Section 508 compliance requirements to make final reports accessible to all. Most studies are multi-year and may have multiple reports. Many projects also have short policy briefs that summarize the main findings.

Illustrative examples of CEO’s research practices

How CEO Works with Key Partners

Below we provide an example of how the above process works with an actual project – the 2018 Employee and Worksite Perspectives of the Family and Medical Leave Act National Surveys.

Producing Rigorous and Relevant FMLA Deliverables

Project Background: To gain knowledge of how employees and employers understand and experience FMLA, CEO contracted independent contractors to survey employees and employers in 1995, 2000, 2012, and 2018. The 2018 surveys consisted of two separate surveys, one for employees and one for worksites. Interviews were conducted by phone and online. The 4,470 employees surveyed were working-age adults employed in the public or private sector in the 12 months prior to the survey. The worksite survey consisted of 2,206 employers, both covered and uncovered by FMLA. The below graphic illustrates the resources that provided inputs to this research, the resulting deliverables, and some of the ongoing interest this work has spurred inside and outside of the Department.

FMLA 2018 Graphic

How CEO Works with Key Partners

Below we provide an example of how the above process works with an actual project – the Reemployment Services and Eligibility Assessments (RESEA) Evidence Building and Implementation Study.

Producing Rigorous and Relevant RESEA Deliverables

Project Background: To address the evidence requirements in the Bipartisan Budget Act of 2018 related to RESEA programs, CEO contracted independent contractors to understand the state of the evidence, conduct an implementation study, develop evidence-building options, and provide evaluation technical assistance to states to build their technical capacity. The project has provides important evidence and services for state RESEA programs and represents extensive collaboration and coordination with numerous public and private stakeholders as shown in the graphic below.

RESEA annual report graphic

Explore our website for more information on current and completed studies.

For questions about CEO, please email us at ChiefEvaluationOffice@dol.gov

Sign up for updates from CEO

The Department of Labor’s (DOL) Chief Evaluation Office (CEO) sponsors independent evaluations and research, primarily conducted by external, third-party contractors in accordance with the Department of Labor Evaluation Policy. CEO’s research development process includes extensive technical review at the design, data collection and analysis stage, including: external contractor review and OMB review and approval of data collection methods and instruments per the Paperwork Reduction Act (PRA), Institutional Review Board (IRB) review to ensure studies adhere to the highest ethical standards, review by academic peers (e.g., Technical Working Groups), and inputs from relevant DOL agency and program officials and CEO technical staff. Final reports undergo an additional independent expert technical review and a review for Section 508 compliance prior to publication. The resulting reports represent findings from this independent research and do not represent DOL positions or policies