About the Study
In 2020, the Chief Evaluation Office funded contractor American Institutes for Research (AIR) to develop and conduct an innovative assessment of the Department of Labor’s (DOL) capacity to generate and use evidence. This initial capacity assessment encompassed 16 DOL agencies and was designed to:
- establish a baseline understanding of the Department’s evidence capacity,
- continue an important dialogue around evidence within the Department,
- inform next steps for improving capacity across the Department, and
- lay the groundwork for future reviews while addressing a requirement of the Evidence Act of 2020.
The assessment drew on a survey that allowed AIR to understand staff’s use and production of evidence when working on policies, programs, and/or operations, separate focus groups with staff and leadership in 28 focus groups conducted between 2020 and 2022.
The assessment findings offer important insights on baseline patterns and opportunities to improve capacity for evidence use and production going forward. Its insights are informing tangible next steps that DOL is taking to build on current strengths and address capacity gaps to support better informed decision-making on policies and program implementation that affect American workers.
- Department of Labor – Across Agencies: Evidence Capacity Summary (Summary Brief, November 2023)
- Supporting an Evidence-Driven Culture: U.S. Department of Labor’s Evidence Capacity Assessment (Final Report, November 2023)
AIR will build and improve upon the following initial questions to conduct the capacity assessments.
- To what extent are DOL’s statistics, evaluation, research, and analysis efforts appropriate in terms of coverage, quality, methods, effectiveness and independence?
- To what extent are agencies able to access and use evidence relevant to their work?
- To what extent is evidence used in agencies’ decision-making around policies, procedures, and programs?
- What strengths, challenges and obstacles influence how agencies generate, access, and use evidence in decision-making? How can DOL build upon these strengths, and overcome challenges.
- The Department and its agencies are building on strengths that include a strong sense of mission and strategic goals; thoughtful visions of improved evidence use; and strong leadership support for evidence-based decision making.
- Data access is a multifaceted constraint cited across agencies. Agency staff need better data systems and processes to reduce costs of accessing and merging existing data, the ability to access and merge data from other agencies and access data sets from external parties, and access more granular data collection.
- Data visualization software can be game-changing in helping agencies use data, but several agencies indicated a lack of necessary tools or the training for their use. Agency staff need data visualization tools and training to generate customized, just-in-time analyses.
- Several challenges interfere with agencies’ ability to gather and use evidence that is aligned to their decision-making needs. For example, agencies struggle with evaluation results becoming available after program, policy, or context changes make their findings less relevant.
- Competing demands, understaffed agencies, and time-consuming processes strain the ability of agency staff to focus on efficient use and production of evidence.
- Agency leadership sets the stage for a strong culture of evidence use and production. Staff generally feel supported by their supervisors and agencies in using and producing evidence but desire greater and more consistent support over time. For example, there is a need for more consistent championing of evidence strategies across political cycles.
Summary Brief
Corea, C., Amin, S., Ahlstrand, A., Cadima, J., Stockham, J., & Rege, G. (2022). American Institutes of Research. Department of Labor – Across Agencies: Evidence Capacity Summary. Chief Evaluation Office, U.S. Department of Labor.
Final Report
Corea, C., Amin, S., Ahlstrand, A., Cadima, J., Stockham, J., & Rege, G. (2022). American Institutes of Research. Supporting an Evidence-Driven Culture: U.S. Department of Labor’s Evidence Capacity Assessment. Chief Evaluation Office, U.S. Department of Labor.
Project Duration: 44 Months
Contract End Date: January 2023
Contractors: American Institutes for Research (AIR)
For More Information: ChiefEvaluationOffice@dol.gov
The Department of Labor’s (DOL) Chief Evaluation Office (CEO) sponsors independent evaluations and research, primarily conducted by external, third-party contractors in accordance with the Department of Labor Evaluation Policy. CEO’s research development process includes extensive technical review at the design, data collection and analysis stage, including: external contractor review and OMB review and approval of data collection methods and instruments per the Paperwork Reduction Act (PRA), Institutional Review Board (IRB) review to ensure studies adhere to the highest ethical standards, review by academic peers (e.g., Technical Working Groups), and inputs from relevant DOL agency and program officials and CEO technical staff. Final reports undergo an additional independent expert technical review and a review for Section 508 compliance prior to publication. The resulting reports represent findings from this independent research and do not represent DOL positions or policies.