About the Study
In 2017, the Chief Evaluation Office’s (CEO’s) Administrative Data and Research Analysis (ADRA) contract partnered with the Employment and Training Administration (ETA) and funded contractor Mathematica Policy Research to conduct Data on Earnings: A Review of Resources for Research and Comparing Job Training Impact Estimates Using Survey and Administrative Data. The descriptive analyses aim to review potential opportunities and implications of using different earnings data sources to analyze employment outcomes for research and evaluation studies. In Data on Earnings: A Review of Resources for Research, researchers described the characteristics of and considerations for administrative data, surveys of study participants, and existing general population surveys. For the Comparing Job Training Impact Estimates Using Survey and Administrative Data study, researchers compared earnings levels and impact estimates from three sources to gather insights regarding their strengths, drawbacks, and tradeoffs.
This Department of Labor-funded study is a resource to help build the labor evidence-base to inform data and reporting programs and policies and addresses Departmental strategic goals and priorities.
- Comparing Job Training Impact Estimates using Survey and Administrative Data (Final Report, December 2018)
- Data on Earnings: A Review of Resources for Research (Final Report, November 2018)
- To describe the potential opportunities and implications of using different earnings data for research and evaluation studies.
Data on Earnings: A Review of Resources for Research
- Access is an issue for all five administrative earnings data sources. Government tax data can only be accessed on-site or from approved locations. In addition, analysis products require official review for potential disclosure issues. State-level data on unemployment insurance (UI) may be de-identified or otherwise limited depending on the providing state.
- Coverage varies between administrative data sources. For example, state UI wage data excludes federal and military employment, while some tax data sources report taxable—not gross—income. All administrative sources exclude informal sources of income which may be captured by survey data.
- Earnings data from self-employment may not be reliable. State UI data do not capture self-employment, and self-employment income is underreported to the IRS by about two thirds. Further, most self-employment income does not correspond to amounts reported on pay stubs, so there may not be a reliable reference for survey respondents to report.
- Periodicity of the data is an important consideration. UI data is updated quarterly, while tax data are annual. Tax data can be easier to work with, but quarterly data enable better measurement of employment over time and greater flexibility in measurement.
Comparing Job Training Impact Estimates using Survey and Administrative Data
- Across data sources, analysis demonstrated that WIA-funded intensive services were effective at improving customers’ earnings. However, the magnitude of the estimated impact differed depending on the data source.
- Impacts appeared smaller when measured with administrative data and appeared larger when measured with survey data; this finding is consistent with previous studies.
- Several factors drive the difference in results derived from survey and administrative data. For example, survey data on jobs held early in a follow-up period are more subject to recall error. In addition, survey data on earnings from self-employment and contract work indicates higher income than administrative data would suggest.
Czajka, J. L., Patnaik, A., Negoita, M. (2018). Mathematica Policy Research. Data on Earnings: A Review of Resources for Research. Chief Evaluation Office, U.S. Department of Labor.
Mastri, A., Rotz, D., Hanno, E. S. (2018). Mathematica Policy Research. Comparing Job Training Impact Estimates using Survey and Administrative Data. Chief Evaluation Office, U.S. Department of Labor.
The Department of Labor’s (DOL) Chief Evaluation Office (CEO) sponsors independent evaluations and research, primarily conducted by external, third-party contractors in accordance with the Department of Labor Evaluation Policy. CEO’s research development process includes extensive technical review at the design, data collection and analysis stage, including: external contractor review and OMB review and approval of data collection methods and instruments per the Paperwork Reduction Act (PRA), Institutional Review Board (IRB) review to ensure studies adhere to the highest ethical standards, review by academic peers (e.g., Technical Working Groups), and inputs from relevant DOL agency and program officials and CEO technical staff. Final reports undergo an additional independent expert technical review and a review for Section 508 compliance prior to publication. The resulting reports represent findings from this independent research and do not represent DOL positions or policies.