State- or System-Level Needs Assessment | Career Readiness Data Quality and Use

This needs assessment can be used to evaluate policy and practice at the state or system level. It breaks down the core elements of a high-quality career readiness data ecosystem into multiple sections and illustrates what both undefined and established practice look like.

To complete the needs assessment

Read through the criteria for each sub-element. Note that criteria are provided only for the low end (1 — Undefined) and high end (4 — Established) of the rubric, though it is expected that states may fall somewhere in between.
Choose a rating from 1 to 4 that best describes where the state falls on the criteria for that sub-element. For example, if the state scores a 4 on most of the criteria and 1 on some of the criteria, then it might make sense to choose a 3 for the overall score for that sub-element.
Add any additional evidence to describe why you selected this score.
At the end of each part, pick a summative score for the core element (this could be based on your overall assessment of the core element or an average of the scores for each sub-element).

Ratings definitions

1 (Undefined): This component is not yet defined or is just beginning to emerge; current state policy meets most of the criteria listed under 1.

2 (Building): This component has some bright spots, but there are still many improvements to be made; it meets some of the criteria under 1, but key considerations allow for more optimism.

3 (Promising): This component is fairly well developed, though there are still some improvements to be made; it meets some but not most of the criteria under 4 and is considered to be more developed than a 2.

4 (Established): This component is extremely well developed and effective, even if there are still minor adjustments to be made; it meets most or all of the criteria listed under 4.

Also note that, while the needs assessment sets a benchmark for a high-quality career readiness data ecosystem, states can take different approaches to reach the outcomes described in the theory of action. The needs assessment is not meant to be prescriptive but rather to provide a benchmark, based on consensus from experts, you can use to assess your state’s progress and identify opportunities for further action. The tool can be applied to the career readiness data ecosystem as a whole — spanning K-12, postsecondary and workforce data — or with a particular focus on one education sector.

GlOSSARY

Refer to the glossary for more details on key terms and definitions as you complete the needs assessment.

Indicates required field

Once the form is complete, print a copy of your responses or save a copy for your records. The current version of this website does not allow for submission of the form.

1. Data is collected consistently and accurately.

For data to be trusted, policies and protocols must be in place to ensure the consistent collection of reliable, valid and complete career readiness data. If the data is accurate and complete and if stakeholders understand what the data represents and how it will be used, public trust in the data will grow.

A. Data is reliable and collected consistently around the state, across different career pathway programs and across institutions.

Criteria for Rating 1 (Undefined)

There are limited to no uniform definitions or business rules for collecting career readiness data.
Career pathway participants and concentrators are identified manually at the local level and reported up to the state.
There is limited to no statewide guidance for collecting, validating and submitting data.
There are limited to no trainings provided to local data administrators on collecting and validating data.
Spreadsheets are used to collect and report data to the state.

Criteria for Rating 4 (Established)

Statewide definitions and business rules are in place to describe all data elements. Business rules are changed only to improve them or align them to new legislation or policy, and data is collected consistently and in the same format across the state.
Designations such as career pathway “participants,” “concentrators” and “completers” are made using automated data processes.
Statewide guidelines clearly and effectively describe processes for collecting, validating and submitting data elements.
Local data administrators have regular, sufficient and timely training (at least once a year) to ensure a full understanding of data rules, policies and technology.
Local data administrators submit data directly to the state data system using a secure, web-based portal.

Your Evidence

Your Rating

B. Processes and protocols are in place to validate career readiness data.

Criteria for Rating 1 (Undefined)

There are limited to no processes in place to validate data at the state level.
Once local practitioners and data administrators submit their data to the state, they do not see the data again until it is reported.
State-led audits of career readiness data are limited in scope and reach.

Criteria for Rating 4 (Established)

Automated data validation protocols are in place to flag inconsistencies and/or errors in the data at the point of submission.
State data staff validate submissions using previous years’ data in addition to administrative records from partner agencies and/or vendor files.
Local practitioners and data administrators are given the opportunity to review and validate data by program and by institution before it is reported publicly.
State data staff conduct annual audits to review and validate locally submitted data. Audits are based on an initial risk assessment and cover between 5 percent and 25 percent of local institutions, depending on the size of the state.

Your Evidence

Your Rating

C. Stakeholders are aware of what the data represents, how it will be used and its limitations.

Criteria for Rating 1 (Undefined)

Information on business rules, definitions and protocols for collecting data is difficult to find or not publicly available.
Limited to no explanation is provided when data is incomplete.
Data collection is just “something we have to do to be compliant,” and stakeholders do not understand how data will be used.
There is limited to no transparency about the process for changing data elements or collection processes, leading to confusion and multiple demands from different stakeholders.

Criteria for Rating 4 (Established)

Business rules and definitions for career readiness data elements, along with processes and protocols for collecting and validating data, are available to the public, including online.
The state describes when data is incomplete.
Stakeholders understand how the data will be used, the limitations of the data, and why data is important and relevant to them.
State leaders, state staff and policymakers understand processes for making changes to data collections and publicly share clear and transparent expectations for what it takes to make said changes.

Your Evidence

Your Rating

Overall Rating

2. Processes and protocols are in place to ensure effective data governance.

An effective career readiness data ecosystem has a clear governance structure in place that designates roles and responsibilities for collecting, validating and reporting career readiness data as well as for setting a strategic vision for the publication and use of data. Capacity is deployed to ensure that data is high quality and that learner records are protected, and measures are taken to prevent disruption in the inevitable event of personnel or political transition.

A. Roles and responsibilities for collecting, validating and reporting data are clearly laid out in statute and/or policy.

Criteria for Rating 1 (Undefined)

Responsibilities for collecting, validating and reporting career readiness data at the state level are not designated in state policy.
There is a lack of clarity about who is accountable for different data collection, analysis and reporting responsibilities, leading to duplication of effort and bureaucratic inefficiency.
Inter-agency data sharing agreements are ad hoc, time limited and narrow in scope.
Data is understood only within the narrow context of the agency that collects it.

Criteria for Rating 4 (Established)

Responsibilities for collecting, validating and reporting career readiness data at the state level are designated in state policy.
Inter-agency memoranda of understanding outline roles, processes and timelines across agencies for sharing and linking data, issuing reports, identifying research questions and more.
Data sharing agreements do not have a sunset date and are revisited for quality, efficiency and effectiveness every three years or when leadership changes.

Your Evidence

Your Rating

B. Decisions related to the collection and use of career readiness data are coordinated across agencies and responsive to stakeholder needs.

Criteria for Rating 1 (Undefined)

CTE is not involved in decisionmaking related to statewide education and workforce data.
There are limited to no feedback loops in place for stakeholders to inform processes, uses of data and the state’s research agenda.

Criteria for Rating 4 (Established)

Representatives from the secondary and postsecondary state CTE offices have a formal advisory role in decisions related to statewide education and workforce data (e.g., about the statewide longitudinal data system).
Feedback loops are in place for stakeholders — such as practitioners and employers — to inform processes, uses of data and the state’s research agenda.
Researchers are regularly engaged to inform a comprehensive policy and research agenda.

Your Evidence

Your Rating

C. State agencies are sufficiently staffed and funded, and structures are in place to withstand personnel and political transitions.

Criteria for Rating 1 (Undefined)

Funding for state data capacity is limited and/or fluctuates from year to year.
The state is insufficiently staffed and/or staff are insufficiently trained to support the collection, analysis and reporting of career readiness data.
There is a high level of turnover among state data staff, and there are limited to no processes in place to provide for knowledge management and sustainability.

Criteria for Rating 4 (Established)

There is a consistent state funding stream to support and sustain data systems and personnel.
The state is adequately staffed with a breadth of expertise related to career readiness data, including information technology staff, data analysts, research staff and program support staff.
Processes, protocols and decisions are well documented to preserve knowledge through periods of turnover and transition.

Your Evidence

Your Rating

D. Measures are in place to protect the privacy of learner records.

Criteria for Rating 1 (Undefined)

State data staff receive limited to no training on data privacy and security protocols.
There is limited to no clarity on how permissions should be structured, allowing some users to access unmasked learner-level data they do not have authority to see.
Permissions are rarely updated, allowing state staff and local practitioners to continue to access learner-level data once they have left their positions.
Limited to no training is provided on handling small cell size data.

Criteria for Rating 4 (Established)

State data staff are trained upon hire and then annually thereafter on data privacy, including legal requirements and protocols for protecting learner records.
Clear standards and procedures are in place to ensure data security and learner privacy, including guidance for differentiating permission levels by user type.
Permissions are automatically updated to reflect personnel changes at the state and local levels and are audited annually to ensure that only current staff and relevant stakeholders can access learner records.
State and local staff are provided training on how to work with small cell size data, and measures are taken (e.g., averaging data over multiple years) to allow users to access disaggregated data.

Your Evidence

Your Rating

Overall Rating

3. Data systems, policies and practices are fully aligned across agencies and learner levels.

Silos between and within state-level agencies are broken down, resulting in career readiness indicators and definitions that are aligned and uniform across the state. Additionally, processes and protocols are streamlined to ensure that learner-level records are fully interoperable across data systems and that data collection and reporting cycles are, to the extent possible, synchronized.

A. Learner-level records are reliably linked across agencies and across states, as appropriate.

Criteria for Rating 1 (Undefined)

The state is unable to link learner-level records across education levels, relying on self-reported student surveys for accountability purposes.
The state is unable to link learner-level records across other federal or state social service programs.
The state is unable to link learner-level records with administrative labor and employment data, relying on self-reported student surveys for accountability purposes.
K-12, postsecondary and/or workforce/labor data systems use different unique identifiers.
Each state agency operates its own data system, and these systems are not interoperable.
The state cannot track learners if they cross state lines to work or attend a postsecondary institution.
Researchers have limited to no access to career readiness data.

Criteria for Rating 4 (Established)

Learner-level records are linked across all education levels, including early childhood education, K-12 education, two-year postsecondary institutions, four-year postsecondary institutions, adult education, non-credit workforce training programs, and state systems such as corrections and foster care.
Learner-level records are linked across all federal and state social service programs, including TANF, SNAP and Medicaid.
Learner-level records are linked with administrative labor and employment data, including unemployment insurance, military participation databases, national community service databases, and national industry certification and licensure databases.
The state uses a common unique identifier across the P20W system.
Matching protocols are in place to accurately link learner-level records, resulting in at least a 90 percent match rate.
Data systems are fully interoperable such that learner records are automatically updated across state-level data systems.
Inter-state data sharing agreements (e.g., SWIS) are in place that allow the state to monitor career pathway participation and outcomes within the regional economic or commuting area.
Researchers have access to validated, appropriately labeled and matched datasets.

Your Evidence

Your Rating

B. The collection and reporting of career readiness data are coordinated and, to the extent possible, aligned across programs, agencies and learner levels.

Criteria for Rating 1 (Undefined)

K-12, postsecondary, workforce and labor agencies operate in silos with limited to no coordination for data collection or reporting.
Career readiness indicators are not included in relevant reports across programs, agencies and learner levels.

Criteria for Rating 4 (Established)

Data collection and reporting cycles are aligned across the K-12, postsecondary, workforce and labor systems.
Data collection protocols are regularly examined and updated to minimize duplication of effort.
Career readiness indicators are included in relevant reports across programs, agencies and education levels (e.g., in school report cards or college score cards).

Your Evidence

Your Rating

C. State agencies use common indicators and business rules for measuring career readiness and align their goals and performance targets.

Criteria for Rating 1 (Undefined)

Definitions, business rules and indicators of career readiness vary across agencies and education levels.
There is limited to no alignment of career readiness performance targets between ESSA and Perkins at the secondary level or among WIOA, Perkins and other attainment goals at the postsecondary level.
Stakeholders, agencies and sectors do not have an aligned vision for career readiness. If there is a statewide career readiness goal, it is disconnected from the visions or goals set by individual agencies.

Criteria for Rating 4 (Established)

Common definitions, business rules and indicators of career readiness are used across agencies and education levels.
Career readiness performance targets are coordinated across ESSA and Perkins at the secondary level and WIOA and Perkins at the postsecondary level. Performance targets are also aligned with and support any statewide postsecondary attainment goals.
State agencies have a common statewide vision for career readiness and learner success across education levels that is anchored by a shared statewide career readiness goal.

Your Evidence

Your Rating

Overall Rating

4. Information is relevant, timely and disaggregated.

Data collection and reporting processes address the needs of practitioners, policymakers, researchers and the public, ensuring that stakeholders have access to the information they need when they need it. All data is contextualized and differentiated by audience, allowing for a nuanced understanding of the career readiness system.

A. All information is contextualized to provide a clear understanding of the career readiness system.

Criteria for Rating 1 (Undefined)

Historical data is not made available in public reporting or practitioner dashboards.
The state reports enrollment and outcomes only for Perkins-funded — not other state-funded — career pathways.
The data that is collected and reported statewide is driven only by federal accountability requirements.
Placement data for program completers is aggregated, making differentiating outcomes for various groups of completers impossible.
Practitioners have limited to no information about enrollment and outcomes for learners outside of their education sector.
Limited to no information is made available to the public regarding the educational and industry credentials available to learners in the state and the competencies, costs, career pathways and outcomes associated with those credentials.

Criteria for Rating 4 (Established)

Reports and data systems provide at least five years’ worth of data to provide historical context, as appropriate. Any changes to data elements as a result of changes to policy, assessments or accountability systems are clearly documented.
Data collection includes the entire career readiness system, not only Perkins-funded programs.
State performance indicators are aligned to the state’s vision for career readiness and not driven only by federal accountability.
Publicly reported placement data for program completers is disaggregated by outcome (e.g., two- or four-year postsecondary enrollment, the workforce, the military).
Practitioners can access insights from across the career pathway continuum (e.g., K-12, postsecondary, workforce) to both understand the outcomes of former learners and anticipate the needs of future learners.
All educational and industry credentials and their associated competencies, costs, career pathways and employment outcomes are made available to the public.

Your Evidence

Your Rating

B. Reports and dashboards are differentiated by user depending on their need and understanding of the data.

Criteria for Rating 1 (Undefined)

Reports and dashboards provided by the state do not address the questions or needs of local stakeholders and practitioners.
Data that is shared with practitioners is masked, only allowing them to see aggregated data.
Dashboards and reports are not interactive.

Criteria for Rating 4 (Established)

The state has stakeholder engagement routines in place to understand the questions of different user groups and develop reports that respond to their needs.
Practitioners are able to access unmasked enrollment and performance data, as appropriate.
Feedback reports are shared with practitioners at least once a year to provide information on learner outcomes one year, three years, five years and 10 years after completion.
Dashboards allow for customization based on users’ needs/sophistication.

Your Evidence

Your Rating

C. Information is made available in a timely manner.

Criteria for Rating 1 (Undefined)

Practitioners cannot access data or can access data only after it is too late to make instructional or programmatic interventions.
Data collection cycles often lead to lags in data reporting.
Reports and dashboards are not updated regularly or consistently.
The process for collecting and reporting new data elements is inefficient and slow.

Criteria for Rating 4 (Established)

Practitioners have access to real-time course enrollment, course completion, assessment, work-based learning and credential attainment data to enact just-in-time interventions to support learner success.
Data collection cycles allow for timely reporting for federally or state-mandated reports as well as practitioner dashboards and public reports.
Reports and dashboards are updated at least once a year and in a consistent fashion.
New data elements can be added to existing data collection and reporting, with reasonable delays, to address evolving data needs.

Your Evidence

Your Rating

D. Data is disaggregated by population, institution and career pathway and available to relevant users.

Criteria for Rating 1 (Undefined)

Data cannot be disaggregated by major racial/ethnic groups, gender and special population status.
Data cannot be disaggregated by school, district, college, technical center, etc.
Data cannot be disaggregated at the Career Cluster® and career pathway levels.

Criteria for Rating 4 (Established)

Data is disaggregated by major racial/ethnic groups, gender and special population status. When possible, data can be disaggregated by multiple identities (e.g., gender and race/ethnicity) to identify intersectional outcomes.
Data is disaggregated by school, district, college, technical center, etc.
Data is disaggregated at the Career Cluster and career pathway levels.

Your Evidence

Your Rating

Overall Rating

5. Practitioners and the public are equipped to understand and leverage data.

In a high-quality career readiness data ecosystem, data is not reported for data’s sake but rather to foster understanding and value and to spur users to action. Meeting these goals requires a thoughtful approach to designing and presenting career readiness data and a robust system of professional development, technical assistance and supports to ensure that practitioners understand how to use the data. Additionally, data elements are integrated into the state’s communications strategy to tell a career readiness story.

A. Public reports are accessible and easy to understand.

Criteria for Rating 1 (Undefined)

Reports and dashboards are complex and use jargon.
Reports and dashboards rely on number-heavy tables instead of charts and visualizations to communicate data.
Reports and dashboards are available only in English.
It is difficult for users to find information on the methodology, business rules and explanation of data elements provided on public reports.
Reports and dashboards are not 508 compliant.
There are limited to no internal processes to ensure that reports and data publications are developed consistently over time and across schools, districts, colleges, technical centers, etc.

Criteria for Rating 4 (Established)

Reports and dashboards are written at no higher than an eighth-grade reading level and avoid jargon, to the extent practicable. Reports use clear and concise descriptions to help people understand the data.
Reports and dashboards leverage data visualization best practices to help users understand the data.
Reports and dashboards are available in multiple languages and formats, as appropriate within the context of the community served.
Links to the methodology, business rules and explanations of data elements are provided on public reports.
Reports are 508 compliant.
Internal processes are in place to guide reports and data publications to ensure consistency.

Your Evidence

Your Rating

B. Professional development and technical assistance are provided to practitioners to build data literacy and help them leverage the data.

Criteria for Rating 1 (Undefined)

The state does not have a professional development strategy to help practitioners understand and use career readiness data.
There is no targeted engagement or support for local data administrators at the district, school and institution levels.
The state has limited to no onboarding protocols in place for local data administrators.

Criteria for Rating 4 (Established)

The state has a comprehensive professional development strategy in place to bolster data literacy among local practitioners (including CTE educators, advisers, faculty and administrators). The strategy includes a focus on understanding and leveraging career readiness data for continuous improvement and equity.
The state has identified and regularly engages local data administrators at the district, school and institution levels, including through regular trainings and supports.
The state has an onboarding protocol in place for local data administrators.

Your Evidence

Your Rating

C. A statewide career readiness communications strategy is in place that leverages career readiness indicators to tell a story of impact.

Criteria for Rating 1 (Undefined)

The state provides limited to no training, tools, templates or guides to assist local practitioners in communicating their data to stakeholders.
State staff do not regularly engage with stakeholders to help them understand the data and what it means.
Career readiness data is not used to support the state’s communications plan, if such a plan exists.
The state does not regularly identify and leverage career readiness champions.
The state does not identify or target communications to different audiences.

Criteria for Rating 4 (Established)

The state provides trainings, tools, templates and guides to help local practitioners translate their data for stakeholders in their communities.
State staff are deployed in the field to help stakeholders understand the data and what it means.
Career readiness indicators are integrated into the state’s communications plan and regularly shared across multiple platforms and media.
Career readiness champions are equipped with relevant data to tell a nuanced story.
The state differentiates assets and communication of data by target audience based on need and relevance.

Your Evidence

Your Rating

Overall Rating

6. Information is used effectively to promote quality and equity in career pathways.

Career readiness data is integrated into policymaking and decisionmaking processes to further a statewide career preparation system that is high quality and equitable.

A. Career pathway approval and renewal processes are data driven.

Criteria for Rating 1 (Undefined)

There is no consistent process for reviewing and approving career pathways, or if such a process exists, it is not sufficiently data driven.
No data-informed processes are in place to retool or phase out career pathways that are outdated, low quality or not serving learners equitably.
There is no statewide, cross-sector process to identify credentials of value using labor market information.
There are limited to no consistent statewide processes for mapping career pathways with priority industry sectors to evaluate labor market alignment.
There are limited to no processes — or processes are underutilized — at the local level to examine labor market information and make decisions about developing, expanding, scaling or phasing out career pathways.

Criteria for Rating 4 (Established)

The state has a clear process for reviewing and approving career pathways that draws on labor market information, learner enrollment data and outcomes data to determine whether career pathways meet quality thresholds.
Processes are in place to retool or phase out, in a timely manner, career pathways that are outdated, low quality or not serving learners equitably.
Robust processes exist to identify credentials of value, and these processes draw on labor market information to identify credentials tied to in-demand opportunities that offer high wages or the ability to stack to higher paying credentials. Processes are reviewed regularly, and approved credential lists are made available publicly.
Program quality and labor market alignment are well defined and measurable. Appropriate crosswalks (e.g., CIP, SOC, SCED, O*NET) are developed to assist in determining whether programs are aligned to labor market needs.
Processes are in place at the local level for leaders in the education, workforce and business communities to review data and make informed decisions about developing, expanding, scaling and phasing out career pathways (e.g., the Perkins V CLNA, WIOA sector partnerships).

Your Evidence

Your Rating

B. State- and local-level decisionmakers regularly reflect and act upon data to inform policy and improve equity, access and quality.

Criteria for Rating 1 (Undefined)

There are limited to no state processes for inter- and intra-agency teams to review and respond to patterns and insights in their career readiness data.
Policymakers are not provided with or encouraged to use career readiness data to enact data-informed policy.
State leaders do not use data to identify and address state policies that disproportionately affect access and success for learners based on race/ethnicity, gender and special population status.

Criteria for Rating 4 (Established)

Inter- and intra-agency teams come together at least once a quarter at the state and local levels and review data to make or inform policy decisions.
Policymakers are equipped with and regularly leverage career readiness data to enact data-informed policy. Data is made available to key policymakers at least once a year through formal reports.
State and local leaders leverage data to identify and address policies that disproportionately affect access and success for learners based on race/ethnicity, gender and special population status.
State and local leaders leverage data to enact policies to improve the quality of career pathways.

Your Evidence

Your Rating

C. State and local leaders regularly identify and respond to opportunity gaps by race/ethnicity, gender and special population status to ensure equitable access to and success in career pathways.

Criteria for Rating 1 (Undefined)

Limited to no information is available to identify disparities by race/ethnicity, gender and special population status in the enrollment, success, completion and outcomes of learners in career pathways.
Local practitioners rarely review opportunity gaps by race/ethnicity, gender and special population status to address disparities in access to and success in career pathways.
No workshops or trainings are provided to help local practitioners identify opportunity gaps in their data and examine root causes.
The state provides limited to no information about evidence-based practices for closing opportunity gaps for learners in career pathways.
Accountability is leveraged only to comply with federal law, and the state does not intentionally or proactively target resources and supports to institutions with significant and persistent opportunity gaps.

Criteria for Rating 4 (Established)

Automated feedback reports are available at the college, district, school and program levels to illustrate disparities by race/ethnicity, gender and special population status in the enrollment, success, completion and outcomes of learners in career pathways.
Processes are in place at the local level to review equity and opportunity gaps by race/ethnicity, gender and special population status at least once a year and address disparities in access to and success in career pathways.
Workshops and trainings are provided at least once a year to help local practitioners identify opportunity gaps in their data and examine root causes.
The state identifies, evaluates and disseminates evidence-based practices to close opportunity gaps for learners in career pathways.
Data is used to hold schools and institutions accountable for learner success; resources and supports are targeted to institutions with significant and persistent opportunity gaps.

Your Evidence

Your Rating

D. Local practitioners have access to real-time data that they use to target interventions.

Criteria for Rating 1 (Undefined)

No early warning systems are in place at the school, college, district and technical center levels to identify learners who need additional support or intervention to be successful in their career pathway.
Local practitioners do not regularly leverage their data to target integrated student supports to learners who are most in need.
Limited to no training is provided to local leaders on how they can examine their data to identify early warning indicators.

Criteria for Rating 4 (Established)

Early warning systems are used at the school, college, district and technical center levels to identify and support learners who need additional support or intervention to be successful in their career pathway.
Integrated student supports are deployed in response to early warning indicators.
Local leaders are trained on how to monitor early warning indicators that are predictive of future career pathways milestones such as credential attainment, work-based learning completion, and successful transition into high-wage employment.

Your Evidence

Your Rating

Overall Rating

Once the form is complete, print a copy of your responses or save a copy for your records. The current version of this website does not allow for submission of the form.