Skip to main content

Evaluation and Research Plan

Overview: The Living Lab Model​

El Segundo USD commits to functioning as a Living Lab for AI education research. This commitment extends beyond program evaluation to knowledge generation that advances the field of K-12 AI literacy education nationally.

The Living Lab model means:

  • Rigorous documentation of implementation processes and outcomes
  • Partnership with university researchers studying AI and education
  • Publication of findings for practitioner and academic audiences
  • Hosting of visiting educators from districts considering adoption
  • Contribution to policy discussions at state and federal levels

This research plan details the evaluation methodology, data collection approaches, dissemination strategy, and longitudinal tracking that will transform the El Segundo experience into transferable knowledge.

Research Questions​

The El Segundo Model evaluation addresses five primary research questions:

Question 1: AI Literacy Development​

Can a comprehensive K-12 AI literacy program produce measurable improvements in student AI fluency, and what program components are most effective?

Sub-questions:

  • What is the trajectory of AI skill development across grade levels?
  • How does studio team participation compare to classroom-only instruction?
  • What assessment methods most accurately capture AI literacy?

Question 2: Gender Equity Outcomes​

Can deliberate intervention design close the gender gap in AI tool adoption, confidence, and career interest?

Sub-questions:

  • Do girls-only studio teams produce better outcomes for female students than mixed-gender teams?
  • What program elements are most effective in reducing AI anxiety among female students?
  • Do equity gains persist after program completion?

Question 3: Workforce Readiness​

Do portfolio-based demonstrations of AI capability provide measurable advantages in college admission and career placement?

Sub-questions:

  • How do employers evaluate student portfolios compared to traditional credentials?
  • Do program graduates show different career trajectories than comparison groups?
  • What portfolio elements are most valued by employers and admissions officers?

Question 4: Implementation Factors​

What implementation factors predict program success, and what are the primary barriers to adoption?

Sub-questions:

  • What teacher characteristics predict effective studio team mentorship?
  • What resources are minimally necessary for program implementation?
  • What district characteristics predict successful adoption?

Question 5: Scalability and Transferability​

Can the El Segundo Model be effectively replicated in districts with different characteristics?

Sub-questions:

  • What adaptations are needed for districts with different demographics?
  • What is the minimum resource level for effective implementation?
  • What implementation support do adopting districts require?

Quasi-Experimental Design​

Given the ethical and practical constraints of random assignment in educational settings, the El Segundo evaluation employs a quasi-experimental design with multiple comparison strategies.

Primary Design: Regression Discontinuity​

For research question outcomes where participation has a threshold (e.g., studio team enrollment), regression discontinuity design provides causal inference:

  • Students just above and just below participation thresholds serve as comparison groups
  • Analysis focuses on discontinuities at thresholds that cannot be attributed to selection effects
  • Multiple threshold points (e.g., grade eligibility, capacity limits) provide replication

Secondary Design: Difference-in-Differences​

For district-wide outcomes, difference-in-differences design compares:

  • El Segundo students before and after implementation
  • El Segundo students compared to similar students in comparison districts
  • The interaction term (El Segundo x post-implementation) provides treatment effect estimate

Comparison District Selection​

Three comparison districts are identified based on:

  • Similar demographic composition to El Segundo
  • Similar baseline academic achievement levels
  • Geographic proximity (Southern California)
  • No current AI literacy program implementation

Data sharing agreements are established before program launch to ensure comparison data availability.

Within-District Comparisons​

Within El Segundo, additional comparisons include:

  • Studio team participants versus non-participants (with propensity score matching)
  • Girls-only team participants versus mixed-gender team female participants
  • Early cohort versus later cohort (to assess program maturation effects)

Data Collection Methods​

Student Assessments​

AI Literacy Assessment Battery

  • Pre-program baseline assessment for all students
  • Annual administration for all students grades 6-12
  • Post-program assessment for studio team completers
  • Standardized instrument with established psychometric properties
  • Custom items developed for portfolio-based competencies

Assessment domains:

  • AI concept knowledge (recognition of AI applications, understanding of machine learning principles)
  • AI tool proficiency (practical skill demonstration)
  • Critical evaluation (ability to assess AI outputs, recognize bias)
  • Ethical reasoning (understanding of AI ethics issues)
  • Creative application (novel use of AI tools for problem-solving)

AI Attitudes and Self-Efficacy Scale

  • Measures AI anxiety, perceived competence, and career interest
  • Administered pre-program, mid-year, and post-program
  • Validated instrument with gender-specific subscales
  • Includes items on stereotype threat and belonging

Portfolio Assessment

  • Standardized rubric for portfolio evaluation
  • Multiple rater design (teacher, external rater, employer panel)
  • Inter-rater reliability calculated and reported
  • Portfolio elements tagged for component analysis

Teacher Data Collection​

Teacher AI Competency Assessment

  • Baseline and annual assessment of teacher AI skills
  • Self-report and performance-based components
  • Tracks progression through champion cohort tiers

Implementation Fidelity Measures

  • Classroom observation protocol for AI integration
  • Studio team observation protocol
  • Teacher logs of AI tool usage in instruction
  • Champion mentor tracking of peer support activities

Teacher Experience Surveys

  • Quarterly surveys on program implementation
  • Items on support adequacy, barriers encountered, perceived effectiveness
  • Open-ended items for qualitative analysis

Administrative Data​

Participation Records

  • Studio team enrollment and attendance
  • Professional development completion
  • Parent engagement participation

Academic Records

  • Course grades (with AI integration flagging)
  • Standardized test scores
  • College application and acceptance data
  • Scholarship awards

Long-Term Tracking

  • College major selection
  • College persistence and graduation
  • Employment outcomes (through alumni surveys and public records where available)
  • Wage data (self-reported through surveys)

Qualitative Data Collection​

Focus Groups

  • Student focus groups by grade band and studio team type
  • Teacher focus groups by champion status
  • Parent focus groups by participation level
  • Employer partner interviews

Case Studies

  • Purposive selection of 10-12 students for intensive tracking
  • Selection criteria include demographic diversity and range of outcomes
  • Annual interviews, portfolio analysis, and outcome tracking
  • Multi-year follow-up through college and early career

Implementation Documentation

  • Monthly implementation team meeting notes
  • Barrier and solution logs
  • Adaptation tracking as program evolves
  • Decision documentation for program modifications

External Evaluator Role​

El Segundo USD commits to engaging an external evaluation partner with the following responsibilities:

Independence and Credibility​

The external evaluator:

  • Has no financial stake in program continuation
  • Maintains data independence (separate storage and analysis)
  • Reports findings regardless of favorability to program
  • Has publication rights for all evaluation findings

Evaluation Functions​

Formative Evaluation

  • Quarterly implementation reports to program leadership
  • Early identification of implementation challenges
  • Recommendations for program adjustment
  • Monitoring of data collection quality

Summative Evaluation

  • Annual outcome reports
  • Comparison group analysis
  • Cost-effectiveness analysis
  • Scalability assessment

Research Functions

  • Primary responsibility for peer-reviewed publications
  • Conference presentations of findings
  • Contribution to policy briefs
  • Technical assistance to adopting districts

Evaluator Selection Criteria​

The external evaluator is selected based on:

  • Track record in K-12 education evaluation
  • Expertise in quasi-experimental methods
  • Experience with equity-focused program evaluation
  • Capacity for multi-year engagement
  • Commitment to open publication of findings

Finalist candidates are reviewed by an advisory committee including district leadership, teacher representatives, and university partners.

Publications and Dissemination Strategy​

The Living Lab commitment requires systematic dissemination of findings to maximize impact.

Peer-Reviewed Publications​

Target Journals

  • Educational Researcher (flagship educational research journal)
  • Journal of Research on Technology in Education
  • Computers and Education
  • Educational Evaluation and Policy Analysis
  • Journal of Computer Science Education

Publication Timeline

  • Year 1: Implementation design and baseline data (methodology paper)
  • Year 2: Early outcomes and implementation findings
  • Year 3: Comparative outcomes and equity analysis
  • Year 4: Longitudinal tracking and scalability findings
  • Year 5: Cost-effectiveness and policy implications

Practitioner Publications​

Target Outlets

  • Educational Leadership (ASCD)
  • Phi Delta Kappan
  • District Administration
  • EdSurge
  • THE Journal

Content Focus

  • Practical implementation guidance
  • Lessons learned and adaptations
  • Teacher perspectives and experiences
  • Student success stories (with permission)

Conference Presentations​

National Conferences

  • American Educational Research Association (AERA)
  • Association for Supervision and Curriculum Development (ASCD)
  • International Society for Technology in Education (ISTE)
  • National Science Teachers Association (NSTA)
  • Computer Science Teachers Association (CSTA)

Policy Conferences

  • Education Commission of the States
  • National Conference of State Legislatures
  • Congressional briefings (through education policy organizations)

Open Educational Resources​

All curriculum materials, assessment instruments, and implementation guides are published as open educational resources under Creative Commons licensing:

  • Curriculum framework documents
  • Studio team operational guides
  • Portfolio assessment rubrics
  • Teacher professional development materials
  • Parent education resources
  • Employer partnership templates

Media and Public Communication​

  • Annual press release of evaluation findings
  • Op-eds in education and general audience publications
  • Podcast and webinar appearances
  • Social media documentation of program activities
  • Documentary video production (pending funding)

Longitudinal Tracking: Graduate Outcomes​

The ultimate test of the El Segundo Model is graduate outcomes in college and career. The evaluation includes systematic longitudinal tracking.

Tracking Duration​

  • All program participants tracked for minimum 10 years post-graduation
  • Tracking begins with current high school students and continues with each cohort
  • First comprehensive graduate outcome data available Year 5 (for initial pilot cohort)

Tracking Methods​

Alumni Survey

  • Annual survey invitation to all program graduates
  • Incentivized participation (gift cards, lottery entries)
  • Covers education status, employment, wages, AI tool usage
  • Includes comparison items for non-participant alumni

Social Media and Public Records

  • LinkedIn profile analysis (with consent)
  • Public employment records where available
  • College outcome data through National Student Clearinghouse

Intensive Follow-Up Cohort

  • 50 graduates from first two cohorts selected for intensive tracking
  • Annual in-depth interviews
  • Career trajectory documentation
  • Retrospective program evaluation

Key Long-Term Outcomes​

Education Outcomes

  • College enrollment rate
  • College persistence and graduation rate
  • Major field selection (especially STEM/AI-related)
  • Graduate school enrollment

Career Outcomes

  • Employment rate
  • Employment in AI-related roles
  • Wage levels (compared to demographic-matched controls)
  • Career advancement trajectory
  • Entrepreneurship rates

Equity Outcomes

  • Gender gap in wages among program graduates versus comparison
  • Socioeconomic mobility indicators
  • Career outcome gaps by demographic category

University Research Partnerships​

The Living Lab model includes formal partnerships with university researchers:

Partnership Structure​

Research Affiliate Program

  • University faculty engaged as research affiliates
  • Provided access to program data (with appropriate protections)
  • Supported in developing grant proposals using El Segundo as research site
  • Given presentation opportunities at program events

Graduate Student Placements

  • Doctoral students placed as evaluation team members
  • Master's students conduct bounded studies within program
  • Undergraduate research assistants support data collection

Joint Grant Applications

  • University partners co-develop grant proposals
  • El Segundo serves as implementation site
  • Funding supports both research and program enhancement

Partner University Criteria​

Partner universities are selected based on:

  • Relevant faculty expertise (AI education, equity, workforce development)
  • Doctoral programs in education or related fields
  • Commitment to open publication
  • Geographic accessibility to El Segundo

Initial Partnership Development​

Year 1 priorities:

  • Establish partnerships with two to three universities
  • Develop data sharing agreements and IRB protocols
  • Launch first joint research project
  • Integrate graduate students into evaluation team

Data Governance and Ethics​

Student Data Protection​

All student data collection follows:

  • FERPA compliance requirements
  • California Student Privacy laws
  • District data governance policies
  • Parental consent protocols for research participation

Data Security​

  • All personally identifiable data stored on encrypted, access-controlled systems
  • External evaluator maintains separate secure data environment
  • De-identified datasets created for research analysis
  • Data retention policies specify storage duration and destruction protocols

Ethical Research Practice​

  • IRB approval obtained before any research data collection
  • Informed consent/assent obtained from participants
  • Right to withdraw from research participation without program consequences
  • Findings reported accurately regardless of favorability to program

Equity in Research​

  • Research questions explicitly address equity outcomes
  • Disaggregated analysis ensures no groups are invisible in aggregate data
  • Community advisory input on research priorities and interpretation
  • Findings shared with participant communities before publication

Budget for Evaluation and Research​

The evaluation and research component requires dedicated funding:

Year 1: $75,000

  • External evaluator contract: $50,000
  • Assessment instrument licensing and development: $10,000
  • Data systems development: $10,000
  • Graduate student support: $5,000

Year 2: $85,000

  • External evaluator contract: $55,000
  • Longitudinal tracking system: $15,000
  • Publication and dissemination: $10,000
  • Graduate student support: $5,000

Year 3+: $65,000/year

  • External evaluator contract: $45,000
  • Longitudinal tracking: $10,000
  • Publication and dissemination: $5,000
  • Graduate student support: $5,000

Grant Funding for Research Enhancement​

Additional evaluation capacity is sought through:

  • Institute of Education Sciences research grants
  • National Science Foundation STEM education grants
  • Foundation funding for equity-focused research
  • Corporate research partnerships

Timeline for Research Activities​

Pre-Implementation (Months 1-3)​

  • External evaluator selection and contracting
  • Comparison district agreements
  • Baseline data collection planning
  • IRB protocol submission
  • University partnership initiation

Year 1​

  • Baseline assessment administration
  • Implementation fidelity monitoring
  • Quarterly formative evaluation reports
  • First methodology paper drafted
  • University partnerships formalized

Year 2​

  • First outcome data analysis
  • Comparison group analysis begins
  • Conference presentations
  • Practitioner publication submissions
  • Joint grant proposal development

Year 3​

  • Comprehensive outcome analysis
  • Equity-focused publications
  • Longitudinal tracking system fully operational
  • First scalability studies
  • Policy brief development

Years 4-5​

  • Graduate outcome tracking begins
  • Cost-effectiveness analysis
  • Scalability replication studies
  • National dissemination emphasis
  • Model documentation for adoption

Conclusion: Knowledge Generation as Program Obligation​

The El Segundo Model represents a significant public investment in educational innovation. With that investment comes an obligation to generate knowledge that benefits not only El Segundo students but students nationally.

The Living Lab commitment ensures that:

  • Program effectiveness is rigorously evaluated with credible external assessment
  • Findings are transparently reported regardless of outcomes
  • Knowledge is widely disseminated through academic and practitioner channels
  • Other districts can learn from El Segundo's experience
  • Policy discussions are informed by evidence from implementation

This research plan provides the infrastructure for fulfilling that obligation. Through quasi-experimental design, comprehensive data collection, university partnerships, and systematic dissemination, the El Segundo experience will advance the field of K-12 AI education for all students.

El Segundo USD is committed to not only implementing an innovative program but to understanding why it works, for whom it works, and how it can be improved. This commitment to learning is, ultimately, what education is about.