Emma Klugman

Emma Klugman

Education Policy and Data Science Doctoral Candidate

Harvard University

Biography

I am a psychometrician and statistician with research focusing on the intersection of data science, testing, education, and social good. I am in my final year as a doctoral student at the Harvard Graduate School of Education in Education Policy & Program Evaluation and in Data Science. I hold a Masters in Education and a Bachelors in Statistics and Computer Science from Harvard. My work has been published in Educational Measurement: Issues and Practice and presented at the annual meeting of the National Council on Measurement in Education. I also do cross-disciplinary work, including a recent article in the Journal of Social Media in Society, and an article I co-authored with some former students appeared in the journal Cannabis and Cannabinoid Research.

My dissertation committee is composed of my advisor, Andrew Ho (chair), Luke Miratrix, and Sebastian Munoz-Najar Galvez.

My dissertation research focuses on the teaching of data science and statistics, across the following three projects:

  1. I employ text analysis and psychometric methods to explore what is being taught in college statistics and data science courses, using a large dataset of course syllabi. I seek to measure modern vs. traditional pedagogical approaches, as well as describing continuity vs. change.

  2. I use latent profile analysis and visualizations to describe and characterize the population of US students who take statistics while in high school.

  3. I carry out an original mixed methods study measuring the interpretability and trustworthiness of machine learning models among highschoolers, as well as exploring their attitudes towards ethical questions around AI’s role in society.

In addition to my dissertation work, I recently developed and taught a new course titled “Big Data, Surveillance, Algorithmic Bias, and Ethics in Education Data Science” with my inimitable colleague Avriel Epps-Darling, offered for graduate students at Harvard in the spring of 2022. During my graduate career, I have also worked with my adviser, Professor Andrew Ho, in assessing validity evidence for state test score uses, creating recommendations and tools for an appropriate role for testing in a COVID-disrupted era, teaching psychometric methods, and updating the Stanford Education Data Archive.

Beyond my PhD, I have experience across government, industry, and academia, including work with ETS, the Behavioural Economics Team of the Australian Government, the National Center for the Improvement of Educational Assessment, the World Bank, a successful state lawsuit for education funding equity, and several years of teaching, mentoring, and course development. In my free time, I enjoy baking, tea, volleyball, and hosting board game nights.

Interests

  • Conducting Research on Statistics & Data Science Education
  • Teaching Critical Thinking & Ethics in Data Science
  • Exploring Impacts of College Admissions Policies
  • Evaluating Uses of Testing in Education

Education

  • PhD Candidate in Education Policy and Data Science

    Harvard University

  • MA in Education, 2020

    Harvard University

  • BA in Statistics and Computer Science, 2018

    Harvard College

Open Tests in Covid-19 era

We present a recipe that shows how schools, districts, and states can create and score a test comprised of publicly available ingredients (past questions) from state websites. Code and tutorial available online.

See The Code Read The Article

Teaching Experience

Original Course: HGSE A218A: Big Data, Surveillance, Algorithmic Bias, and Ethics in Education Data Science

I was as an Instructor at Harvard for my original course “Big Data, Surveillance, Algorithmic Bias, and Ethics in Education Data Science”. I co-designed and co-taught this course with my inimitable colleague Avriel Epps-Darling. I have since shared one of the resources we developed for this course at a teaching conference. I am very proud of this course, and am happy to talk more about our lessons learned in teaching it. Our course summary:

Data permeates education. Multimodal learning analytics, personalized learning, and data-driven autonomous decision making have all gained widespread acceptance, but is it ethical to collect millions of data points from children? Is this data collection and data use fair? Ethical? Oppressive? Who gets to decide?

In this course, we will uncover the choices, biases, and assumptions that permeate data pipelines, from data collection through labeling, modeling, de-identifying, publishing, and model deployment. Students will develop their critical awareness of ongoing issues in this space, gain familiarity with a suite of ethical and ideological frameworks that can be used to evaluate cases, and develop skills in writing and speaking about these topics.

We will do this in the context of several case studies, including the use of machine learning algorithms for identifying “at-risk” students by merging disciplinary and welfare data, surveillance of student data on school devices and networks, the statistical risks associated with anonymizing data by suppressing sample minorities, and the challenges of assessing algorithmic fairness in the context of proprietary data and black-box algorithms. Throughout, we will discuss themes of cost-benefit analysis, the different stakeholder interests, responsibility and buck-passing, anti-racism, intersectionality, and the role of the law.

My other teaching experiences are detailed below:

Statistics: I served as a Teaching Fellow with Prof. Luke Miratrix for Stat 151/HGSE S043: Multilevel and Longitudinal Models (Fall 2021). I co-taught review sections and provided feedback on student work, including students’ original research for their final projects, one of we have since helped publish with me as a co-author. I received a Certificate of Distinction in Teaching for my work in this course, based on outstanding student evaluations.

I have also served as a Teaching Fellow with Dr. Joe McIntyre to teach HGSE S010B: Designing Surveys and Questionnaires: Principles and Methods, where I taught weekly review sessions and gave feedback on student projects. I worked with Prof. Kerrie Nelson as a Course Assistant for Harvard Stat 100: Introduction to Quantitative Methods for the Social Sciences and Humanities (Fall 2017). I planned and taught weekly sections, graded problem sets and exams, and held weekly office hours. I served as a Peer Tutor in 2017 for Harvard Stat 111: Introduction to Statistical Inference.

Mathematics: I served as a Teaching Fellow with Dr. Brendan Kelly for Math Q: Quantitative Analysis for Economics and the Social Sciences (Fall 2022 and Spring 2023). I helped to build out the course’s curriculum and resources in data analysis, including randomization-based statistical inference, R programming, and statistical modelling. I previously served as a Quantitative Reasoning Mentor at the Crimson Summer Academy (Summer 2018), a Harvard programme for low-income, high-achieving secondary school students from the Cambridge/Boston area. I also served as a Teacher Aide in high school Maths, Statistics, and Calculus classes at Otumoetai College in New Zealand (2015).

Text Analysis: I worked with Prof. Sebastian Munoz-Najar Galvez to build and teach a new spring course, HGSE S59: Quantitative Text Analysis for Education Research (Spring 2021). I served as the course’s Teaching Fellow, writing and teaching weekly lab sessions, grading student work, and helping to guide final projects. We also worked together in the fall of 2020 to develop the course syllabus, assignments, and pedagogy.

Data Science: I served as a Teaching Fellow with Prof. Luke Miratrix to teach HGSE S22: Introduction to Statistical Computing and Data Science in Education (Spring 2021). I taught fortnightly lab sessions, supported students with office hours, and provided feedback on student work and final projects.

Psychometrics: I served as a Teaching Fellow with Prof. Andrew Ho for HGSE S61: Statistical and Psychometric Methods for Educational Measurement (Fall 2020). I co-taught review sections and provided feedback on student work.

Policymaking: I served as the Senior Teaching Assistant with Dr. Teddy Svoronos’ for Evidence for Decisions (Summer 2019), a Harvard Kennedy School course in the new Public Leadership Credential. I managed a team of three other teaching assistants, liaising with programme staff, providing feedback on student work, and holding office hours to support learners.

Ethics: In addition to building out my original data ethics course (described above) I have twice served as a Teaching Fellow with Prof. Jim Waldo and Prof. Mike Smith for Applied Computation 221: Critical Thinking in Data Science (Spring 2019 & Spring 2022), a course required for students in Harvard’s Master of Science in Data Science programme. I graded and provided feedback on student essays and problem sets, held office hours, and advised final projects.