Schedule

Title: Two F-words in Peer Review (Fraud and Feedback)

Abstract: We will present two major challenges in peer review, propose solutions with guarantees, and discuss important open problems.

(1) Fraud: There have been several recent discoveries of fraud in peer review: A group of participants form a coalition, get assigned each other’s papers by manipulating the system, and then accept each others’ papers. We present an algorithm which mitigates such fraud by randomizing reviewer assignments, and does not rely on assumptions about the malicious behavior. The algorithm yields an optimal-quality assignment subject to randomization constraints, and we will also discuss experiments characterizing this tradeoff.

(2) Feedback: Real-world systems rely on feedback about their performance for their continual improvement. A useful means of obtaining feedback about the peer-review process is to ask authors’ opinions. However, author opinions are significantly biased by whether their paper was accepted. We formulate this problem and present algorithms, with statistical guarantees, to debias such feedback. Our work relies on the key observation that the direction of this bias is known: the program chairs know which authors’ papers were accepted.

For those interested to know more, an overview of challenges and computational solutions in peer review is available here.

Title: Designing School Choice for Diversity in the San Francisco Unified School District

Abstract: More than 65 years after the “Brown v. Board of Education” ruling that school segregation is unconstitutional, public schools across the U.S. are resegregating. In attempts to disentangle school segregation from neighborhood segregation, many cities have adopted policies for city-wide choice. However, these policies have largely not improved patterns of segregation. From 2018-2020, we worked with the San Francisco Unified School District (SFUSD) to design a new policy for student assignment system that meets the district’s goals of diversity, predictability, and proximity. To develop potential policies, we used optimization techniques to augment and operationalize the district’s proposal of restricting choice to zones. We compared these to district-wide choice approaches typically suggested by the school choice literature. We find that appropriately designed zones with minority reserves can achieve all the district’s goals, at the expense of choice, and choice can resegregate diverse zones. Using predictive choice models developed using historical choice data, we show that a zone-based policy can decrease the percentage of racial minorities in high-poverty schools from 29% to 11%, decrease the average travel distance from 1.39 miles to 1.29 miles, and improve predictability, but reduce the percentage of students assigned to one of their top 3 programs from 80% to 59%. Traditional district-wide choice approaches can improve diversity and choice at the expense of proximity. Our work informed the design and approval of a zone-based policy for use starting the 2024-25 school year.

“Efficient and targeted COVID-19 border testing via reinforcement learning”

Title: Reimagining the Machine Learning Life Cycle in Education (and Beyond)

Abstract: With the rapid proliferation of machine learning technologies in the education sphere, we address an urgent need to investigate whether the development of these machine learning technologies supports holistic education principles and goals. We present findings from a cross-disciplinary interview study of education researchers, investigating whether the stated or implied ``social good’’ objectives of ML4Ed research papers are aligned with the ML problem formulation, objectives, and interpretation of results. Our findings shed light on two main gaps that each calls to attention an overlooked step in the ML life cycle: 1) the translation of education goals via ML problem formulation and 2) the translation of predictions to interventions. This talk is based on joint work with Serena Wang, Rediet Abebe and Tolani Britton.