Week-4: Evaluating Fairness and Generalization in Native Language Identification

Evaluating Fairness and Generalization in Native Language Identification

An Evaluation perspective of NLI model development

Introduction

Native Language Identification (NLI) seeks to infer an author’s first language (L1) from their writing in a second language (L2). While earlier studies reported strong performance on curated learner corpora, contemporary deployments confront a markedly different landscape: user‑generated content (UGC) that is informal, topical, and noisy. In such settings, conventional accuracy metrics can obscure a critical issue – models may succeed by exploiting spurious topical cues rather than genuine cross‑linguistic transfer. This blog post explains the evaluation framework that treats performance, fairness, and generalization as co‑equal objectives, with explicit tests for topic leakage and mechanisms for rejecting unseen languages.

The practical question is not only “How accurate is the model?” but “Accurate on what basis, and under what distributional shifts?” We therefore emphasize (i) cross‑topic evaluation to decouple linguistic signal from domain content, (ii) bias‑leakage auditing to quantify spurious correlations, and (iii) open‑set recognition so the system can state “unknown” when confronted with L1s absent from training. Together, these components support trustworthy NLI suitable for research and pedagogical use.

Continue reading “Week-4: Evaluating Fairness and Generalization in Native Language Identification”

Week-3: Choosing the Right Research Method for My AI-Based NLI Study

Choosing the Right Research Method for My AI-Based NLI Study

Introduction

Research method selection constitutes a pivotal decision in academic inquiry: it structures how evidence is gathered, the standards by which results are evaluated, and the extent to which conclusions can be generalized and replicated. In the context of my MSc. project, I investigate Native Language Identification (NLI) within user-generated English text by developing a bias-aware, generalizable framework that integrates Large Language Model (LLM) embeddings, topic debiasing, and open-set recognition. This post articulates the justification for adopting a quantitative experimental–comparative design and explains how this approach enables systematic assessment and evaluation of project outcomes including model accuracy, fairness, and robustness.

Continue reading “Week-3: Choosing the Right Research Method for My AI-Based NLI Study”

Week-2: Project Management Tool – JIRA

Using JIRA and SCRUM for MSc Research Project Management: A Modern Alternative to Traditional Tools

Introduction

Effective project management is a cornerstone of successful postgraduate research. Yet, many students and researchers still rely on rigid, timeline-based tools such as Microsoft Project, which follow a Waterfall approach – linear, sequential, and often unsuitable for the dynamic nature of research.

In contrast, JIRA, a product by Atlassian originally designed for software engineering teams, offers a flexible, agile-based approach to managing tasks, uncertainties, and evolving priorities. When used with the SCRUM framework, JIRA can transform a Master’s research project into a structured yet adaptive workflow – ideal for data-driven and iterative fields such as Artificial Intelligence, Computer Science, and Data Analytics. Continue reading “Week-2: Project Management Tool – JIRA”

Paper#1: A Report on the First Native Language Identification Shared Task

Paper link: A Report on the First Native Language Identification Shared Task (Tetreault, Blanchard & Cahill, 2013)

Paper Reading – First Native Language Identification (NLI) Shared Task

This post summarizes the first shared task on Native Language Identification (NLI)- predicting a writer’s native language (L1) from essays written in a learned language (here, English). It standardizes data, tasks, and evaluation to enable meaningful comparison across 29 participating teams, and remains a foundational benchmark for educational NLP and authorship profiling.

Why this matters

NLI supports targeted feedback for language learners (different L1s show distinct error tendencies) and contributes to authorship profiling. Before this effort, research relied on small, inconsistent corpora (often ICLE), making results hard to compare. This shared task fixed that by providing a large, balanced corpus and uniform evaluation.

Continue reading “Paper#1: A Report on the First Native Language Identification Shared Task”

Week-1: My Research

Mitigating Bias in Native Language Identification Using Large Language Models and Open-Set Recognition

Introduction

Language reflects traces of our linguistic background – even when we write in a second language. My research explores this fascinating connection through the task of Native Language Identification (NLI), which aims to predict an author’s first language (L1) based on their writing in another language (L2).

While early studies relied on formal learner essays such as the TOEFL11 corpus (Tetreault et al., 2013), these datasets capture only controlled, classroom-like writing. In contrast, today’s online communication is full of informal expressions, emojis, and cultural slang. My work therefore focuses on user-generated content (UGC) – specifically the Reddit-L2 dataset (Rabinovich et al., 2018) – to study NLI “in the wild,” where text is messy but authentically human. Continue reading “Week-1: My Research”