publications

2023

1. A Static Evaluation of Code Completion by Large Language Models

   Hantian Ding, Varun Kumar, Yuchen Tian, and 7 more authors

   In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 5: Industry Track), Jul 2023

   Abs

   Large language models trained on code have shown great potential to increase productivity of software developers. Several execution-based benchmarks have been proposed to evaluate functional correctness of model-generated code on simple programming problems. Nevertheless, it is expensive to perform the same evaluation on complex real-world projects considering the execution cost. On the other hand, static analysis tools such as linters, which can detect errors without running the program, haven’t been well explored for evaluating code generation models. In this work, we propose a static evaluation framework to quantify static errors in Python code completions, by leveraging Abstract Syntax Trees. Compared with execution-based evaluation, our method is not only more efficient, but also applicable to code in the wild. For experiments, we collect code context from open source repos to generate one million function bodies using public models. Our static analysis reveals that Undefined Name and Unused Variable are the most common errors among others made by language models. Through extensive studies, we also show the impact of sampling temperature, model size, and context on static errors in code completions.

2. ReCode: Robustness Evaluation of Code Generation Models

   Shiqi Wang, Zheng Li, Haifeng Qian, and 11 more authors

   In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Jul 2023

   Abs

   Code generation models have achieved impressive performance. However, they tend to be brittle as slight edits to a prompt could lead to very different generations; these robustness properties, critical for user experience when deployed in real-life applications, are not well understood. Most existing works on robustness in text or code tasks have focused on classification, while robustness in generation tasks is an uncharted area and to date there is no comprehensive benchmark for robustness in code generation. In this paper, we propose ReCode, a comprehensive robustness evaluation benchmark for code generation models. We customize over 30 transformations specifically for code on docstrings, function and variable names, code syntax, and code format. They are carefully designed to be natural in real-life coding practice, preserve the original semantic meaning, and thus provide multifaceted assessments of a model’s robustness performance. With human annotators, we verified that over 90% of the perturbed prompts do not alter the semantic meaning of the original prompt. In addition, we define robustness metrics for code generation models considering the worst-case behavior under each type of perturbation, taking advantage of the fact that executing the generated code can serve as objective evaluation. We demonstrate ReCode on SOTA models using HumanEval, MBPP, as well as function completion tasks derived from them. Interesting observations include: better robustness for CodeGen over InCoder and GPT-J; models are most sensitive to syntax perturbations; more challenging robustness evaluation on MBPP over HumanEval.

3. Resolving Ambiguities in Text-to-Image Generative Models

   Ninareh Mehrabi, Palash Goyal, Apurv Verma, and 7 more authors

   In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Jul 2023

   Abs

   Natural language often contains ambiguities that can lead to misinterpretation and miscommunication. While humans can handle ambiguities effectively by asking clarifying questions and/or relying on contextual cues and common-sense knowledge, resolving ambiguities can be notoriously hard for machines. In this work, we study ambiguities that arise in text-to-image generative models. We curate the Text-to-image Ambiguity Benchmark (TAB) dataset to study different types of ambiguities in text-to-image generative models. We then propose the Text-to-ImagE Disambiguation (TIED) framework to disambiguate the prompts given to the text-to-image generative models by soliciting clarifications from the end user. Through automatic and human evaluations, we show the effectiveness of our framework in generating more faithful images aligned with end user intention in the presence of ambiguities.

4. An analysis of the effects of decoding algorithms on fairness in open-ended language generation

   Jwala Dhamala, Varun Kumar, Rahul Gupta, and 2 more authors

   In 2022 IEEE Spoken Language Technology Workshop (SLT), Jul 2023

2022

1. Mitigating Gender Bias in Distilled Language Models via Counterfactual Role Reversal

   Umang Gupta, Jwala Dhamala, Varun Kumar, and 7 more authors

   In Findings of the Association for Computational Linguistics: ACL 2022, Jul 2022
2. On the Intrinsic and Extrinsic Fairness Evaluation Metrics for Contextualized Language Representations

   Yang Trista Cao, Yada Pruksachatkun, Kai-Wei Chang, and 4 more authors

   In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), Jul 2022
3. Multi-lingual Evaluation of Code Generation Models

   Ben Athiwaratkun, Sanjay Krishna Gouda, Zijian Wang, and 8 more authors

   In The Eleventh International Conference on Learning Representations, Jul 2022

2021

1. Industry Scale Semi-Supervised Learning for Natural Language Understanding

   Luoxin Chen, Francisco Garcia, Varun Kumar, and 2 more authors

   In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: Industry Papers, Jun 2021

   Abs

   This paper presents a production Semi-Supervised Learning (SSL) pipeline based on the student-teacher framework, which leverages millions of unlabeled examples to improve Natural Language Understanding (NLU) tasks. We investigate two questions related to the use of unlabeled data in production SSL context: 1) how to select samples from a huge unlabeled data pool that are beneficial for SSL training, and 2) how does the selected data affect the performance of different state-of-the-art SSL techniques. We compare four widely used SSL techniques, Pseudo-label (PL), Knowledge Distillation (KD), Virtual Adversarial Training (VAT) and Cross-View Training (CVT) in conjunction with two data selection methods including committee-based selection and submodular optimization based selection. We further examine the benefits and drawbacks of these techniques when applied to intent classification (IC) and named entity recognition (NER) tasks, and provide guidelines specifying when each of these methods might be beneficial to improve large scale NLU systems.

2. BOLD: Dataset and Metrics for Measuring Biases in Open-Ended Language Generation

   Jwala Dhamala, Tony Sun, Varun Kumar, and 4 more authors

   In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, Jun 2021

   Abs

   Recent advances in deep learning techniques have enabled machines to generate cohesive open-ended text when prompted with a sequence of words as context. While these models now empower many downstream applications from conversation bots to automatic storytelling, they have been shown to generate texts that exhibit social biases. To systematically study and benchmark social biases in open-ended language generation, we introduce the Bias in Open-Ended Language Generation Dataset (BOLD), a large-scale dataset that consists of 23,679 English text generation prompts for bias benchmarking across five domains: profession, gender, race, religion, and political ideology. We also propose new automated metrics for toxicity, psycholinguistic norms, and text gender polarity to measure social biases in open-ended text generation from multiple angles. An examination of text generated from three popular language models reveals that the majority of these models exhibit a larger social bias than human-written Wikipedia text across all domains. With these results we highlight the need to benchmark biases in open-ended language generation and caution users of language generation models on downstream tasks to be cognizant of these embedded prejudices.

3. Protoda: Efficient Transfer Learning for Few-Shot Intent Classification

   Manoj Kumar, Varun Kumar, Hadrien Glaude, and 3 more authors

   In 2021 IEEE Spoken Language Technology Workshop (SLT), Jun 2021

2020

1. Data Augmentation using Pre-trained Transformer Models

   Varun Kumar, Ashutosh Choudhary, and Eunah Cho

   In Proceedings of the 2nd Workshop on Life-long Learning for Spoken Language Systems, Dec 2020

   Abs

   Language model based pre-trained models such as BERT have provided significant gains across different NLP tasks. In this paper, we study different types of transformer based pre-trained models such as auto-regressive models (GPT-2), auto-encoder models (BERT), and seq2seq models (BART) for conditional data augmentation. We show that prepending the class labels to text sequences provides a simple yet effective way to condition the pre-trained models for data augmentation. Additionally, on three classification benchmarks, pre-trained Seq2Seq model outperforms other data augmentation methods in a low-resource setting. Further, we explore how different pre-trained model based data augmentation differs in-terms of data diversity, and how well such methods preserve the class-label information.

2. Digging into User Control: Perceptions of Adherence and Instability in Transparent Models

   Alison Smith-Renner, Varun Kumar, Jordan Boyd-Graber, and 2 more authors

   In Proceedings of the 25th International Conference on Intelligent User Interfaces, Dec 2020

   Abs

   We explore predictability and control in interactive systems where controls are easy to validate. Human-in-the-loop techniques allow users to guide unsupervised algorithms by exposing and supporting interaction with underlying model representations, increasing transparency and promising fine-grained control. However, these models must balance user input and the underlying data, meaning they sometimes update slowly, poorly, or unpredictably—either by not incorporating user input as expected (adherence) or by making other unexpected changes (instability). While prior work exposes model internals and supports user feedback, less attention has been paid to users’ reactions when transparent models limit control. Focusing on interactive topic models, we explore user perceptions of control using a study where 100 participants organize documents with one of three distinct topic modeling approaches. These approaches incorporate input differently, resulting in varied adherence, stability, update speeds, and model quality. Participants disliked slow updates most, followed by lack of adherence. Instability was polarizing: some participants liked it when it surfaced interesting information, while others did not. Across modeling approaches, participants differed only in whether they noticed adherence.

2019

1. A Closer Look At Feature Space Data Augmentation For Few-Shot Intent Classification

   Varun Kumar, Hadrien Glaude, Cyprien Lichy, and 1 more author

   In Proceedings of the 2nd Workshop on Deep Learning Approaches for Low-Resource NLP (DeepLo 2019), Nov 2019

   Abs

   New conversation topics and functionalities are constantly being added to conversational AI agents like Amazon Alexa and Apple Siri. As data collection and annotation is not scalable and is often costly, only a handful of examples for the new functionalities are available, which results in poor generalization performance. We formulate it as a Few-Shot Integration (FSI) problem where a few examples are used to introduce a new intent. In this paper, we study six feature space data augmentation methods to improve classification performance in FSI setting in combination with both supervised and unsupervised representation learning methods such as BERT. Through realistic experiments on two public conversational datasets, SNIPS, and the Facebook Dialog corpus, we show that data augmentation in feature space provides an effective way to improve intent classification performance in few-shot setting beyond traditional transfer learning approaches. In particular, we show that (a) upsampling in latent space is a competitive baseline for feature space augmentation (b) adding the difference between two examples to a new example is a simple yet effective data augmentation method.

2. Why Didn’t You Listen to Me? Comparing User Control of Human-in-the-Loop Topic Models

   Varun Kumar, Alison Smith-Renner, Leah Findlater, and 2 more authors

   In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Jul 2019

   Abs

   To address the lack of comparative evaluation of Human-in-the-Loop Topic Modeling (HLTM) systems, we implement and evaluate three contrasting HLTM modeling approaches using simulation experiments. These approaches extend previously proposed frameworks, including constraints and informed prior-based methods. Users should have a sense of control in HLTM systems, so we propose a control metric to measure whether refinement operations’ results match users’ expectations. Informed prior-based methods provide better control than constraints, but constraints yield higher quality topics.

3. Efficient Semi-Supervised Learning for Natural Language Understanding by Optimizing Diversity

   Eunah Cho, He Xie, John P. Lalor, and 2 more authors

   In 2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), Jul 2019

2018

1. Closing the Loop: User-Centered Design and Evaluation of a Human-in-the-Loop Topic Modeling System

   Alison Smith, Varun Kumar, Jordan Boyd-Graber, and 2 more authors

   In 23rd International Conference on Intelligent User Interfaces, Jul 2018

   Abs

   Human-in-the-loop topic modeling allows users to guide the creation of topic models and to improve model quality without having to be experts in topic modeling algorithms. Prior work in this area has focused either on algorithmic implementation without understanding how users actually wish to improve the model or on user needs but without the context of a fully interactive system. To address this disconnect, we implemented a set of model refinements requested by users in prior work and conducted a study with twelve non-expert participants to examine how end users are affected by issues that arise with a fully interactive, user-centered system. As these issues mirror those identified in interactive machine learning more broadly, such as unpredictability, latency, and trust, we also examined interactive machine learning challenges with non-expert end users through the lens of human-in-the-loop topic modeling. We found that although users experience unpredictability, their reactions vary from positive to negative, and, surprisingly, we did not find any cases of distrust, but instead noted instances where users perhaps trusted the system too much or had too little confidence in themselves.

2016

1. The GW/UMD CLPsych 2016 Shared Task System

   Ayah Zirikly, Varun Kumar, and Philip Resnik

   In Proceedings of the Third Workshop on Computational Linguistics and Clinical Psychology, Jun 2016
2. Mining Shapes of Expertise in Online Social Q&A Communities

   Varun Kumar, and Niranjan Pedanekar

   In Proceedings of the 19th ACM Conference on Computer Supported Cooperative Work and Social Computing Companion, Jun 2016

   Abs

   Expertise of an individual is metaphorically defined by shapes of letters such as I, T, M and hyphen, depending on her expertise in an area (depth) and the number of areas of interest (width). Industries have now started recruiting people with specific shapes of expertise. In this poster, we introduce the idea of mining shapes of user expertise in a typical online social Question and Answer (Q&A) community where expert users often answer questions posed by other users. We report observations on distribution of different shapes of expertise in a StackExchange community called Super User.