Projects | Muhammad Umair

Research

Do LLMs Model Spoken Dialogue Accurately?

Transformer-based Large Language Models (LLMs) have shown impressive performance in various language tasks, but their ability to predict spoken language in natural interactions remains uncertain. Spoken and written languages differ significantly in syntax, pragmatics, and norms. While LLMs understand linguistic rules statistically, they may not grasp the normative structure of interactive spoken language. This study evaluates LLMs, specifically GPT-2 variants, on spoken dialogue. Fine-tuning on English dialogue transcripts, we assessed whether LLMs consider speaker identity in predictions. Although models used speaker identity, they occasionally hallucinated transitions for plausibility. Thus, LLMs generate norm-conforming text but do not yet replicate human conversational behavior accurately.

Transition Duration in Turn-Taking

Smooth turn-taking is vital for natural conversation, ensuring mutual comprehensibility. In human communication, timing between utterances is normatively constrained, with deviations conveying social information. However, spoken dialogue systems struggle with this, highlighting the need for robust turn-taking models. This paper examines human interaction data to develop an evidence-based model for turn-timing. Two models are proposed: a speaker-agnostic and a speaker-sensitive model. The latter predicts listeners' turn-taking propensity, improving dialogue system naturalness

GailBot

Researchers rely on detailed transcriptions of language, including paralinguistic features like overlaps, prosody, and intonation, which are time-consuming to create manually. No current Speech to Text (STT) systems integrate these features. GailBot, a new program, combines STT services with plugins to generate transcripts that follow Conversation Analysis standards. It allows adding and improving plugins for additional features. GailBot's architecture, heuristics, and machine learning are described, showing its improvement over existing transcription software.

LLMs know what to say but not when to speak

Turn-taking is crucial for smooth communication, yet LLMs often fail in natural conversations due to their training on written language and focus on turn-final TRPs. This paper evaluates LLMs' ability to predict within-turn TRPs, essential for natural dialogue. Using a unique dataset of participant-labeled within-turn TRPs, we assess LLMs' prediction accuracy. Results highlight LLMs' limitations in modeling spoken language dynamics, guiding the development of more naturalistic spoken dialogue systems.

Personal

Datasets for Conversational AI

This project allows users to load multiple datasets useful for research in conversational AI, spoken dialogue systems, and linguistics. It leverages the transformers dataset infrastructure to download and provide efficient access to variations of the same datasets - maintaining a single copy of the raw data and using minimal disk space to store dataset variations.

Image Distortion in Surgical Robots

When cameras take pictures, they are effectively mapping the light reflected by a 3D object in the world to a 2D image at the back of the camera. In this process, lenses in cameras produce distortions that are caused by random errors in manufacturing. We use calibration patterns to estimate and account for these distortions in the 'eyes' of a surgical robot.

Course

Nemo: The Server Robot

We've all experienced going out to restaurants - you wait for your server order, and wait to be served. Most of us engage in smalltalk while waiting for our order and are typically interrupted by the server. In this project, we build a prototype of a robot server that determines the best time to interject in a customer's conversation

G-Meta Plus

In this project, we aim to expand the G-Meta framework in two stages. First, we expand the sub-graphs of the support data by incorporating shortest path between generated sub-graphs. Second, we substitute G-Meta’s existing prototypical network component with a Relation Network for label prediction. This relation network component will generate a function that learns to compare support and query data instead of using prototypical networks.

NextGen Alert System

The NextGen alert system was designed to be an in-house SIEM system for Tufts Technology Services (TTS) as it transitions to an inexpensive commercial solution. Ideally, NextGen. would replicate all aspects of monitoring tools (such as ElasticWatcher) and send notifications if specific events occurred within multiple data clusters. Currently, this system is a Minimum Viable Product (MVP) that is able to monitor events, send email notifications, and provide a UI for simple data cluster queries.