HighMMT: Quantifying Modality & Interaction Heterogeneity for High-Modality Representation Learning

Paul Pu Liang , Yiwei Lyu , Xiang Fan , Jeffrey Tsaw , Yudong Liu , Shentong Mo , Dani Yogatama , Louis-Philippe Morency , Ruslan Salakhutdinov

arxiv, code

Submitted to TMLR 2022

An information theory-centric metric quantifying modality heterogeneity is introduced to measure affinity between modalities across tasks. This metric is used to inform parameter groupings in a Transformer based multimodal multitask model that achieves SOTA across several tasks with a significant reduction in parameters

Hybrid Neural Model for Solving Math Word Problems

With Raghav Bansal

Final Paper

We combined SOTA bottom up MWP solver with a SOTA top-down MWP solver into a novel hybrid model. Developed a novel query network to substitute probable subtrees from bottom up network with top down network's goal vectors. Achieved 75% theoretical accuracy on Math23k beating previous SOTA of 74%

Autovöt: An Autonomous RC Vehicle Convoy

With Joel Anyanti, Fausto Leyva

Final Paper | Video | Poster

We developed an RC vehicle convoy from scratch capable of autonomously navigating an obstacle course. Lead vehicle uses a perception stack and SLAM to send directions via bluetooth to following vehicle to navigate obstacles. Won 1st Runner Up out of 40+ projects in Spring 2021

RISCV Processor

Datapath

I designed and wrote a synthesizable superscalar out-of-order pipelined RISCV processor in SystemVerilog that implements a subset of the RV32I ISA. Implemented optimizations such as a BTB, instruction stall buffer, and pipeline flushes. Achieved 280 MIPS on perscribed benchmarks and 1st Quartile performance out of 20+ teams.

Recurrent GANs for Music Generation

With Azaan Rehman

Final Paper

We extended a baseline recurrent BiLSTM GAN for music generation and developed a novel architecture containing an input mapping network, convolution and attention layers, and a peripheral classifier using Tensorflow v1 in an AWS EC2 instance to generate statistically more realistic classical music compared to the baseline.

Power Grid RL Operator

Summer 2020

I implemented and trained a DQN and a Double Duelling DQN with PER agents from scratch on Tensorflow 2.0 to achieve 3500% increase in operation before failure and an 11000% increase in aggregate reward per episode on grid2op environment compared with the baseline agents.

Resume-Job Compatability Algorithm

With Samantha Ramsey and Shaan Dave

Winner of Credit Suisse Coding Challenge 2019 | Summer 2019

We designed an algorithm for picking the most qualified resume based on a job description through an NER model and a matching algorithm to compute an aggregate score. We also identified the most suitable job given a resume using word2vec with cosine similarity.

The Empathetic Jukebox

I trained an emotion classifier with OpenCV and generated Spotify playlists based on the user's emotion using the Spotify API, and plays the songs automatically in real-time by webscraping YouTube.

Apple Inc.

GPU Hardware Intern (virtual) | Fall 2021

I worked on GPU memory verification team. Integrated idle checks and developed coverage for 5 modules within GPU routing block. Developed novel strategy to hit previously un-hit coverage points using Xceligen ML tool to improve coverage of a coverpoint by over 10%

Credit Suisse

Technology Analyst Intern (virtual) | Summer 2020

I interned with Global Markets Data Analytics and developed a 2-stage pipelined model to extract bond information from Bloomberg chat data in Python in an Agile development environment.

18-100 Introduction to Electrical and Computer Engineering

TA | Fall 2021 - Fall 2022

I taught recitation, held office hours and small groups, and wrote exam and homework problems on a variety of concepts including digital logic, transistors, op amps, signal processing and filters, and machine learning.

10-701 Introduction to Machine Learning (PhD)

TA | Spring 2021

I taught recitation, held office hours, and wrote exam and homework problems on various ML concepts, including regression and optimization, deep learning, reinforcement learning, graphical models and learning theory.

18-349 Introduction to Embedded Systems

TA | Fall 2020

I held weekly small group sessions and office hours teaching fundamental concepts in embedded systems including ARM Thumbv2, serial protocols, timers and interrupts, threading and scheduling, networking, and security

33-141 Physics I for Engineering Students

SI Leader | Fall 2019

I organized and led biweekly sessions of 30+ students, as well as several exam review of 100+ students to review lecture material on kinematics, mechanics, energy and momentum, and thermodynamics. I also wrote and prepared problem sets and solutions for each session and exam review.

15-213 Introduction to Computer Systems

TA | Summer 2019

I led lecture activities and held office hours on content covering system-level I/O, network protocols, bit-level representations, x86-64 programming, exceptional control flow (processes and signals), virtual memory, and caching. I also developed and benchmarked test traces to stress students' implementations of malloc memory allocator.

33-142 Physics II for Engineering Students

EXCEL Leader | Spring 2019

I organized and led 3 weekly sessions of 9-10 students, covering material on electric fields, Gauss' law, magnetism, electromagnetic induction, and Maxwell's Equations. I also wrote and prepared problem sets for each session, and co-developed and co-led multiple exam reviews with 100+ students.

21-259 Calculus in Three Dimensions

EXCEL Leader | Fall 2019

I organized and led 3 weekly sessions of 9-10 students, covering material on multivariable limits, differentiation, integration, surface and line integrals, and Stoke's and Divergence Theorems. I also wrote and prepared problem sets for each session, and led 3 exam reviews with up to 30 students.

33-141 Physics I for Engineering Students

TAA | Fall 2019

I co-led biweekly recitations for 25+ undergraduate students, providing support on concepts including kinematics, mechanics, energy and momentum, and thermodynamics