Xin (Eric) WangAssistant ProfessorDepartment of Computer Science and Engineering Jack Baskin School of Engineering University of California, Santa Cruz Email: xwang366 [at] ucsc [dot] edu
|
|
Bio
Xin (Eric) Wang is an Assistant Professor of Computer Science and Engineering at UC Santa Cruz. His research interests include Natural Language Processing, Computer Vision, and Machine Learning, with an emphasis on building embodied AI agents that can communicate with humans using natural language to perform real-world multimodal tasks. He obtained his Ph.D. degree from UC Santa Barbara and Bachelor's degree from Zhejiang University. He also worked at Google AI, Facebook AI Research, Microsoft Research, and Adobe Research.Xin has served as Area Chair for NAACL 2021, CoNLL 2021, NLPCC 2021, and EMNLP 2020, Senior Program Committee (SPC) for AAAI 2022 and IJCAI 2021, and Session Chair for NAACL 2021, EMNLP 2020 and AAAI 2019. He organized multiple workshops and tutorials at ACL, CVPR, ICCV, AACL, etc. He received the CVPR Best Student Paper Award in 2019.
Research Interests
Xin's research interests include Natural Language Processing, Computer Vision, and Machine Learning, with an emphasis on building multimodal embodied agents that can communicate with humans using natural language to perform real-world tasks.- Natural Language Processing (natural language grounding, knowledge reasoning, fairness and bias, multilinguality)
- Computer Vision (vision and language, visual navigation and robotics, activity understanding in videos)
- Machine Learning (multimodal machine learning, self-supervised learning, reinforcement learning, deep learning)
Hiring
I am looking for highly-motivated students. Please read the information for prospective students and visitors and check out the most beautiful and unique campus of UCSC [YouTube video | bilibili video].Teaching
| Winter 2021 | CSE 142: Machine Learning |
| Spring 2021 | CSE 290C: Multimodal Deep Learning |
| Fall 2021 | CSE 142: Machine Learning |
| Winter 2022 | CSE 244B: Deep Learning for Natural Language Processing |
News
| [NEW!] | Our paper on Mitigating Gender Bias in Image Search is accepted to EMNLP 2021. Congratulations Jialu! |
| [NEW!] | Invited talk at Stanford Vision Lab. |
| [NEW!] | Our VALUE paper is accepted to NeurIPS 2021 (Datasets and Benchmarks). Congratulations to all the authors! |
| [NEW!] | Serving as Senior Program Committee (SPC) for AAAI 2022 and IJCAI-ECAI 2022. |
| [NEW!] | Invited talk at Microsoft Research and USC ISI. |
| [NEW!] | Co-organizing the 4th Workshop on Closing the Loop Between Vision and Language (CLVL) at ICCV 2021! |
| [NEW!] | I am giving a Tutorial on "From VQA to VLN: Recent Advances in Vision-and-Language Research" at CVPR 2021! |
| [NEW!] | Co-organizing the Second Workshop on Advances in Language and Vision Research (ALVR) at NAACL 2021! |
| [NEW!] | I am giving a keynote talk at the Third Workshop on Multimodal Artificial Intelligence at NAACL 2021 on June 6th! |
| [NEW!] | Our paper on Visual Question Rewriting was accepted to SIGIR 2021! |
| [NEW!] | I am serving as Area Chair for CoNLL 2021 and NLPCC 2021. |
| [NEW!] | Invited talk at Arizona State University. |
| [NEW!] | Two papers on multimodal style transfer learning for VLN and visual comparison were accepted to EACL 2020! |
| [NEW!] | I am serving as Area Chair for NAACL 2021. |
| [NEW!] | I am serving as Senior Program Committee (SPC) for IJCAI 2021. |
| [NEW!] | Three papers were accepted to EMNLP 2020 (two conference papers and one Findings paper)! |
| [NEW!] | Two papers were accepted to ECCV 2020 (the adversarial path sampling paper was seleted as Spotlight)! |
| [NEW!] | I successfully defended my Ph.D. Dissertation Closing the Loop Between Language and Vision for Embodied Agents. Thanks to the committee and everyone who has helped me along the Ph.D. journey! |
| [NEW!] | I am serving as Area Chair and Session Chair for EMNLP 2020. |
| [NEW!] | Co-organizing the workshop on Advances in Language and Vision Research (ALVR) at ACL 2020! |
| [NEW!] | Two papers were accepted to CVPR 2020 (the REVERIE paper was selected as Oral)! |
| [03/2020] | Invited panelist at the GPU Technology Conference (GTC) 2020. |
| [11/2019] | Organizer of the workshop on Language & Vision with applications to Video Understanding at CVPR 2020. |
| [11/2019] | Organizer of the tutorial on Self-Supervised Deep Learning for NLP at AACL-IJCNLP 2020. |
| [10/2019] | Invited speaker at the ICCV 2019 Workshop on Person In Context. |
| [06/2019] | Recipient of the CVPR 2019 Best Student Paper Award. |
| [06/2019] | Co-Organizer of the workshop on Closing the Loop Between Vision and Language at ICCV 2019. |
| [06/2019] | Invited talk at Facebook AI. |
| [01/2019] | Session Chair for AAAI 2019 (natural language processing). |
ERIC group members
Yue Fan, PhD studentJing Gu, PhD student
Xuehai He, PhD student
Jialu Wang, PhD student (co-advised with Yang Liu)
Kaizhi Zheng, PhD student
Kaiwen Zhou, PhD student
Tsu-Jui (Ray) Fu, affiliated UCSB PhD student
Wanrong Zhu, affiliated UCSB PhD student
Other UCSC students I work closely with
Swati Jindal, PhD studentEliana Stefani, PhD student
Minghao Liu, PhD student
Diji Yang, undergrad student
Interns
Aviral Aggarwal, IIT KanpurWilliam June Suk Choi, KAIST
Alumni
Jiayi Wei, UCSC MS student (next: Google)An Yan, UCSD MS student (next: UCSD PhD student)
Jiannan Xiang, USTC undergrad student (next: CMU MS student)
Publications
☆ (co-)advised student * equal contribution2021
|
Are Gender-Neutral Queries Really Gender-Neutral? Mitigating Gender Bias in Image Search |
|
VALUE: A Multi-Task Benchmark for Video-and-Language Understanding Evaluation |
|
Visual Question Rewriting for Increasing Response Rate |
|
Multimodal Text Style Transfer for Outdoor Vision-and-Language Navigation |
|
L2C: Describing Visual Differences Needs Semantic Understanding of Individuals |
2020
|
Closing the Loop Between Language and Vision for Embodied Agents |
|
SSCR: Iterative Language-Based Image Editing via Self-Supervised Counterfactual Reasoning |
|
Towards Understanding Sample Variance in Visually Grounded Language Generation: Evaluations and Observations |
|
Learning to Stop: A Simple yet Effective Approach to Urban Vision-Language Navigation |
|
Environment-agnostic Multitask Learning for Natural Language Grounded Navigation |
|
Counterfactual Vision-and-Language Navigation via Adversarial Path Sampling |
|
Relational Graph Learning for Grounded Video Description Generation |
|
Unsupervised Reinforcement Learning of Transferable Meta-Skills for Embodied Navigation |
|
REVERIE: Remote Embodied Visual Referring Expression in Real Indoor Environments |
|
Vision-Language Navigation Policy Learning and Adaptation
|
|
Generative Adversarial Zero-Shot Relational Learning for Knowledge Graphs |
2019
|
TIGEr: Text-to-Image Grounding for Image Caption Evaluation |
|
VATEX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Research |
|
Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigation |
|
MAN: Moment Alignment Network for Natural Language Moment Retrieval via Iterative Graph Adjustment |
|
Self-Supervised Dialogue Learning |
|
Self-Supervised Learning for Contextualized Extractive Summarization |
|
Towards Generating Long and Coherent Text with Multi-Level Latent Variable Models |
|
Extract and Edit: An Alternative to Back-Translation for Unsupervised Neural Machine Translation |
|
Learning to Compose Topic-Aware Mixture of Experts for Zero-Shot Video Captioning |
2018
|
Look Before You Leap: Bridging Model-Free and Model-Based Reinforcement Learning for Planned-Ahead Vision-and-Language Navigation |
|
XL-NBT: A Cross-lingual Neural Belief Tracking Framework |
|
No Metrics Are Perfect: Adversarial Reward Learning for Visual Storytelling |
|
S3D: Single Shot multi-Span Detector via Fully 3D Convolutional Network |
|
Video Captioning via Hierarchical Reinforcement Learning |
|
Watch, Listen, and Describe: Globally and Locally Aligned Cross-Modal Attentions for Video Captioning |
2017
|
Multimodal Transfer: A Hierarchical Deep Convolutional Neural Network for Fast Artistic Style Transfer |
|
Deep Reinforcement Learning for Visual Object Tracking in Videos |
Products
|
ProjectCloak: Remove Unwanted Objects in Video |
|
ArtisticEye: A Real-time Application for High-resolution Artistic Style Transfer |
Experience
|
Google AI, Mountain View, US Research Intern, Summer 2019 Mentors: Sujith Ravi, Zornitsa Kozareva |
|
|
Facebook AI Research (FAIR), Menlo Park, US Graduate Researcher, Spring 2019 Mentors: Xinlei Chen, Marcus Rohrbach, Dhruv Batra |
|
|
Microsoft Research AI, Redmond, US Research Intern, Summer 2018 Mentors: Lei Zhang, Asli Celikyilmaz, Jianfeng Gao |
|
|
Adobe Research, San Francisco, US Research Intern, Summer 2017 Mentors: Geoffrey Oxholm, Oliver Wang, Eli Shechtman, Mike Lukac |
|
|
Adobe Research, San Francisco, US Research Intern, Summer 2016 Mentor: Geoffrey Oxholm |
|
|
Exacloud Inc., Hangzhou, China Software Engineer Intern, 12. 2014 - 03. 2015 |
|
|
HCI, Graphics and Computer Vision Group, HKU Research Assistant, Summer 2014 Advisor: Yizhou Yu |
|
Service
- 4th Workshop on Closing the Loop Between Vision and Language (CLVL), ICCV 2021
- From VQA to VLN: Recent Advances in Vision-and-Language Research Tutorial, CVPR 2021
- 2nd Workshop on Advances in Language and Vision Research (ALVR), NAACL 2021
- Workshop on Advances in Language and Vision Research (ALVR), ACL 2020
- Workshop on Language & Vision with applications to Video Understanding, CVPR 2020
- Tutorial on Self-Supervised Deep Learning for NLP, AACL-IJCNLP 2020
- 3rd Workshop on Closing the Loop Between Vision and Language (CLVL), ICCV 2019
- AAAI 2022
- CoNLL 2021
- NLPCC 2021 (Multimodality)
- NAACL 2021 (Interpretability and Analysis of Models for NLP)
- IJCAI 2021
- EMNLP 2020 (Interpretability and Analysis of Models for NLP)
- NAACL 2021
- EMNLP 2020 (Language Grounding to Vision, Robotics and Beyond)
- AAAI 2019 (Natural Language Processing)