Xin (Eric) Wang

Assistant Professor

Department of Computer Science and Engineering
Jack Baskin School of Engineering
University of California, Santa Cruz

Email: xwang366 [at] ucsc [dot] edu

Bio

Xin (Eric) Wang is an Assistant Professor of Computer Science and Engineering at UC Santa Cruz. He obtained his Ph.D. degree from UC Santa Barbara and Bachelor's degree from Zhejiang University, both in Computer Science. His research interests include Natural Language Processing, Computer Vision, and Machine Learning, with an emphasis on building embodied AI agents that can communicate with humans using natural language to perform real-world multimodal tasks. He worked at Google AI, Facebook AI Research, Microsoft Research, and Adobe Research.
Xin has served as Area Chair for NAACL 2021, CoNLL 2021, NLPCC 2021, and EMNLP 2020, Senior Program Committee (SPC) for IJCAI 2021, and Session Chair for EMNLP 2020 and AAAI 2019. He organized multiple workshops and tutorials at ACL, CVPR, ICCV, AACL, etc. He received the CVPR Best Student Paper Award in 2019.

Research Interests

Xin's research interests include Natural Language Processing, Computer Vision, and Machine Learning, with an emphasis on building multimodal embodied agents that can communicate with humans using natural language to perform real-world tasks.

Natural Language Processing (natural language grounding, knowledge-based reasoning, fairness and bias)
Computer Vision (vision and language, visual navigation and robotics, activity understanding in videos)
Machine Learning (multimodal machine learning, self-supervised learning, reinforcement learning, deep learning)

Hiring

I am looking for highly-motivated students. Please read the information for prospective students and visitors and check out the most beautiful and unique campus of UCSC [YouTube video | bilibili video].

Teaching

Winter 2021	CSE 142: Machine Learning
Spring 2021	CSE 290C: Multimodal Deep Learning
Fall 2021	CSE 142: Machine Learning

News

[NEW!]	I am giving a Tutorial on "From VQA to VLN: Recent Advances in Vision-and-Language Research" at CVPR 2021 on June 20th!
[NEW!]	Co-organizing the Second Workshop on Advances in Language and Vision Research (ALVR) at NAACL 2021 on June 11th!
[NEW!]	I am giving a keynote talk at the Third Workshop on Multimodal Artificial Intelligence at NAACL 2021 on June 6th!
[NEW!]	Our paper on Visual Question Rewriting was accepted to SIGIR 2021!
[NEW!]	I am serving as Area Chair for CoNLL 2021 and NLPCC 2021.
[NEW!]	Co-organizing the 4th Workshop on Closing the Loop Between Vision and Language (CLVL) at ICCV 2021!
[NEW!]	Invited talk at Arizona State University.
[NEW!]	Two papers on multimodal style transfer learning for VLN and visual comparison were accepted to EACL 2020!
[NEW!]	I am serving as Area Chair for NAACL 2021.
[NEW!]	I am serving as Senior Program Committee (SPC) for IJCAI 2021.
[NEW!]	Three papers were accepted to EMNLP 2020 (two conference papers and one Findings paper)!
[NEW!]	Two papers were accepted to ECCV 2020 (the adversarial path sampling paper was seleted as Spotlight)!
[NEW!]	I successfully defended my Ph.D. Dissertation Closing the Loop Between Language and Vision for Embodied Agents. Thanks to the committee and everyone who has helped me along the Ph.D. journey!
[NEW!]	I am serving as Area Chair and Session Chair for EMNLP 2020.
[NEW!]	Co-organizing the workshop on Advances in Language and Vision Research (ALVR) at ACL 2020!
[NEW!]	Two papers were accepted to CVPR 2020 (the REVERIE paper was selected as Oral)!
[03/2020]	Invited panelist at the GPU Technology Conference (GTC) 2020.
[11/2019]	Organizer of the workshop on Language & Vision with applications to Video Understanding at CVPR 2020.
[11/2019]	Organizer of the tutorial on Self-Supervised Deep Learning for NLP at AACL-IJCNLP 2020.
[10/2019]	Invited speaker at the ICCV 2019 Workshop on Person In Context.
[06/2019]	Recipient of the CVPR 2019 Best Student Paper Award.
[06/2019]	Co-Organizer of the workshop on Closing the Loop Between Vision and Language at ICCV 2019.
[06/2019]	Invited talk at Facebook AI.
[01/2019]	Session Chair for AAAI 2019 (natural language processing).

ERIC group members

Yue Fan, PhD student (JHU MS)
Jing Gu, PhD student (UCD MS)
Xuehai He, PhD student (UCSD MS)
Jialu Wang, PhD student (SJTU BS, co-advised with Yang Liu)
Kaizhi Zheng, PhD student (U Michigan MS)
Kaiwen Zhou, PhD student (ZJU BS)
Tsu-Jui (Ray) Fu, affiliated UCSB PhD student
Wanrong Zhu, affiliated UCSB PhD student

Other UCSC students I work closely with

Swati Jindal, PhD student
Eliana Stefani, PhD student
Minghao Liu, PhD student
Diji Yang, undergrad student

Interns

Aviral Aggarwal, IIT Kanpur
William June Suk Choi, KAIST

Alumni

Jiayi Wei, UCSC MS student (next: Google)
An Yan, UCSD MS student (next: UCSD PhD student)
Jiannan Xiang, USTC undergrad student (next: CMU MS student)

Publications

☆ (co-)advised student * equal contribution

2021

Language-based Video Editing via Multi-Modal Multi-Level Transformer
Tsu-Jui Fu☆, Xin Eric Wang, Scott Grafton, Miguel Eckstein, William Yang Wang
arXiv 2021
[Paper]

Diagnosing Vision-and-Language Navigation: What Really Matters
Wanrong Zhu☆, Yuankai Qi, Pradyumna Narayana, Kazoo Sone, Sugato Basu, Xin Eric Wang, Qi Wu, Miguel Eckstein, William Yang Wang
arXiv 2021
[Paper]

Visual Question Rewriting for Increasing Response Rate
Jiayi Wei☆, Xilian Li☆, Yi Zhang, Xin Eric Wang
SIGIR 2021

Multimodal Text Style Transfer for Outdoor Vision-and-Language Navigation
Wanrong Zhu☆, Xin Eric Wang, Tsu-Jui Fu, An Yan, Pradyumna Narayana, Kazoo Sone, Sugato Basu, William Yang Wang
EACL 2021
[Paper] [Code]

L2C: Describing Visual Differences Needs Semantic Understanding of Individuals
An Yan☆, Xin Eric Wang, Tsu-Jui Fu, William Yang Wang
EACL 2021
[Paper]

2020

Closing the Loop Between Language and Vision for Embodied Agents
Xin Wang
UC Santa Barbara
[PhD Dissertation]

SSCR: Iterative Language-Based Image Editing via Self-Supervised Counterfactual Reasoning
Tsu-Jui Fu☆, Xin Eric Wang, Scott Grafton, Miguel Eckstein, William Yang Wang
EMNLP 2020
[Paper] [Code] [Slides]

Towards Understanding Sample Variance in Visually Grounded Language Generation: Evaluations and Observations
Wanrong Zhu☆, Xin Eric Wang, Pradyumna Narayana, Kazoo Sone, Sugato Basu, William Yang Wang
EMNLP 2020
[Paper]

Learning to Stop: A Simple yet Effective Approach to Urban Vision-Language Navigation
Jiannan Xiang☆, Xin Eric Wang, William Yang Wang
Findings of EMNLP 2020
[Paper]

Environment-agnostic Multitask Learning for Natural Language Grounded Navigation
Xin Eric Wang*, Vihan Jain*, Eugene Ie, William Yang Wang, Zornitsa Kozareva, Sujith Ravi
ECCV 2020
Ranking 1st on the CVDN leaderboard
[Paper] [Code] [Video] [Slides]

Counterfactual Vision-and-Language Navigation via Adversarial Path Sampling
Tsu-Jui Fu☆, Xin Eric Wang, Matthew Peterson, Scott Grafton, Miguel Eckstein, William Yang Wang
ECCV 2020
Spotlight presentation
[Paper] [Video] [Slides]

Relational Graph Learning for Grounded Video Description Generation
Wenqiao Zhang☆, Xin Eric Wang, Siliang Tang, Haizhou Shi, Haocheng Shi, Jun Xiao, Yueting Zhuang, William Yang Wang
ACM MM 2020
Oral Presentation
[Paper] [Video]

Unsupervised Reinforcement Learning of Transferable Meta-Skills for Embodied Navigation
Juncheng Li☆, Xin Wang, Siliang Tang, Haizhou Shi, Fei Wu, Yueting Zhuang, William Yang Wang
CVPR 2020
[Paper] [Video]

REVERIE: Remote Embodied Visual Referring Expression in Real Indoor Environments
Yuankai Qi, Qi Wu, Peter Anderson, Xin Wang, William Yang Wang, Chunhua Shen, Anton van den Hengel
CVPR 2020
Oral presentation
[Paper] [Code] [Video]

Vision-Language Navigation Policy Learning and Adaptation
Xin Wang, Qiuyuan Huang, Asli Celikyilmaz, Jianfeng Gao, Dinghan Shen, Yuan-Fang Wang, William Yang Wang, Lei Zhang
IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 2020
Journal version of the CVPR 2019 Best Student Paper

Generative Adversarial Zero-Shot Relational Learning for Knowledge Graphs
Pengda Qin, Xin Wang, Wenhu Chen, Chunyun Zhang, Weiran Xu, William Yang Wang
AAAI 2020
Oral presentation
[Paper] [Code]

2019

TIGEr: Text-to-Image Grounding for Image Caption Evaluation
Ming Jiang, Qiuyuan Huang, Lei Zhang, Xin Wang, Pengchuan Zhang, Zhe Gan, Jana Diesner, Jianfeng Gao
EMNLP-IJCNLP 2019
[Paper] [Code] [bibtex]

VATEX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Research
Xin Wang*, Jiawei Wu*, Junkun Chen, Lei Li, Yuan-Fang Wang, William Yang Wang
ICCV 2019
Oral presentation
[Paper] [Website] [Video] [bibtex]

Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigation
Xin Wang, Qiuyuan Huang, Asli Celikyilmaz, Jianfeng Gao, Dinghan Shen, Yuan-Fang Wang, William Yang Wang, Lei Zhang
CVPR 2019
Best Student Paper (1/5160=0.02%)
[Paper] [Video] [bibtex]

MAN: Moment Alignment Network for Natural Language Moment Retrieval via Iterative Graph Adjustment
Da Zhang, Xiyang Dai, Xin Wang, Yuan-Fang Wang, Larry S. Davis
CVPR 2019
[Paper] [bibtex]

Self-Supervised Dialogue Learning
Jiawei Wu, Xin Wang, William Yang Wang
ACL 2019
[Paper] [bibtex]

Self-Supervised Learning for Contextualized Extractive Summarization
Hong Wang, Xin Wang, Wenhan Xiong, Mo Yu, Xiaoxiao Guo, Shiyu Chang, William Yang Wang
ACL 2019
[Paper] [Code] [bibtex]

Towards Generating Long and Coherent Text with Multi-Level Latent Variable Models
Dinghan Shen, Asli Celikyilmaz, Yizhe Zhang, Liqun Chen, Xin Wang, Jianfeng Gao, Lawrence Carin
ACL 2019
[Paper] [bibtex]

Extract and Edit: An Alternative to Back-Translation for Unsupervised Neural Machine Translation
Jiawei Wu, Xin Wang, William Yang Wang
NAACL 2019
Oral presentation
[Paper] [bibtex]

Learning to Compose Topic-Aware Mixture of Experts for Zero-Shot Video Captioning
Xin Wang, Jiawei Wu, Da Zhang, Yu Su, William Yang Wang
AAAI 2019
Oral presentation
[Paper] [Code] [bibtex]

2018

Look Before You Leap: Bridging Model-Free and Model-Based Reinforcement Learning for Planned-Ahead Vision-and-Language Navigation
Xin Wang*, Wenhan Xiong*, Hongmin Wang, William Yang Wang
ECCV 2018
[Paper] [bibtex]

XL-NBT: A Cross-lingual Neural Belief Tracking Framework
Wenhu Chen, Jianshu Chen, Yu Su, Xin Wang, Dong Yu, Xifeng Yan, William Yang Wang
EMNLP 2018
[Paper] [Code] [bibtex]

No Metrics Are Perfect: Adversarial Reward Learning for Visual Storytelling
Xin Wang*, Wenhu Chen*, Yuan-Fang Wang, William Yang Wang
ACL 2018
Oral presentation
[Paper] [Code] [Video] [Slides (pptx)] [Slides (pdf)] [bibtex]

S3D: Single Shot multi-Span Detector via Fully 3D Convolutional Network
Da Zhang, Xiyang Dai, Xin Wang, Yuan-Fang Wang
BMVC 2018
Oral presentation
[Paper] [Code] [Video] [Slides] [bibtex]

Video Captioning via Hierarchical Reinforcement Learning
Xin Wang, Wenhu Chen, Jiawei Wu, Yuan-Fang Wang, William Yang Wang
CVPR 2018
[Paper] [Supp] [Dataset] [bibtex]

Watch, Listen, and Describe: Globally and Locally Aligned Cross-Modal Attentions for Video Captioning
Xin Wang, Yuan-Fang Wang, William Yang Wang
NAACL-HLT 2018
[Paper] [Paper] [bibtex]

2017

Multimodal Transfer: A Hierarchical Deep Convolutional Neural Network for Fast Artistic Style Transfer
Xin Wang, Geoffrey Oxholm, Da Zhang, Yuan-Fang Wang
CVPR 2017
[Paper] [Supp] [Images] [Code (Third-Party)] [bibtex]

Deep Reinforcement Learning for Visual Object Tracking in Videos
Da Zhang, Hamid Maei, Xin Wang, Yuan-Fang Wang
Tech report 2017
[Paper] [bibtex]

Products

ProjectCloak: Remove Unwanted Objects in Video
In collabration with Geoffrey Oxholm, Oliver Wang, Eli Shechtman, Mike Lukac
2017 MAX Sneak
2018 MAX Keynote
Featured in Adobe After Effects! Link

[Project] [Blog] [Video]

ArtisticEye: A Real-time Application for High-resolution Artistic Style Transfer
In collabration with Geoffrey Oxholm
Applied to the de Young Museum in San Francisco
Honored to present the product prototype to the Adobe CEO Shantanu Narayen face to face

[Blog] [Video]

Experience

Google AI, Mountain View, US Research Intern, Summer 2019 Mentors: Sujith Ravi, Zornitsa Kozareva
Facebook AI Research (FAIR), Menlo Park, US Graduate Researcher, Spring 2019 Mentors: Xinlei Chen, Marcus Rohrbach, Dhruv Batra
Microsoft Research AI, Redmond, US Research Intern, Summer 2018 Mentors: Lei Zhang, Asli Celikyilmaz, Jianfeng Gao
Adobe Research, San Francisco, US Research Intern, Summer 2017 Mentors: Geoffrey Oxholm, Oliver Wang, Eli Shechtman, Mike Lukac
Adobe Research, San Francisco, US Research Intern, Summer 2016 Mentor: Geoffrey Oxholm
Exacloud Inc., Hangzhou, China Software Engineer Intern, 12. 2014 - 03. 2015
HCI, Graphics and Computer Vision Group, HKU Research Assistant, Summer 2014 Advisor: Yizhou Yu