Home Experience Publications Products Teaching Service CV

Xin (Eric) Wang

Incoming Assistant Professor

Department of Computer Science and Engineering
Jack Baskin School of Engineering
University of California, Santa Cruz

Email: xwang366 [at] ucsc [dot] edu


I am joining UC Santa Cruz as an Assistant Professor in the Department of Computer Science and Engineering this summer. I obtained my Ph.D. degree in computer science at UC Santa Barbara and B.Eng. degree in computer science and technology at Zhejiang University. My research interests include natural language processing, computer vision, and machine learning, with an emphasis on building multimodal embodied agents that can communicate with humans using natural language to perform real-world tasks. I worked at Google AI, Facebook AI Research, Microsoft Research, and Adobe Research. I received the CVPR Best Student Paper Award in 2019.

I am looking for highly-motivated students, so feel free to drop me an email if you are interested in working with me on natural language processing, computer vision, robotics, and machine learning (due to the high volumn I may not be able to reply to every email).


[NEW!] I successfully defended my Ph.D. Thesis "Closing the Loop Between Language and Vision for Embodied Agents". Thanks to the committee and everyone who has helped me along the Ph.D. journey!
[NEW!] I am serving as an Area Chair for EMNLP 2020.
[NEW!] Organizing the workshop on Advances in Language and Vision Research (ALVR) at ACL 2020. Welcome to participate on July 9th!
[03/2020] Invited panelist at the GPU Technology Conference (GTC) 2020.
[11/2019] Organizer of the workshop on Language & Vision with applications to Video Understanding at CVPR 2020.
[11/2019] Organizer of the tutorial on Self-Supervised Deep Learning for NLP at AACL-IJCNLP 2020.
[10/2019] Invited speaker at the ICCV 2019 Workshop on Person In Context.
[06/2019] Recipient of the CVPR 2019 Best Student Paper Award.
[06/2019] Co-Organizer of the workshop on Closing the Loop Between Vision and Language at ICCV 2019.
[06/2019] Invited talk at Facebook AI.
[01/2019] Session Chair for AAAI 2019 (natural language processing).


Google AI, Mountain View, US

       Research Intern,   Summer 2019

       Mentors: Sujith Ravi, Zornitsa Kozareva

Facebook AI Research (FAIR), Menlo Park, US

       Graduate Researcher,   Spring 2019

       Mentors: Xinlei Chen, Marcus Rohrbach, Dhruv Batra

Microsoft Research AI, Redmond, US

       Research Intern,   Summer 2018

       Mentors: Lei Zhang, Asli Celikyilmaz, Jianfeng Gao

Adobe Research, San Francisco, US

       Research Intern,   Summer 2017

       Mentors: Geoffrey Oxholm, Oliver Wang, Eli Shechtman, Mike Lukac

Adobe Research, San Francisco, US

       Research Intern,   Summer 2016

       Mentor: Geoffrey Oxholm

Exacloud Inc., Hangzhou, China

       Software Engineer Intern,   12. 2014 - 03. 2015

HCI, Graphics and Computer Vision Group, HKU

       Research Assistant, Summer 2014

       Advisor: Yizhou Yu

Selected Publications


Multimodal Text Style Transfer for Outdoor Vision-and-Language Navigation
Wanrong Zhu, Xin Wang, Tsu-Jui Fu, An Yan, Pradyumna Narayana, Kazoo Sone, Sugato Basu, William Yang Wang
Tech report 2020

Cross-Lingual Vision-Language Navigation
An Yan*, Xin Wang*, Jiangtao Feng, Lei Li, William Yang Wang
Tech report 2020


Environment-agnostic Multitask Learning for Natural Language Grounded Navigation
Xin Wang*, Vihan Jain*, Eugene Ie, William Wang, Zornitsa Kozareva, Sujith Ravi
ECCV 2020
Ranking 1st on the CVDN leaderboard
[Paper] [Code]

Counterfactual Vision-and-Language Navigation via Adversarial Path Sampling
Tsu-Jui Fu, Xin Wang, Matthew Peterson, Scott Grafton, Miguel Eckstein, William Yang Wang
ECCV 2020
Spotlight presentation

REVERIE: Remote Embodied Visual Referring Expression in Real Indoor Environments
Yuankai Qi, Qi Wu, Peter Anderson, Xin Wang, William Yang Wang, Chunhua Shen, Anton van den Hengel
CVPR 2020
Oral presentation
[Paper] [Code]

Unsupervised Reinforcement Learning of Transferable Meta-Skills for Embodied Navigation
Juncheng Li, Xin Wang, Siliang Tang, Haizhou Shi, Fei Wu, Yueting Zhuang, William Yang Wang
CVPR 2020

Vision-Language Navigation Policy Learning and Adaptation
Xin Wang, Qiuyuan Huang, Asli Celikyilmaz, Jianfeng Gao, Dinghan Shen, Yuan-Fang Wang, William Yang Wang, Lei Zhang
IEEE transactions on pattern analysis and machine intelligence (TPAMI), 2020
Journal version of the CVPR 2019 Best Student Paper

Generative Adversarial Zero-Shot Relational Learning for Knowledge Graphs
Pengda Qin, Xin Wang, Wenhu Chen, Chunyun Zhang, Weiran Xu, William Yang Wang
AAAI 2020
Oral presentation


TIGEr: Text-to-Image Grounding for Image Caption Evaluation
Ming Jiang, Qiuyuan Huang, Lei Zhang, Xin Wang, Pengchuan Zhang, Zhe Gan, Jana Diesner, Jianfeng Gao
[Paper] [Code] [bibtex]

VATEX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Research
Xin Wang*, Jiawei Wu*, Junkun Chen, Lei Li, Yuan-Fang Wang, William Yang Wang
ICCV 2019
Oral presentation
[Paper] [Website] [bibtex]

Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigation
Xin Wang, Qiuyuan Huang, Asli Celikyilmaz, Jianfeng Gao, Dinghan Shen, Yuan-Fang Wang, William Yang Wang, Lei Zhang
CVPR 2019
Best Student Paper (1/5160=0.02%)
[Paper] [bibtex]

MAN: Moment Alignment Network for Natural Language Moment Retrieval via Iterative Graph Adjustment
Da Zhang, Xiyang Dai, Xin Wang, Yuan-Fang Wang, Larry S. Davis
CVPR 2019
[Paper] [bibtex]

Self-Supervised Dialogue Learning
Jiawei Wu, Xin Wang, William Yang Wang
ACL 2019
[Paper] [bibtex]

Self-Supervised Learning for Contextualized Extractive Summarization
Hong Wang, Xin Wang, Wenhan Xiong, Mo Yu, Xiaoxiao Guo, Shiyu Chang, William Yang Wang
ACL 2019
[Paper] [Code] [bibtex]

Towards Generating Long and Coherent Text with Multi-Level Latent Variable Models
Dinghan Shen, Asli Celikyilmaz, Yizhe Zhang, Liqun Chen, Xin Wang, Jianfeng Gao, Lawrence Carin
ACL 2019
[Paper] [bibtex]

Extract and Edit: An Alternative to Back-Translation for Unsupervised Neural Machine Translation
Jiawei Wu, Xin Wang, William Yang Wang
NAACL 2019
Oral presentation
[Paper] [bibtex]

Learning to Compose Topic-Aware Mixture of Experts for Zero-Shot Video Captioning
Xin Wang, Jiawei Wu, Da Zhang, Yu Su, William Yang Wang
AAAI 2019
Oral presentation
[Paper] [Code] [bibtex]


Look Before You Leap: Bridging Model-Free and Model-Based Reinforcement Learning for Planned-Ahead Vision-and-Language Navigation
Xin Wang*, Wenhan Xiong*, Hongmin Wang, William Yang Wang
ECCV 2018
[Paper] [bibtex]

XL-NBT: A Cross-lingual Neural Belief Tracking Framework
Wenhu Chen, Jianshu Chen, Yu Su, Xin Wang, Dong Yu, Xifeng Yan, William Yang Wang
EMNLP 2018
[Paper] [Code] [bibtex]

No Metrics Are Perfect: Adversarial Reward Learning for Visual Storytelling
Xin Wang*, Wenhu Chen*, Yuan-Fang Wang, William Yang Wang
ACL 2018
Oral presentation
[Paper] [Code] [Video] [Slides (pptx)] [Slides (pdf)] [bibtex]

S3D: Single Shot multi-Span Detector via Fully 3D Convolutional Network
Da Zhang, Xiyang Dai, Xin Wang, Yuan-Fang Wang
BMVC 2018
Oral presentation
[Paper] [Code] [Video] [Slides] [bibtex]

Video Captioning via Hierarchical Reinforcement Learning
Xin Wang, Wenhu Chen, Jiawei Wu, Yuan-Fang Wang, William Yang Wang
CVPR 2018
[Paper] [Supp] [Dataset] [bibtex]

Watch, Listen, and Describe: Globally and Locally Aligned Cross-Modal Attentions for Video Captioning
Xin Wang, Yuan-Fang Wang, William Yang Wang
[Paper] [Paper] [bibtex]


Multimodal Transfer: A Hierarchical Deep Convolutional Neural Network for Fast Artistic Style Transfer
Xin Wang, Geoffrey Oxholm, Da Zhang, Yuan-Fang Wang
CVPR 2017
[Paper] [Supp] [Images] [Code (Third-Party)] [bibtex]

Deep Reinforcement Learning for Visual Object Tracking in Videos
Da Zhang, Hamid Maei, Xin Wang, Yuan-Fang Wang
Tech report 2017
[Paper] [bibtex]


ProjectCloak: Remove Unwanted Objects in Video
In collabration with Geoffrey Oxholm, Oliver Wang, Eli Shechtman, Mike Lukac
2017 MAX Sneak
2018 MAX Keynote
Featured in Adobe After Effects! Link

[Project] [Blog] [Video]

ArtisticEye: A Real-time Application for High-resolution Artistic Style Transfer
In collabration with Geoffrey Oxholm
Applied to the de Young Museum in San Francisco
Honored to present the product prototype to the Adobe CEO Shantanu Narayen face to face

[Blog] [Video]


2018 Winter Guest Lecturer CS281b: Computer Vision and Image Analysis
2017 Winter Teaching Assistant CS190I: Introduction to Natual Language Processing
2016 Fall Teaching Assistant CS180: Computer Graphics
2016 Winter Teaching Assistant CS48: Computer Science Project
2015 Fall Teaching Assistant CS16: Problem Solving with C++


  • Organizer:
  • Area Chair:
    • EMNLP 2020  (Interpretability and Analysis of Models for NLP)
  • Session Chair:
  • Conference Program Committee or Reviewer: NeurIPS 2020,   ACL 2020,   ECCV 2020,   CVPR 2020,   AAAI 2020,   CoRL 2019,   NeurIPS 2019,   EMNLP-IJCNLP 2019,   ICCV 2019,   CVPR 2019,   ACL 2019,   NAACL 2019,   SGRL 2019,   CoNLL 2019,   SpLU-RoboNLP 2019,   AAAI 2019,   EMNLP 2018,   SoCal NLP 2018

  • Journal Reviewer: TPAMI,  IJCV