Xin (Eric) Wang

Assistant Professor

Department of Computer Science and Engineering
Jack Baskin School of Engineering
University of California, Santa Cruz

Email: xwang366 [at] ucsc [dot] edu



Bio

Xin (Eric) Wang is an Assistant Professor of Computer Science and Engineering at UC Santa Cruz. He obtained his Ph.D. at UC Santa Barbara and B.Eng. at Zhejiang University, both in Computer Science. He worked at Google AI, Facebook AI Research, Microsoft Research, and Adobe Research and has built wide collaboration with many industrial research labs. He received the CVPR Best Student Paper Award in 2019.

Research Interests

Xin's research interests include Computer Vision, Natural Language Processing, and Machine Learning, with an emphasis on building embodied AI agents that can communicate with humans using natural language to perform real-world tasks.

Hiring

I am looking for highly-motivated students (esp. PhD students starting in Fall 2021). Please read the information for prospective students and visitors and check out the most beautiful and unique campus of UCSC [YouTube video | bilibili video].

News

[NEW!] Three papers were accepted to EMNLP 2020 (two conference papers and one Findings paper)!
[NEW!] Two papers were accepted to ECCV 2020 (the adversarial path sampling paper was seleted as Spotlight)!
[NEW!] I successfully defended my Ph.D. Thesis "Closing the Loop Between Language and Vision for Embodied Agents". Thanks to the committee and everyone who has helped me along the Ph.D. journey!
[NEW!] I am serving as Area Chair for EMNLP 2020.
[NEW!] Organizing the workshop on Advances in Language and Vision Research (ALVR) at ACL 2020. Welcome to participate on July 9th!
[NEW!] Two papers were accepted to CVPR 2020 (the REVERIE paper was selected as Oral)!
[03/2020] Invited panelist at the GPU Technology Conference (GTC) 2020.
[11/2019] Organizer of the workshop on Language & Vision with applications to Video Understanding at CVPR 2020.
[11/2019] Organizer of the tutorial on Self-Supervised Deep Learning for NLP at AACL-IJCNLP 2020.
[10/2019] Invited speaker at the ICCV 2019 Workshop on Person In Context.
[06/2019] Recipient of the CVPR 2019 Best Student Paper Award.
[06/2019] Co-Organizer of the workshop on Closing the Loop Between Vision and Language at ICCV 2019.
[06/2019] Invited talk at Facebook AI.
[01/2019] Session Chair for AAAI 2019 (natural language processing).


Selected Publications

I have been publishing under Xin Eric Wang since 2020 (previously Xin Wang).

2020

Iterative Language-Based Image Editing via Self-Supervised Counterfactual Reasoning
Tsu-Jui Fu, Xin Eric Wang, Scott Grafton, Miguel Eckstein, William Yang Wang
EMNLP 2020

Towards Understanding Sample Variance in Visually Grounded Language Generation: Evaluations and Observations
Wanrong Zhu, Xin Eric Wang, William Yang Wang
EMNLP 2020

Learning to Stop: A Simple yet Effective Approach to Urban Vision-Language Navigation
Jiannan Xiang, Xin Eric Wang, William Yang Wang
Findings of EMNLP 2020

Environment-agnostic Multitask Learning for Natural Language Grounded Navigation
Xin Eric Wang*, Vihan Jain*, Eugene Ie, William Yang Wang, Zornitsa Kozareva, Sujith Ravi
ECCV 2020
Ranking 1st on the CVDN leaderboard
[Paper] [Code] [Video] [Slides]

Counterfactual Vision-and-Language Navigation via Adversarial Path Sampling
Tsu-Jui Fu, Xin Eric Wang, Matthew Peterson, Scott Grafton, Miguel Eckstein, William Yang Wang
ECCV 2020
Spotlight presentation
[Paper] [Video] [Slides]

Relational Graph Learning for Grounded Video Description Generation
Wenqiao Zhang, Xin Eric Wang, Siliang Tang, Haizhou Shi, Haocheng Shi, Jun Xiao, Yueting Zhuang, William Yang Wang
ACM MM 2020
Oral Presentation

REVERIE: Remote Embodied Visual Referring Expression in Real Indoor Environments
Yuankai Qi, Qi Wu, Peter Anderson, Xin Wang, William Yang Wang, Chunhua Shen, Anton van den Hengel
CVPR 2020
Oral presentation
[Paper] [Code] [Video]

Unsupervised Reinforcement Learning of Transferable Meta-Skills for Embodied Navigation
Juncheng Li, Xin Wang, Siliang Tang, Haizhou Shi, Fei Wu, Yueting Zhuang, William Yang Wang
CVPR 2020
[Paper] [Video]

Vision-Language Navigation Policy Learning and Adaptation
Xin Wang, Qiuyuan Huang, Asli Celikyilmaz, Jianfeng Gao, Dinghan Shen, Yuan-Fang Wang, William Yang Wang, Lei Zhang
IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 2020
Journal version of the CVPR 2019 Best Student Paper

Generative Adversarial Zero-Shot Relational Learning for Knowledge Graphs
Pengda Qin, Xin Wang, Wenhu Chen, Chunyun Zhang, Weiran Xu, William Yang Wang
AAAI 2020
Oral presentation
[Paper] [Code]

2019

TIGEr: Text-to-Image Grounding for Image Caption Evaluation
Ming Jiang, Qiuyuan Huang, Lei Zhang, Xin Wang, Pengchuan Zhang, Zhe Gan, Jana Diesner, Jianfeng Gao
EMNLP-IJCNLP 2019
[Paper] [Code] [bibtex]

VATEX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Research
Xin Wang*, Jiawei Wu*, Junkun Chen, Lei Li, Yuan-Fang Wang, William Yang Wang
ICCV 2019
Oral presentation
[Paper] [Website] [Video] [bibtex]

Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigation
Xin Wang, Qiuyuan Huang, Asli Celikyilmaz, Jianfeng Gao, Dinghan Shen, Yuan-Fang Wang, William Yang Wang, Lei Zhang
CVPR 2019
Best Student Paper (1/5160=0.02%)
[Paper] [Video] [bibtex]

MAN: Moment Alignment Network for Natural Language Moment Retrieval via Iterative Graph Adjustment
Da Zhang, Xiyang Dai, Xin Wang, Yuan-Fang Wang, Larry S. Davis
CVPR 2019
[Paper] [bibtex]

Self-Supervised Dialogue Learning
Jiawei Wu, Xin Wang, William Yang Wang
ACL 2019
[Paper] [bibtex]

Self-Supervised Learning for Contextualized Extractive Summarization
Hong Wang, Xin Wang, Wenhan Xiong, Mo Yu, Xiaoxiao Guo, Shiyu Chang, William Yang Wang
ACL 2019
[Paper] [Code] [bibtex]

Towards Generating Long and Coherent Text with Multi-Level Latent Variable Models
Dinghan Shen, Asli Celikyilmaz, Yizhe Zhang, Liqun Chen, Xin Wang, Jianfeng Gao, Lawrence Carin
ACL 2019
[Paper] [bibtex]

Extract and Edit: An Alternative to Back-Translation for Unsupervised Neural Machine Translation
Jiawei Wu, Xin Wang, William Yang Wang
NAACL 2019
Oral presentation
[Paper] [bibtex]

Learning to Compose Topic-Aware Mixture of Experts for Zero-Shot Video Captioning
Xin Wang, Jiawei Wu, Da Zhang, Yu Su, William Yang Wang
AAAI 2019
Oral presentation
[Paper] [Code] [bibtex]

2018

Look Before You Leap: Bridging Model-Free and Model-Based Reinforcement Learning for Planned-Ahead Vision-and-Language Navigation
Xin Wang*, Wenhan Xiong*, Hongmin Wang, William Yang Wang
ECCV 2018
[Paper] [bibtex]

XL-NBT: A Cross-lingual Neural Belief Tracking Framework
Wenhu Chen, Jianshu Chen, Yu Su, Xin Wang, Dong Yu, Xifeng Yan, William Yang Wang
EMNLP 2018
[Paper] [Code] [bibtex]

No Metrics Are Perfect: Adversarial Reward Learning for Visual Storytelling
Xin Wang*, Wenhu Chen*, Yuan-Fang Wang, William Yang Wang
ACL 2018
Oral presentation
[Paper] [Code] [Video] [Slides (pptx)] [Slides (pdf)] [bibtex]

S3D: Single Shot multi-Span Detector via Fully 3D Convolutional Network
Da Zhang, Xiyang Dai, Xin Wang, Yuan-Fang Wang
BMVC 2018
Oral presentation
[Paper] [Code] [Video] [Slides] [bibtex]

Video Captioning via Hierarchical Reinforcement Learning
Xin Wang, Wenhu Chen, Jiawei Wu, Yuan-Fang Wang, William Yang Wang
CVPR 2018
[Paper] [Supp] [Dataset] [bibtex]

Watch, Listen, and Describe: Globally and Locally Aligned Cross-Modal Attentions for Video Captioning
Xin Wang, Yuan-Fang Wang, William Yang Wang
NAACL-HLT 2018
[Paper] [Paper] [bibtex]

2017

Multimodal Transfer: A Hierarchical Deep Convolutional Neural Network for Fast Artistic Style Transfer
Xin Wang, Geoffrey Oxholm, Da Zhang, Yuan-Fang Wang
CVPR 2017
[Paper] [Supp] [Images] [Code (Third-Party)] [bibtex]

Deep Reinforcement Learning for Visual Object Tracking in Videos
Da Zhang, Hamid Maei, Xin Wang, Yuan-Fang Wang
Tech report 2017
[Paper] [bibtex]



Products

ProjectCloak: Remove Unwanted Objects in Video
In collabration with Geoffrey Oxholm, Oliver Wang, Eli Shechtman, Mike Lukac
2017 MAX Sneak
2018 MAX Keynote
Featured in Adobe After Effects! Link

[Project] [Blog] [Video]

ArtisticEye: A Real-time Application for High-resolution Artistic Style Transfer
In collabration with Geoffrey Oxholm
Applied to the de Young Museum in San Francisco
Honored to present the product prototype to the Adobe CEO Shantanu Narayen face to face

[Blog] [Video]



Experience

Google AI, Mountain View, US

       Research Intern,   Summer 2019

       Mentors: Sujith Ravi, Zornitsa Kozareva

Facebook AI Research (FAIR), Menlo Park, US

       Graduate Researcher,   Spring 2019

       Mentors: Xinlei Chen, Marcus Rohrbach, Dhruv Batra

Microsoft Research AI, Redmond, US

       Research Intern,   Summer 2018

       Mentors: Lei Zhang, Asli Celikyilmaz, Jianfeng Gao

Adobe Research, San Francisco, US

       Research Intern,   Summer 2017

       Mentors: Geoffrey Oxholm, Oliver Wang, Eli Shechtman, Mike Lukac

Adobe Research, San Francisco, US

       Research Intern,   Summer 2016

       Mentor: Geoffrey Oxholm

Exacloud Inc., Hangzhou, China

       Software Engineer Intern,   12. 2014 - 03. 2015

HCI, Graphics and Computer Vision Group, HKU

       Research Assistant, Summer 2014

       Advisor: Yizhou Yu



Teaching

2018 Winter Guest Lecturer CS281b: Computer Vision and Image Analysis
2017 Winter Teaching Assistant CS190I: Introduction to Natual Language Processing
2016 Fall Teaching Assistant CS180: Computer Graphics
2016 Winter Teaching Assistant CS48: Computer Science Project
2015 Fall Teaching Assistant CS16: Problem Solving with C++


Service

  • Organizer:
  • Area Chair:
    • EMNLP 2020  (Interpretability and Analysis of Models for NLP)
  • Session Chair:
  • Conference Program Committee or Reviewer: NeurIPS 2020,   ACL 2020,   ECCV 2020,   CVPR 2020,   AAAI 2020,   CoRL 2019,   NeurIPS 2019,   EMNLP-IJCNLP 2019,   ICCV 2019,   CVPR 2019,   ACL 2019,   NAACL 2019,   SGRL 2019,   CoNLL 2019,   SpLU-RoboNLP 2019,   AAAI 2019,   EMNLP 2018,   SoCal NLP 2018

  • Journal Reviewer: TPAMI,  IJCV