Look Before you Speak: Visually Contextualized Utterances

Google Research

Paper

Visually Contextualised Future Utterance Prediction. Given an instructional video with paired text and video data, we predict the next utterance in the video using a Co-attentional Multimodal Video Transformer. Our model trained on this task also achieves state-of-the-art performance on downstream VideoQA benchmarks.

Abstract

While most conversational AI systems focus on textual dialogue only, conditioning utterances on visual context (when it's available) can lead to more realistic conversations. Unfortunately, a major challenge for incorporating visual context into conversational dialogue is the lack of large-scale labeled datasets. We provide a solution in the form of a new visually conditioned Future Utterance Prediction task. Our task involves predicting the next utterance in a video, using both visual frames and transcribed speech as context. By exploiting the large number of instructional videos online, we train a model to solve this task at scale, without the need for manual annotations. Leveraging recent advances in multimodal learning, our model consists of a novel co-attentional multimodal video transformer, and when trained on both textual and visual context, outperforms baselines that use textual inputs alone. Further, we demonstrate that our model trained for this task on unlabelled videos achieves state-of-the-art performance on a number of downstream VideoQA benchmarks such as MSRVTT-QA, MSVD-QA, ActivityNet-QA and How2QA.

Model

Our goal is to effectively learn from both vision and text in a video. We propose a Co-attentional Multimodal Video Transformer (CoMVT), which given a video clip, extracts contextualized word embeddings and visual features from transcribed words and video frames respectively, and fuses the extracted features to form a multimodal video feature using a co-attentional transformer. We train the network to predict the true next utterance from a set of candidate utterances by minimizing the negative log-likelihood of the true next utterance. Additionally, we also train with a BERT-style masked language modelling loss, which appears to have a regularisation effect.

Qualitative Results

Multimodal video inputs

Predictions

✓

It's grown.

Now do we have a finished product?

That was my watermelon had completely ripened weeks ago.

I don't think I would drive a bit long enough to get in there.

✓

We like to eat with bone like quit bone.

So here are the carrots.

You need to see what these things are.

They're looking fabulous.

✓

So underneath that is some really really nice meat.

It is nice to write down a date on the jar.

I guess the skin anyway, so it's ready.

Don't click it.

✓

That's bad.

But so basically I just spray the folder and then I just paste this down and I try to get them in all the exact same position.

It doesn't take much time at all.

We need to mount this on our a/c condenser.

Qualitative results on HowToFUP. On the right, we show the results of the baseline model that uses text inputs only (highlighted in red) and our multimodal model (highlighted in green). The GT utterance has a ✓ next to it. Note how the transcript often contains phrases with subtle indications to visual content, such as `here's another one' (second row) and `should come off right like that' (third row). In many of these cases, the correct future utterance refers to an object which can only be known from the visual context (highlighted in bold). The text only model often selects generics utterances, or those which are referred to specifically in previous dialogue (third row, selected candidate has the word `skin').

Bibtex

@inproceedings{seo2021look, title = {{Look Before you Speak: Visually Contextualized Utterances}}, author = {Seo, Paul Hongsuck and Nagrani, Arsha and Schmid, Cordelia}, booktitle = {Computer Vision and Pattern Recognition (CVPR)}, Year = {2021} }

Contact phseo [AT] google [DOT] com for any queries.