AV4D: Visual Learning of Sounds in Spaces

We see and hear things every second of our lives. Before sounds arrive at our ears, they are first produced by some objects situated in the space, and then undergo the transformation of its surrounding space as a function of the geometry of the environment, materials, etc. Our perceived binaural sound not only tells us about the semantic property of the sound, e.g., telephone ringing, baby crying, but also helps us infer the spatial location of the sounding object. Both of these acoustic and spatial properties are captured by the visual stream, and require models to go beyond 2D understanding of images (3D with audio) and study the spatial (3D) aspect of audio in visuals (4D with audio). This is of vital importance for applications such as egocentric video understanding, robotic perception, AR/VR, etc. In support of robotic perception, where embodied agents can move around with both visual and auditory sensing, audio-visual simulations are also recently developed to facilitate research in this direction. The goal of this workshop is to share recent progress of audio-visual studies on the spatial-temporal (4D) dimensions, and also to discuss which directions the field should investigate next.

The AV4D workshop will bring together researchers in different subareas of visual learning of sound in spaces including computer vision, robotics, machine learning, room acoustics, and graphics to examine the challenges and opportunities emerging from visual learning of sounds embodied in spaces. We will review the current state and identify the research infrastructure needed to enable a stronger collaboration between researchers working on different subareas of this workshop.

We invite submissions of 2-4 pages extended abstracts in topics related to (but not limited to):

• Visual learning of spatial audio
• Visual learning of room acoustics
• Visual learning of impact sounds
• Audio-visual self-supervised and semi-supervised learning
• Audio-visual speaker localization and diarization
• Audio-visual source separation
• Audio-visual embodied learning
• Audio-visual simulation
• Robotic perception with vision and sound
• Audio-visual for AR/VR

A submission should take the form of an extended abstract (2-4 pages long excluding references) in PDF format using the ECCV style. We will accept submissions of (1) papers that have not been previously published or accepted for publication in substantially similar form; (2) papers that have been published or accepted for publication in recent venues including journal, conference, workshop, and arXiv; and (3) research proposals for future work with a focus on well-defined concepts and ideas. All submissions will be reviewed with single blind policy. Accepted extended abstracts will not appear in ECCV proceedings, and hence will not affect future publication of the work. We will publish all accepted extended abstracts on the workshop webpage.

CMT submissions website: https://cmt3.research.microsoft.com/AV4D2022

Key Dates:

• Extended abstract submission deadline: August 15th, 2022 (11:59 pm Pacific time)
• Notification to authors: September 6th, 2022
• Camera-ready version deadline: September 15th, 2022 (11:59 pm Pacific time)
• Workshop date: October 23rd, 2022

• We'll have two paper presentation sessions: 10:15 - 11:15 and 15:00 - 16:00. Each session will be a mix of in-person and video presentations. Throughout the paper sessions, there will be short Q&A sessions for all of the papers that precede them. We'll also release recordings on our website for offline viewing.
The posters will be sized A0 (84.1 * 118.9 cm) landscape.
• For virtual speakers, please make sure you register at ECCV for the workshop and can log onto the web platform. You are requested to join the Zoom room 20 minutes prior to your session.
• For in-person speakers, prepare your presentations on Microsoft PowerPoint 16:9 ratio and submit to the technician in the hall 2 hours (30 minutes for morning sessions) prior to your session on a USB stick.
• For people who are presenting a poster, set up your poster on the poster board during breaks.
• Read the complete official speaker instructions here.