AV4D: Visual Learning of Sounds in Spaces

We see and hear things every second of our lives. Before sounds arrive at our ears, they are first produced by some objects situated in the space, and then undergo the transformation of its surrounding space as a function of the geometry of the environment, materials, etc. Our perceived binaural sound not only tells us about the semantic property of the sound, e.g., telephone ringing, baby crying, but also helps us infer the spatial location of the sounding object. Both of these acoustic and spatial properties are captured by the visual stream, and require models to go beyond 2D understanding of images (3D with audio) and study the spatial (3D) aspect of audio in visuals (4D with audio). This is of vital importance for applications such as egocentric video understanding, robotic perception, AR/VR, etc. In support of robotic perception, where embodied agents can move around with both visual and auditory sensing, audio-visual simulations are also recently developed to facilitate research in this direction. The goal of this workshop is to share recent progress of audio-visual studies on the spatial-temporal (4D) dimensions, and also to discuss which directions the field should investigate next.

The AV4D workshop will bring together researchers in different subareas of visual learning of sound in spaces including computer vision, robotics, machine learning, room acoustics, and graphics to examine the challenges and opportunities emerging from visual learning of sounds embodied in spaces. We will review the current state and identify the research infrastructure needed to enable a stronger collaboration between researchers working on different subareas of this workshop.

We invite submissions of 2-4 pages extended abstracts in topics related to (but not limited to):

• Visual learning of spatial audio
• Visual learning of room acoustics
• Visual learning of impact sounds
• Audio-visual self-supervised and semi-supervised learning
• Audio-visual speaker localization and diarization
• Audio-visual source separation
• Audio-visual embodied learning
• Audio-visual simulation
• Robotic perception with vision and sound
• Audio-visual for AR/VR

A submission should take the form of an extended abstract (2-4 pages long excluding references) in PDF format using the ICCV style. We will accept submissions of (1) papers that have not been previously published or accepted for publication in substantially similar form; (2) papers that have been published or accepted for publication in recent venues including journal, conference, workshop, and arXiv; and (3) research proposals for future work with a focus on well-defined concepts and ideas. All submissions will be reviewed with single blind policy. Accepted extended abstracts will not appear in ICCV proceedings, and hence will not affect future publication of the work. We will publish all accepted extended abstracts on the workshop webpage.

CMT submissions website: https://cmt3.research.microsoft.com/AV4D2023

Key Dates:

• Extended abstract submission deadline: August 31st, 2023 (11:59 pm Pacific time)
• Notification to authors: September 12th, 2023
• Camera-ready version deadline: September 24th, 2023 (11:59 pm Pacific time)
• Workshop date: October 3rd, 2023

• We will have two paper presentation sessions: 9:30 - 10:30 and 14:15 - 15:15. Throughout the paper sessions, there will be short Q&A sessions for all papers that precede them. You can present a poster during the lunch and coffee breaks. The posters will be sized A0 (84.1 * 118.9 cm) landscape.
• For remote attendees, please upload your five-minute presentation video on CMT before the conference.
• For in-person speakers, prepare a five-minute presentation and test your laptop with the technician during breaks.
• For people who are presenting a poster, set up your poster on the poster board during breaks.