Annotating time boundaries of sound events is labor-intensive, limiting the scalability of strongly supervised learning in audio detection. To reduce annotation costs, weakly-supervised learning with only clip-level labels has been widely adopted. As an alternative, partial label learning offers a cost-effective approach, where a set of possible labels is provided instead of exact weak annotations. However, partial label learning for audio analysis remains largely unexplored. Motivated by the observation that acoustic scenes provide contextual information for constructing a set of possible sound events, we utilize acoustic scene information to construct partial labels of sound events. On the basis of this idea, in this paper, we propose a multitask learning framework that jointly performs acoustic scene classification and sound event detection with partial labels of sound events. While reducing annotation costs, weakly-supervised and partial label learning often suffer from decreased detection performance due to lacking the precise event set and their temporal annotations. To better balance between annotation cost and detection performance, we also explore a semi-supervised framework that leverages both strong and partial labels. Moreover, to refine partial labels and achieve better model training, we propose a label refinement method based on self-distillation for the proposed approach with partial labels. © 2025 K. Imoto.

This paper investigates automatic piano transcription based on computationally-efficient yet high-performant variants of the Transformer that can capture longer-term dependency over the whole musical piece. Recently, transformer-based sequence-to-sequence models have demonstrated excellent performance in piano transcription. These models, however, fail to deal with the whole piece at once due to the quadratic complexity of the self-attention mechanism, and music signals are thus typically processed in a sliding-window manner in practice. To overcome this limitation, we propose an efficient architecture with sparse attention mechanisms. Specifically, we introduce sliding-window self-attention mechanisms for both the encoder and decoder, and a hybrid global-local cross-attention mechanism that attends to various spans according to the MIDI token types. We also use a hierarchical pooling strategy between the encoder and decoder to further reduce computational load. Our experiments on the MAESTRO dataset showed that the proposed model achieved a significant reduction in computational cost and memory usage, accelerating inference speed, while maintaining transcription performance comparable to the full-attention baseline. This allows for training with longer audio contexts on the same hardware, demonstrating the viability of sparse attention for building efficient and high-performance piano transcription systems. The code is available at https://github.com/WX-Wei/efficient-seq2seq-piano-trans. © 2025 IEEE.

This paper presents TAPA-ICL, a novel in-context learning (ICL) framework for few/zero-shot symbolic music understanding task that generates human-readable analysis. Conventional ICL approaches mainly rely on the ability of large language models (LLMs) to infer patterns and tasks from contextual examples. However, while LLMs have shown basic capability of understanding symbolic music, how to enhance such capability via additional context still remains unexplored. To tackle this issue, we focus on the two major challenges as follows: (1) Sparsity of Score Input through taxonomy-aware prompt augmentation (TAPA) that distills label-to-feature profiles, reducing context length by over 10 times; and (2) Complexity of Musical Semantics Reasoning via structured chain-of-thought (CoT) prompts enforcing feature extraction, context-aware analysis, and decisionmaking. By combining TAPA strategy and the CoT reasoning prompts, our method enables effective few/zero-shot adaptation across emotion recognition, composer identification, and genre classification tasks. Experimental results show that our TAPAICL method significantly outperforms conventional few-shot ICL baselines (including those based on ultra-large and mixture of experts (MoE) models) on each downstream task and achieves slightly weaker performance to existing many-shot approaches. © 2025 IEEE.

Building Sound Event Localization and Detection (SELD) models that are robust to diverse acoustic environments remains one of the major challenges in multichannel signal processing, as reflections and reverberation can significantly confuse both the source direction and event detection. Introducing priors such as microphone geometry or room impulse response (RIR) into the model has proven effective in addressing this issue. Existing methods typically incorporate such priors in a deterministic way, often through data augmentation to enlarge data diversity. However, the uncertainty arising from the complex nature of audio acoustics remains largely underexplored in the SELD literature and naturally call for incorporating a stochastic modeling of acoustic prior. In this paper, we propose regularizing deep learning based SELD models with a physically constructed spatial covariance matrix (SCM) based on the estimated direction of arrival (DOA) and sound event detection (SED). © 2025 IEEE.

This paper presents a novel video summarization approach that prioritizes the narrative quality of the summarized video to enhance its enjoyment and appeal. While most video summarization studies focus on extracting salient scenes using lowlevel visual features, they often neglect the storytelling aspect to optimize numerical performance on standard benchmarks. To address this, we propose a multifaceted video summarization method that leverages vision and language foundation models to assess shot-level importance (e.g., 2-sec intervals) based on both visual salience and textual narrativity. Specifically, our method employs a vision-language model (VLM) to generate objective captions for individual shots. These shot-wise textual descriptions are then fed into a large language model (LLM) with a prompt designed to produce a semantically-coherent text summary with strong narrativity. The narrativity-aware text embeddings obtained by the LLM, combined with visual embeddings from a vision foundation model, are processed by a recurrent neural network (RNN) to predict importance scores. The LLM and RNN are jointly fine-tuned to align with existing benchmarks. Experiments on the SumMe benchmark demonstrated the effectiveness of our multifaceted approach, highlighting significant performance improvements and the potential of text-domain video summarization. © 2025 IEEE.

This paper describes a sound source localization (SSL) technique that combines an α-stable model for the observed signal with a neural network-based approach for modeling steering vectors. Specifically, a physics-informed neural network, referred to as Neural Steerer, is used to interpolate measured steering vectors (SVs) on a fixed microphone array. This allows for a more robust estimation of the so-called α-stable spatial measure, which represents the most plausible direction of arrival (DOA) of a target signal. As an α-stable model for the non-Gaussian case (α ∈ (0, 2)) theoretically defines a unique spatial measure, we choose to leverage it to account for residual reconstruction error of the Neural Steerer in the downstream tasks. The objective scores indicate that our proposed technique outperforms state-of-the-art methods in the case of multiple sound sources. © 2025 European Signal Processing Conference, EUSIPCO. All rights reserved.

This paper proposes visually-informed sound source separation for audio-visual understanding of indoor scenes captured by distributed microphone arrays and cameras. Our approach leverages the 3D information of sound-emitting objects, reconstructed via 3D Gaussian splatting (3DGS), to overcome a limitation of modern blind source separation methods like multichannel nonnegative matrix factorization (MNMF). While adaptable and potentially performant, the iterative optimization of MNMF often converges to poor local minima due to the highly-expressive full-rank spatial covariance matrices (SCMs) of sources. Our key idea is to treat the set of 3D Gaussians representing a sizable sound source object as a collection of sub-sources that share an audio signal but have unique emission weights, both of which are to be estimated jointly from an observed mixture. To enforce this structure, we guide MNMF by regularizing the SCM of each source object at each frequency. Specifically, we use a prior that centers the SCM estimate around a weighted sum of theoretical SCMs, which are analytically derived from the 3D Gaussian positions. Experiments with simulated data, featuring two 3D human models, demonstrated the effectiveness of the proposed method. To our knowledge, this is the first work to use 3D Gaussians as a common primitive for joint audio-visual analysis. © 2025 IEEE.

This paper presents the first attempt at zero-shot music emotion recognition (MER) to map musical pieces, represented in symbolic formats (e.g., ABC notation), onto the valence-arousal space. Conventional MER approaches typically train an end-to-end deep neural network (DNN). However, the performance of such supervised methods is limited due to the multifaceted and ambiguous nature of music emotions, compounded by the scarcity of MER datasets. To address this, we leverage knowledge transfer from large language models (LLMs) pre-trained on vast text and symbolic data. We hypothesize that LLMs possess capabilities in low-level music description and high-level emotion reasoning (not necessarily in a musical context). Accordingly, we propose a multi-agent framework that performs zero-shot MER by associating objective musical attributes (harmony, melody, rhythm, and structure) with subjective attributes (valence and arousal). Our system employs a hierarchical architecture comprising (i) musical element descriptors, (ii) chain-of-thought emotion analysts, and (iii) comprehensive predictors. Knowledge injection and zero-shot prompting are utilized to mitigate inherent model biases. Evaluations on the EMOPIA dataset demonstrate that our system, built on the Gemini-2.0-Flash backbone, significantly outperforms baseline LLM models, including ultra-large models and mixture-of-experts (MoE) systems, and performs comparably to fully supervised or fine-tuned models. © 2025 Copyright held by the owner/author(s).

This paper describes the first attempt at separation and tracking (3 D localization) of multiple moving sound sources using multiple microphone arrays fixed at known locations in an indoor environment. As for static sources, location-dependent priors have been incorporated on the time-invariant spatial covariance matrices (SCMs) of sources in the statistical framework of blind source separation based on multichannel nonnegative matrix factorization (MNMF), achieving the maximum likelihood estimation of source locations. One may thus make both the SCMs and their priors vary over time to deal with source movements. This naive extension, however, fails to localize sources when the sources are inactive, yielding non-smooth, non-continuous trajectory estimates. To solve this problem, we formulate a hierarchical probabilistic model for multichannel mixture signals that consists of inertial Markov models for source locations, location-aware moving-average models for source SCMs, and NMF-based lowrank models for the power spectral densities (PSDs) of sources. All the time-varying attributes of sources are jointly estimated under a maximum-a-posteriori (MAP) principle, and the source images are then estimated with a multichannel Wiener filter. The experiment using simulated data with two moving sources and four four-channel arrays showed that the proposed method achieved better separation and smoother localization. © 2025 IEEE.