Vincent's Arxiv FrontPage

Sound designers and Foley artists usually sonorize a scene, such as from a movie or video game, by manually annotating and sonorizing each action of interest in the video.In our case, the intent is to leave full creative control to sound designers with a tool that allows them to bypass the more repetitive parts of their work, thus being able to focus on the creative aspects of sound production.We achieve this presenting Stable-V2A, a two-stage model consisting of: an RMS-Mapper that estimates an envelope representative of the audio characteristics associated with the input video; and Stable-Foley, a diffusion model based on Stable Audio Open that generates audio semantically and temporally aligned with the target video.Temporal alignment is guaranteed by the use of the envelope as a ControlNet input, while semantic alignment is achieved through the use of sound representations chosen by the designer as cross-attention conditioning of the diffusion process.We train and test our model on Greatest Hits, a dataset commonly used to evaluate V2A models.In addition, to test our model on a case study of interest, we introduce Walking The Maps, a dataset of videos extracted from video games depicting animated characters walking in different locations. 0.714Samples and code available on our demo page at https://ispamm.github.io/Stable-V2A.

link

The advances in the development of Facilitative Playbacks extracted from High-Speed videoendoscopic sequences of the vocal folds are hindered by a notable lack of publicly available datasets annotated with the semantic segmentations corresponding to the area of the glottal gap.This fact also limits the reproducibility and further exploration of existing research in this field. To address this gap, GIRAFE is a data repository designed to facilitate the development of advanced techniques for the semantic segmentation, analysis, and fast evaluation of High-Speed videoendoscopic sequences of the vocal folds.The repository includes 65 high-speed videoendoscopic recordings from a cohort of 50 patients (30 female, 20 male).The dataset comprises 15 recordings from healthy controls, 26 from patients with diagnosed voice disorders, and 24 with an unknown health condition. 0.91All of them were manually annotated by an expert, including the masks corresponding to the semantic segmentation of the glottal gap.The repository is also complemented with the automatic segmentation of the glottal area using different state-of-the-art approaches. This data set has already supported several studies, which demonstrates its usefulness for the development of new glottal gap segmentation algorithms from High-Speed-Videoendoscopic sequences to improve or create new Facilitative Playbacks.Despite these advances and others in the field, the broader challenge of performing an accurate and completely automatic semantic segmentation method of the glottal area remains open.

link

Conflict scholars have used rule-based approaches to extract information about political violence from news reports and texts.Recent Natural Language Processing developments move beyond rigid rule-based approaches.We review our recent ConfliBERT language model (Hu et al. 2022) to process political and violence related texts.The model can be used to extract actor and action classifications from texts about political conflict.When fine-tuned, results show that ConfliBERT has superior performance in accuracy, precision and recall over other large language models (LLM) like Google's Gemma 2 (9B), Meta's Llama 3.1 (7B), and Alibaba's Qwen 2.5 (14B) within its relevant domains.It is also hundreds of times faster than these more generalist LLMs.These results are illustrated using texts from the BBC, re3d, and the Global Terrorism Dataset (GTD). 0.772

link

Drought is one of the most destructive and expensive natural disasters, severely impacting natural resources and risks by depleting water resources and diminishing agricultural yields.Under climate change, accurately predicting drought is critical for mitigating drought-induced risks.However, the intricate interplay among the physical and biological drivers that regulate droughts limits the predictability and understanding of drought, particularly at a subseasonal to seasonal (S2S) time scale.While deep learning has been demonstrated with potential in addressing climate forecasting challenges, its application to drought prediction has received relatively less attention.In this work, we propose a new dataset, DroughtSet, which integrates relevant predictive features and three drought indices from multiple remote sensing and reanalysis datasets across the contiguous United States (CONUS). 0.847DroughtSet specifically provides the machine learning community with a new real-world dataset to benchmark drought prediction models and more generally, time-series forecasting methods.Furthermore, we propose a spatial-temporal model SPDrought to predict and interpret S2S droughts.Our model learns from the spatial and temporal information of physical and biological features to predict three types of droughts simultaneously.Multiple strategies are employed to quantify the importance of physical and biological features for drought prediction.Our results provide insights for researchers to better understand the predictability and sensitivity of drought to biological and physical conditions.We aim to contribute to the climate field by proposing a new tool to predict and understand the occurrence of droughts and provide the AI community with a new benchmark to study deep learning applications in climate science.

link

Determining the sustainability impact of companies is a highly complex subject which has garnered more and more attention over the past few years.Today, investors largely rely on sustainability-ratings from established rating-providers in order to analyze how responsibly a company acts.However, those ratings have recently been criticized for being hard to understand and nearly impossible to reproduce. An independent way to find out about the sustainability practices of companies lies in the rich landscape of news article data.In this paper, we explore a different approach to identify key opportunities and challenges of companies in the sustainability domain.We present a novel dataset of more than 840,000 news articles which were gathered for major German companies between January 2023 and September 2024. 0.859By applying a mixture of Natural Language Processing techniques, we first identify relevant articles, before summarizing them and extracting their sustainability-related sentiment and aspect using Large Language Models (LLMs).Furthermore, we conduct an evaluation of the obtained data and determine that the LLM-produced answers are accurate.We release both datasets at https://github.com/Bailefan/Nano-ESG.

link

Gaussian Splatting has enabled real-time 3D human avatars with unprecedented levels of visual quality.While previous methods require a desktop GPU for real-time inference of a single avatar, we aim to squeeze multiple Gaussian avatars onto a portable virtual reality headset with real-time drivable inference.We begin by training a previous work, Animatable Gaussians, on a high quality dataset captured with 512 cameras. 0.788The Gaussians are animated by controlling base set of Gaussians with linear blend skinning (LBS) motion and then further adjusting the Gaussians with a neural network decoder to correct their appearance.When deploying the model on a Meta Quest 3 VR headset, we find two major computational bottlenecks: the decoder and the rendering.To accelerate the decoder, we train the Gaussians in UV-space instead of pixel-space, and we distill the decoder to a single neural network layer.Further, we discover that neighborhoods of Gaussians can share a single corrective from the decoder, which provides an additional speedup.To accelerate the rendering, we develop a custom pipeline in Vulkan that runs on the mobile GPU.Putting it all together, we run 3 Gaussian avatars concurrently at 72 FPS on a VR headset.Demo videos are at https://forresti.github.io/squeezeme.

link

The suite of datasets commonly used to train and evaluate the mathematical capabilities of AI-based mathematical copilots (primarily large language models) exhibit several shortcomings.These limitations include a restricted scope of mathematical complexity, typically not exceeding lower undergraduate-level mathematics, binary rating protocols and other issues, which makes comprehensive proof-based evaluation suites difficult.We systematically explore these limitations and contend that enhancing the capabilities of large language models, or any forthcoming advancements in AI-based mathematical assistants (copilots or "thought partners"), necessitates a paradigm shift in the design of mathematical datasets and the evaluation criteria of mathematical ability: It is necessary to move away from result-based datasets (theorem statement to theorem proof) and convert the rich facets of mathematical research practice to data LLMs can train on.Examples of these are mathematical workflows (sequences of atomic, potentially subfield-dependent tasks that are often performed when creating new mathematics), which are an important part of the proof-discovery process.Additionally, we advocate for mathematical dataset developers to consider the concept of "motivated proof", introduced by G. P\'olya in 1949, which can serve as a blueprint for datasets that offer a better proof learning signal, alleviating some of the mentioned limitations.Lastly, we introduce math datasheets for datasets, extending the general, dataset-agnostic variants of datasheets: We provide a questionnaire designed specifically for math datasets that we urge dataset creators to include with their datasets. 0.772This will make creators aware of potential limitations of their datasets while at the same time making it easy for readers to assess it from the point of view of training and evaluating mathematical copilots.

link

Automated analysis of vast Earth observation data via interactive Vision-Language Models (VLMs) can unlock new opportunities for environmental monitoring, disaster response, and resource management.Existing generic VLMs do not perform well on Remote Sensing data, while the recent Geo-spatial VLMs remain restricted to a fixed resolution and few sensor modalities.In this paper, we introduce EarthDial, a conversational assistant specifically designed for Earth Observation (EO) data, transforming complex, multi-sensory Earth observations into interactive, natural language dialogues. 0.733EarthDial supports multi-spectral, multi-temporal, and multi-resolution imagery, enabling a wide range of remote sensing tasks, including classification, detection, captioning, question answering, visual reasoning, and visual grounding.To achieve this, we introduce an extensive instruction tuning dataset comprising over 11.11M instruction pairs covering RGB, Synthetic Aperture Radar (SAR), and multispectral modalities such as Near-Infrared (NIR) and infrared. 0.741Furthermore, EarthDial handles bi-temporal and multi-temporal sequence analysis for applications like change detection.Our extensive experimental results on 37 downstream applications demonstrate that EarthDial outperforms existing generic and domain-specific models, achieving better generalization across various EO tasks.

link

Recent advancements in large vision language models (VLMs) tailored for autonomous driving (AD) have shown strong scene understanding and reasoning capabilities, making them undeniable candidates for end-to-end driving systems.However, limited work exists on studying the trustworthiness of DriveVLMs -- a critical factor that directly impacts public transportation safety.In this paper, we introduce AutoTrust, a comprehensive trustworthiness benchmark for large vision-language models in autonomous driving (DriveVLMs), considering diverse perspectives -- including trustfulness, safety, robustness, privacy, and fairness.We constructed the largest visual question-answering dataset for investigating trustworthiness issues in driving scenarios, comprising over 10k unique scenes and 18k queries. 0.725We evaluated six publicly available VLMs, spanning from generalist to specialist, from open-source to commercial models.Our exhaustive evaluations have unveiled previously undiscovered vulnerabilities of DriveVLMs to trustworthiness threats.Specifically, we found that the general VLMs like LLaVA-v1.6 and GPT-4o-mini surprisingly outperform specialized models fine-tuned for driving in terms of overall trustworthiness.DriveVLMs like DriveLM-Agent are particularly vulnerable to disclosing sensitive information.Additionally, both generalist and specialist VLMs remain susceptible to adversarial attacks and struggle to ensure unbiased decision-making across diverse environments and populations.Our findings call for immediate and decisive action to address the trustworthiness of DriveVLMs -- an issue of critical importance to public safety and the welfare of all citizens relying on autonomous transportation systems.Our benchmark is publicly available at \url{https://github.com/taco-group/AutoTrust}, and the leaderboard is released at \url{https://taco-group.github.io/AutoTrust/}.

link

Formal verification using proof assistants, such as Coq, enables the creation of high-quality software.However, the verification process requires significant expertise and manual effort to write proofs.Recent work has explored automating proof synthesis using machine learning and large language models (LLMs).This work has shown that identifying relevant premises, such as lemmas and definitions, can aid synthesis.We present Rango, a fully automated proof synthesis tool for Coq that automatically identifies relevant premises and also similar proofs from the current project and uses them during synthesis.Rango uses retrieval augmentation at every step of the proof to automatically determine which proofs and premises to include in the context of its fine-tuned LLM.In this way, Rango adapts to the project and to the evolving state of the proof.We create a new dataset, CoqStoq, of 2,226 open-source Coq projects and 196,929 theorems from GitHub, which includes both training data and a curated evaluation benchmark of well-maintained projects. 0.86On this benchmark, Rango synthesizes proofs for 32.0% of the theorems, which is 29% more theorems than the prior state-of-the-art tool Tactician.Our evaluation also shows that Rango adding relevant proofs to its context leads to a 47% increase in the number of theorems proven.

link

Keyword extraction involves identifying the most descriptive words in a document, allowing automatic categorisation and summarisation of large quantities of diverse textual data.Relying on the insight that real-world keyword detection often requires handling of diverse content, we propose a novel supervised keyword extraction approach based on the mixture of experts (MoE) technique.MoE uses a learnable routing sub-network to direct information to specialised experts, allowing them to specialize in distinct regions of the input space.SEKE, a mixture of Specialised Experts for supervised Keyword Extraction, uses DeBERTa as the backbone model and builds on the MoE framework, where experts attend to each token, by integrating it with a recurrent neural network (RNN), to allow successful extraction even on smaller corpora, where specialisation is harder due to lack of training data.The MoE framework also provides an insight into inner workings of individual experts, enhancing the explainability of the approach.We benchmark SEKE on multiple English datasets, achieving state-of-the-art performance compared to strong supervised and unsupervised baselines. 0.711Our analysis reveals that depending on data size and type, experts specialize in distinct syntactic and semantic components, such as punctuation, stopwords, parts-of-speech, or named entities.Code is available at: https://github.com/matejMartinc/SEKE_keyword_extraction

link

Machine learning (ML) has been extensively employed in planar perovskite photovoltaics to screen effective organic molecular additives, while encountering predictive biases for novel materials due to small datasets and reliance on predefined descriptors.Present work thus proposes an effective approach, Co-Pilot for Perovskite Additive Screener (Co-PAS), an ML-driven framework designed to accelerate additive screening for perovskite solar cells (PSCs).Co-PAS overcomes predictive biases by integrating the Molecular Scaffold Classifier (MSC) for scaffold-based pre-screening and utilizing Junction Tree Variational Autoencoder (JTVAE) latent vectors to enhance molecular structure representation, thereby enhancing the accuracy of power conversion efficiency (PCE) predictions.Leveraging Co-PAS, we integrate domain knowledge to screen an extensive dataset of 250,000 molecules from PubChem, prioritizing candidates based on predicted PCE values and key molecular properties such as donor number, dipole moment, and hydrogen bond acceptor count. 0.717This workflow leads to the identification of several promising passivating molecules, including the novel Boc-L-threonine N-hydroxysuccinimide ester (BTN), which, to our knowledge, has not been explored as an additive in PSCs and achieves a device PCE of 25.20%.Our results underscore the potential of Co-PAS in advancing additive discovery for high-performance PSCs.

link

Geospatial models must adapt to the diversity of Earth observation data in terms of resolutions, scales, and modalities.However, existing approaches expect fixed input configurations, which limits their practical applicability.We propose AnySat, a multimodal model based on joint embedding predictive architecture (JEPA) and resolution-adaptive spatial encoders, allowing us to train a single model on highly heterogeneous data in a self-supervised manner.To demonstrate the advantages of this unified approach, we compile GeoPlex, a collection of $5$ multimodal datasets with varying characteristics and $11$ distinct sensors. 0.814We then train a single powerful model on these diverse datasets simultaneously.Once fine-tuned, we achieve better or near state-of-the-art results on the datasets of GeoPlex and $4$ additional ones for $5$ environment monitoring tasks: land cover mapping, tree species identification, crop type classification, change detection, and flood segmentation. 0.736The code and models are available at https://github.com/gastruc/AnySat.

link

We propose scaling up 3D scene reconstruction by training with synthesized data.At the core of our work is MegaSynth, a procedurally generated 3D dataset comprising 700K scenes - over 50 times larger than the prior real dataset DL3DV - dramatically scaling the training data. 0.785To enable scalable data generation, our key idea is eliminating semantic information, removing the need to model complex semantic priors such as object affordances and scene composition.Instead, we model scenes with basic spatial structures and geometry primitives, offering scalability.Besides, we control data complexity to facilitate training while loosely aligning it with real-world data distribution to benefit real-world generalization.We explore training LRMs with both MegaSynth and available real data.Experiment results show that joint training or pre-training with MegaSynth improves reconstruction quality by 1.2 to 1.8 dB PSNR across diverse image domains.Moreover, models trained solely on MegaSynth perform comparably to those trained on real data, underscoring the low-level nature of 3D reconstruction.Additionally, we provide an in-depth analysis of MegaSynth's properties for enhancing model capability, training stability, and generalization.

link

Recent progress in generative diffusion models has greatly advanced text-to-video generation.While text-to-video models trained on large-scale, diverse datasets can produce varied outputs, these generations often deviate from user preferences, highlighting the need for preference alignment on pre-trained models.Although Direct Preference Optimization (DPO) has demonstrated significant improvements in language and image generation, we pioneer its adaptation to video diffusion models and propose a VideoDPO pipeline by making several key adjustments.Unlike previous image alignment methods that focus solely on either (i) visual quality or (ii) semantic alignment between text and videos, we comprehensively consider both dimensions and construct a preference score accordingly, which we term the OmniScore.We design a pipeline to automatically collect preference pair data based on the proposed OmniScore and discover that re-weighting these pairs based on the score significantly impacts overall preference alignment.Our experiments demonstrate substantial improvements in both visual quality and semantic alignment, ensuring that no preference aspect is neglected.Code and data will be shared at https://videodpo.github.io/. 0.708

link

Scalable learning of humanoid robots is crucial for their deployment in real-world applications.While traditional approaches primarily rely on reinforcement learning or teleoperation to achieve whole-body control, they are often limited by the diversity of simulated environments and the high costs of demonstration collection.In contrast, human videos are ubiquitous and present an untapped source of semantic and motion information that could significantly enhance the generalization capabilities of humanoid robots.This paper introduces Humanoid-X, a large-scale dataset of over 20 million humanoid robot poses with corresponding text-based motion descriptions, designed to leverage this abundant data. 0.958Humanoid-X is curated through a comprehensive pipeline: data mining from the Internet, video caption generation, motion retargeting of humans to humanoid robots, and policy learning for real-world deployment. 0.758With Humanoid-X, we further train a large humanoid model, UH-1, which takes text instructions as input and outputs corresponding actions to control a humanoid robot.Extensive simulated and real-world experiments validate that our scalable training approach leads to superior generalization in text-based humanoid control, marking a significant step toward adaptable, real-world-ready humanoid robots.

link

As the Information Retrieval (IR) field increasingly recognizes the importance of inclusivity, addressing the needs of low-resource languages remains a significant challenge.This paper introduces the first large-scale Urdu IR dataset, created by translating the MS MARCO dataset through machine translation. 0.801We establish baseline results through zero-shot learning for IR in Urdu and subsequently apply the mMARCO multilingual IR methodology to this newly translated dataset.Our findings demonstrate that the fine-tuned model (Urdu-mT5-mMARCO) achieves a Mean Reciprocal Rank (MRR@10) of 0.247 and a Recall@10 of 0.439, representing significant improvements over zero-shot results and showing the potential for expanding IR access for Urdu speakers.By bridging access gaps for speakers of low-resource languages, this work not only advances multilingual IR research but also emphasizes the ethical and societal importance of inclusive IR technologies.This work provides valuable insights into the challenges and solutions for improving language representation and lays the groundwork for future research, especially in South Asian languages, which can benefit from the adaptable methods used in this study.

link

Ultrasound imaging of the medial elbow is crucial for the early identification of Ulnar Collateral Ligament (UCL) injuries.Specifically, measuring the elbow joint space in ultrasound images is used to assess the valgus instability of elbow.To automate this measurement, a precisely annotated dataset is necessary; however, no publicly available dataset has been proposed thus far.This study introduces a novel ultrasound medial elbow dataset for measuring joint space to diagnose Ulnar Collateral Ligament (UCL) injuries.The dataset comprises 4,201 medial elbow ultrasound images from 22 subjects, with landmark annotations on the humerus and ulna. 0.847The annotations are made precisely by the authors under the supervision of three orthopedic surgeons.We evaluated joint space measurement methods using our proposed dataset with several landmark detection approaches, including ViTPose, HRNet, PCT, YOLOv8, and U-Net.In addition, we propose using Shape Subspace (SS) for landmark refinement in heatmap-based landmark detection.The results show that the mean Euclidean distance error of joint space is 0.116 mm when using HRNet.Furthermore, the SS landmark refinement improves the mean absolute error of landmark positions by 0.010 mm with HRNet and by 0.103 mm with ViTPose on average.These highlight the potential for high-precision, real-time diagnosis of UCL injuries and associated risks, which could be leveraged in large-scale screening.Lastly, we demonstrate point-based segmentation of the humerus and ulna using the detected landmarks as input.The dataset will be made publicly available upon acceptance of this paper at: https://github.com/Akahori000/Ultrasound-Medial-Elbow-Dataset. 0.801

link

We present NAVCON, a large-scale annotated Vision-Language Navigation (VLN) corpus built on top of two popular datasets (R2R and RxR). 0.853The paper introduces four core, cognitively motivated and linguistically grounded, navigation concepts and an algorithm for generating large-scale silver annotations of naturally occurring linguistic realizations of these concepts in navigation instructions.We pair the annotated instructions with video clips of an agent acting on these instructions.NAVCON contains 236, 316 concept annotations for approximately 30, 0000 instructions and 2.7 million aligned images (from approximately 19, 000 instructions) showing what the agent sees when executing an instruction.To our knowledge, this is the first comprehensive resource of navigation concepts.We evaluated the quality of the silver annotations by conducting human evaluation studies on NAVCON samples.As further validation of the quality and usefulness of the resource, we trained a model for detecting navigation concepts and their linguistic realizations in unseen instructions.Additionally, we show that few-shot learning with GPT-4o performs well on this task using large-scale silver annotations of NAVCON.

link

This study introduces CLASP (Contrastive Language-Speech Pretraining), a multilingual, multimodal representation tailored for audio-text information retrieval.CLASP leverages the synergy between spoken content and textual data.During training, we utilize our newly introduced speech-text dataset, which encompasses 15 diverse categories ranging from fiction to religion. 0.859CLASP's audio component integrates audio spectrograms with a pre-trained self-supervised speech model, while its language encoding counterpart employs a sentence encoder pre-trained on over 100 languages.This unified lightweight model bridges the gap between various modalities and languages, enhancing its effectiveness in handling and retrieving multilingual and multimodal data.Our evaluations across multiple languages demonstrate that CLASP establishes new benchmarks in HITS@1, MRR, and meanR metrics, outperforming traditional ASR-based retrieval approaches in specific scenarios.

link

Online reporting platforms have enabled citizens around the world to collectively share their opinions and report in real time on events impacting their local communities.Systematically organizing (e.g., categorizing by attributes) and geotagging large amounts of crowdsourced information is crucial to ensuring that accurate and meaningful insights can be drawn from this data and used by policy makers to bring about positive change.These tasks, however, typically require extensive manual annotation efforts.In this paper we present Uchaguzi-2022, a dataset of 14k categorized and geotagged citizen reports related to the 2022 Kenyan General Election containing mentions of election-related issues such as official misconduct, vote count irregularities, and acts of violence. 0.855We use this dataset to investigate whether language models can assist in scalably categorizing and geotagging reports, thus highlighting its potential application in the AI for Social Good space.

link

Evaluation plays a crucial role in the advancement of information retrieval (IR) models.However, current benchmarks, which are based on predefined domains and human-labeled data, face limitations in addressing evaluation needs for emerging domains both cost-effectively and efficiently.To address this challenge, we propose the Automated Heterogeneous Information Retrieval Benchmark (AIR-Bench).AIR-Bench is distinguished by three key features: 1) Automated.The testing data in AIR-Bench is automatically generated by large language models (LLMs) without human intervention.2) Heterogeneous.The testing data in AIR-Bench is generated with respect to diverse tasks, domains and languages.3) Dynamic.The domains and languages covered by AIR-Bench are constantly augmented to provide an increasingly comprehensive evaluation benchmark for community developers.We develop a reliable and robust data generation pipeline to automatically create diverse and high-quality evaluation datasets based on real-world corpora. 0.722Our findings demonstrate that the generated testing data in AIR-Bench aligns well with human-labeled testing data, making AIR-Bench a dependable benchmark for evaluating IR models.The resources in AIR-Bench are publicly available at https://github.com/AIR-Bench/AIR-Bench.

link

Text-driven human motion synthesis is capturing significant attention for its ability to effortlessly generate intricate movements from abstract text cues, showcasing its potential for revolutionizing motion design not only in film narratives but also in virtual reality experiences and computer game development.Existing methods often rely on 3D motion capture data, which require special setups resulting in higher costs for data acquisition, ultimately limiting the diversity and scope of human motion.In contrast, 2D human videos offer a vast and accessible source of motion data, covering a wider range of styles and activities.In this paper, we explore leveraging 2D human motion extracted from videos as an alternative data source to improve text-driven 3D motion generation.Our approach introduces a novel framework that disentangles local joint motion from global movements, enabling efficient learning of local motion priors from 2D data.We first train a single-view 2D local motion generator on a large dataset of text-motion pairs.To enhance this model to synthesize 3D motion, we fine-tune the generator with 3D data, transforming it into a multi-view generator that predicts view-consistent local joint motion and root dynamics.Experiments on the HumanML3D dataset and novel text prompts demonstrate that our method efficiently utilizes 2D data, supporting realistic 3D human motion generation and broadening the range of motion types it supports. 0.709Our code will be made publicly available at https://zju3dv.github.io/Motion-2-to-3/.

link

This study introduces an AI-driven platform for continuous and passive patient monitoring in hospital settings, developed by LookDeep Health.Leveraging advanced computer vision, the platform provides real-time insights into patient behavior and interactions through video analysis, securely storing inference results in the cloud for retrospective evaluation.The dataset, compiled in collaboration with 11 hospital partners, encompasses over 300 high-risk fall patients and over 1,000 days of inference, enabling applications such as fall detection and safety monitoring for vulnerable patient populations. 0.899To foster innovation and reproducibility, an anonymized subset of this dataset is publicly available. 0.857The AI system detects key components in hospital rooms, including individual presence and role, furniture location, motion magnitude, and boundary crossings.Performance evaluation demonstrates strong accuracy in object detection (macro F1-score = 0.92) and patient-role classification (F1-score = 0.98), as well as reliable trend analysis for the "patient alone" metric (mean logistic regression accuracy = 0.82 \pm 0.15).These capabilities enable automated detection of patient isolation, wandering, or unsupervised movement-key indicators for fall risk and other adverse events.This work establishes benchmarks for validating AI-driven patient monitoring systems, highlighting the platform's potential to enhance patient safety and care by providing continuous, data-driven insights into patient behavior and interactions.

link

Artificial intelligence generative models exhibit remarkable capabilities in content creation, particularly in face image generation, customization, and restoration.However, current AI-generated faces (AIGFs) often fall short of human preferences due to unique distortions, unrealistic details, and unexpected identity shifts, underscoring the need for a comprehensive quality evaluation framework for AIGFs.To address this need, we introduce FaceQ, a large-scale, comprehensive database of AI-generated Face images with fine-grained Quality annotations reflecting human preferences. 0.756The FaceQ database comprises 12,255 images generated by 29 models across three tasks: (1) face generation, (2) face customization, and (3) face restoration.It includes 32,742 mean opinion scores (MOSs) from 180 annotators, assessed across multiple dimensions: quality, authenticity, identity (ID) fidelity, and text-image correspondence.Using the FaceQ database, we establish F-Bench, a benchmark for comparing and evaluating face generation, customization, and restoration models, highlighting strengths and weaknesses across various prompts and evaluation dimensions.Additionally, we assess the performance of existing image quality assessment (IQA), face quality assessment (FQA), AI-generated content image quality assessment (AIGCIQA), and preference evaluation metrics, manifesting that these standard metrics are relatively ineffective in evaluating authenticity, ID fidelity, and text-image correspondence.The FaceQ database will be publicly available upon publication.

link

This work presents the BanglishRev Dataset, the largest e-commerce product review dataset to date for reviews written in Bengali, English, a mixture of both and Banglish, Bengali words written with English alphabets. 0.867The dataset comprises of 1.74 million written reviews from 3.2 million ratings information collected from a total of 128k products being sold in online e-commerce platforms targeting the Bengali population. 0.835It includes an extensive array of related metadata for each of the reviews including the rating given by the reviewer, date the review was posted and date of purchase, number of likes, dislikes, response from the seller, images associated with the review etc.With sentiment analysis being the most prominent usage of review datasets, experimentation with a binary sentiment analysis model with the review rating serving as an indicator of positive or negative sentiment was conducted to evaluate the effectiveness of the large amount of data presented in BanglishRev for sentiment analysis tasks.A BanglishBERT model is trained on the data from BanglishRev with reviews being considered labeled positive if the rating is greater than 3 and negative if the rating is less than or equal to 3.The model is evaluated by being testing against a previously published manually annotated dataset for e-commerce reviews written in a mixture of Bangla, English and Banglish.The experimental model achieved an exceptional accuracy of 94\% and F1 score of 0.94, demonstrating the dataset's efficacy for sentiment analysis.Some of the intriguing patterns and observations seen within the dataset and future research directions where the dataset can be utilized is also discussed and explored.The dataset can be accessed through https://huggingface.co/datasets/BanglishRev/bangla-english-and-code-mixed-ecommerce-review-dataset. 0.917

link

Generating realistic human videos remains a challenging task, with the most effective methods currently relying on a human motion sequence as a control signal.Existing approaches often use existing motion extracted from other videos, which restricts applications to specific motion types and global scene matching.We propose Move-in-2D, a novel approach to generate human motion sequences conditioned on a scene image, allowing for diverse motion that adapts to different scenes.Our approach utilizes a diffusion model that accepts both a scene image and text prompt as inputs, producing a motion sequence tailored to the scene.To train this model, we collect a large-scale video dataset featuring single-human activities, annotating each video with the corresponding human motion as the target output. 0.708Experiments demonstrate that our method effectively predicts human motion that aligns with the scene image after projection.Furthermore, we show that the generated motion sequence improves human motion quality in video synthesis tasks.

link

This paper presents the "Speak & Improve Challenge 2025: Spoken Language Assessment and Feedback" -- a challenge associated with the ISCA SLaTE 2025 Workshop.The goal of the challenge is to advance research on spoken language assessment and feedback, with tasks associated with both the underlying technology and language learning feedback.Linked with the challenge, the Speak & Improve (S&I) Corpus 2025 is being pre-released, a dataset of L2 learner English data with holistic scores and language error annotation, collected from open (spontaneous) speaking tests on the Speak & Improve learning platform. 0.839The corpus consists of 340 hours of audio data from second language English learners with holistic scores, and a 60-hour subset with manual transcriptions and error labels. 0.718The Challenge has four shared tasks: Automatic Speech Recognition (ASR), Spoken Language Assessment (SLA), Spoken Grammatical Error Correction (SGEC), and Spoken Grammatical Error Correction Feedback (SGECF).Each of these tasks has a closed track where a predetermined set of models and data sources are allowed to be used, and an open track where any public resource may be used.Challenge participants may do one or more of the tasks.This paper describes the challenge, the S&I Corpus 2025, and the baseline systems released for the Challenge. 0.738

link

We introduce the Speak \& Improve Corpus 2025, a dataset of L2 learner English data with holistic scores and language error annotation, collected from open (spontaneous) speaking tests on the Speak \& Improve learning platform https://speakandimprove.com . 0.85The aim of the corpus release is to address a major challenge to developing L2 spoken language processing systems, the lack of publicly available data with high-quality annotations.It is being made available for non-commercial use on the ELiT website.In designing this corpus we have sought to make it cover a wide-range of speaker attributes, from their L1 to their speaking ability, as well as providing manual annotations.This enables a range of language-learning tasks to be examined, such as assessing speaking proficiency or providing feedback on grammatical errors in a learner's speech.Additionally, the data supports research into the underlying technology required for these tasks including automatic speech recognition (ASR) of low resource L2 learner English, disfluency detection or spoken grammatical error correction (GEC).The corpus consists of around 340 hours of L2 English learners audio with holistic scores, and a subset of audio annotated with transcriptions and error labels.

link

Engaging messages delivered by teachers are a key aspect of the classroom discourse that influences student outcomes.However, improving this communication is challenging due to difficulties in obtaining observations.This study presents a methodology for efficiently extracting actual observations of engaging messages from audio-recorded lessons.We collected 2,477 audio-recorded lessons from 75 teachers over two academic years. 0.893Using automatic transcription and keyword-based filtering analysis, we identified and classified engaging messages.This method reduced the information to be analysed by 90%, optimising the time and resources required compared to traditional manual coding.Subsequent descriptive analysis revealed that the most used messages emphasised the future benefits of participating in school activities.In addition, the use of engaging messages decreased as the academic year progressed.This study offers insights for researchers seeking to extract information from teachers' discourse in naturalistic settings and provides useful information for designing interventions to improve teachers' communication strategies.

link

Most existing video understanding benchmarks for multimodal large language models (MLLMs) focus only on short videos.The limited number of benchmarks for long video understanding often rely solely on multiple-choice questions (MCQs).However, because of the inherent limitation of MCQ-based evaluation and the increasing reasoning ability of MLLMs, models can give the current answer purely by combining short video understanding with elimination, without genuinely understanding the video content.To address this gap, we introduce CG-Bench, a novel benchmark designed for clue-grounded question answering in long videos.CG-Bench emphasizes the model's ability to retrieve relevant clues for questions, enhancing evaluation credibility.It features 1,219 manually curated videos categorized by a granular system with 14 primary categories, 171 secondary categories, and 638 tertiary categories, making it the largest benchmark for long video analysis.The benchmark includes 12,129 QA pairs in three major question types: perception, reasoning, and hallucination.Compensating the drawbacks of pure MCQ-based evaluation, we design two novel clue-based evaluation methods: clue-grounded white box and black box evaluations, to assess whether the model generates answers based on the correct understanding of the video.We evaluate multiple closed-source and open-source MLLMs on CG-Bench.Results indicate that current models significantly underperform in understanding long videos compared to short ones, and a significant gap exists between open-source and commercial models.We hope CG-Bench can advance the development of more trustworthy and capable MLLMs for long video understanding.All annotations and video data are released at https://cg-bench.github.io/leaderboard/. 0.772

link

This paper introduces a novel dataset construction pipeline that samples pairs of frames from videos and uses multimodal large language models (MLLMs) to generate editing instructions for training instruction-based image manipulation models. 0.707Video frames inherently preserve the identity of subjects and scenes, ensuring consistent content preservation during editing.Additionally, video data captures diverse, natural dynamics-such as non-rigid subject motion and complex camera movements-that are difficult to model otherwise, making it an ideal source for scalable dataset construction.Using this approach, we create a new dataset to train InstructMove, a model capable of instruction-based complex manipulations that are difficult to achieve with synthetically generated datasets. 0.707Our model demonstrates state-of-the-art performance in tasks such as adjusting subject poses, rearranging elements, and altering camera perspectives.

link

We present OpenNER 1.0, a standardized collection of openly available named entity recognition (NER) datasets.OpenNER contains 34 datasets spanning 51 languages, annotated in varying named entity ontologies. 0.838We correct annotation format issues, standardize the original datasets into a uniform representation, map entity type names to be more consistent across corpora, and provide the collection in a structure that enables research in multilingual and multi-ontology NER. 0.731We provide baseline models using three pretrained multilingual language models to compare the performance of recent models and facilitate future research in NER.

link

Recovering the geometry and materials of objects from a single image is challenging due to its under-constrained nature.In this paper, we present Neural LightRig, a novel framework that boosts intrinsic estimation by leveraging auxiliary multi-lighting conditions from 2D diffusion priors.Specifically, 1) we first leverage illumination priors from large-scale diffusion models to build our multi-light diffusion model on a synthetic relighting dataset with dedicated designs.This diffusion model generates multiple consistent images, each illuminated by point light sources in different directions.2) By using these varied lighting images to reduce estimation uncertainty, we train a large G-buffer model with a U-Net backbone to accurately predict surface normals and materials.Extensive experiments validate that our approach significantly outperforms state-of-the-art methods, enabling accurate surface normal and PBR material estimation with vivid relighting effects.Code and dataset are available on our project page at https://projects.zxhezexin.com/neural-lightrig. 0.726

link

Rat behavior modeling goes to the heart of many scientific studies, yet the textureless body surface evades automatic analysis as it literally has no keypoints that detectors can find.The movement of the body surface, however, is a rich source of information for deciphering the rat behavior.We introduce two key contributions to automatically recover densely 3D sampled rat body surface points, passively.The first is RatDome, a novel multi-camera system for rat behavior capture, and a large-scale dataset captured with it that consists of pairs of 3D keypoints and 3D body surface points. 0.744The second is RatBodyFormer, a novel network to transform detected keypoints to 3D body surface points.RatBodyFormer is agnostic to the exact locations of the 3D body surface points in the training data and is trained with masked-learning.We experimentally validate our framework with a number of real-world experiments.Our results collectively serve as a novel foundation for automated rat behavior analysis and will likely have far-reaching implications for biomedical and neuroscientific research.

link

End-to-end driving systems have made rapid progress, but have so far not been applied to the challenging new CARLA Leaderboard 2.0.Further, while there is a large body of literature on end-to-end architectures and training strategies, the impact of the training dataset is often overlooked.In this work, we make a first attempt at end-to-end driving for Leaderboard 2.0.Instead of investigating architectures, we systematically analyze the training dataset, leading to new insights: (1) Expert style significantly affects downstream policy performance.(2) In complex data sets, the frames should not be weighted on the basis of simplistic criteria such as class frequencies.(3) Instead, estimating whether a frame changes the target labels compared to previous frames can reduce the size of the dataset without removing important information.By incorporating these findings, our model ranks first and second respectively on the map and sensors tracks of the 2024 CARLA Challenge, and sets a new state-of-the-art on the Bench2Drive test routes.Finally, we uncover a design flaw in the current evaluation metrics and propose a modification for future challenges.Our dataset, code, and pre-trained models are publicly available at https://github.com/autonomousvision/carla_garage. 0.716

link

Multimodal Large Language Models (MLLMs) have achieved impressive results on various vision tasks, leveraging recent advancements in large language models.However, a critical question remains unaddressed: do MLLMs perceive visual information similarly to humans?Current benchmarks lack the ability to evaluate MLLMs from this perspective.To address this challenge, we introduce HVSBench, a large-scale benchmark designed to assess the alignment between MLLMs and the human visual system (HVS) on fundamental vision tasks that mirror human vision.HVSBench curated over 85K multimodal samples, spanning 13 categories and 5 fields in HVS, including Prominence, Subitizing, Prioritizing, Free-Viewing, and Searching. 0.744Extensive experiments demonstrate the effectiveness of our benchmark in providing a comprehensive evaluation of MLLMs.Specifically, we evaluate 13 MLLMs, revealing that even the best models show significant room for improvement, with most achieving only moderate results.Our experiments reveal that HVSBench presents a new and significant challenge for cutting-edge MLLMs.We believe that HVSBench will facilitate research on human-aligned and explainable MLLMs, marking a key step in understanding how MLLMs perceive and process visual information.

link

This study seeks to automate camera movement control for filming existing subjects into attractive videos, contrasting with the creation of non-existent content by directly generating the pixels.We select drone videos as our test case due to their rich and challenging motion patterns, distinctive viewing angles, and precise controls.Existing AI videography methods struggle with limited appearance diversity in simulation training, high costs of recording expert operations, and difficulties in designing heuristic-based goals to cover all scenarios.To avoid these issues, we propose a scalable method that involves collecting real-world training data to improve diversity, extracting camera trajectories automatically to minimize annotation costs, and training an effective architecture that does not rely on heuristics.Specifically, we collect 99k high-quality trajectories by running 3D reconstruction on online videos, connecting camera poses from consecutive frames to formulate 3D camera paths, and using Kalman filter to identify and remove low-quality data.Moreover, we introduce DVGFormer, an auto-regressive transformer that leverages the camera path and images from all past frames to predict camera movement in the next frame.We evaluate our system across 38 synthetic natural scenes and 7 real city 3D scans.We show that our system effectively learns to perform challenging camera movements such as navigating through obstacles, maintaining low altitude to increase perceived speed, and orbiting towers and buildings, which are very useful for recording high-quality videos.Data and code are available at dvgformer.github.io. 0.714

link

As virtual reality gains popularity, the demand for controllable creation of immersive and dynamic omnidirectional videos (ODVs) is increasing.While previous text-to-ODV generation methods achieve impressive results, they struggle with content inaccuracies and inconsistencies due to reliance solely on textual inputs.Although recent motion control techniques provide fine-grained control for video generation, directly applying these methods to ODVs often results in spatial distortion and unsatisfactory performance, especially with complex spherical motions.To tackle these challenges, we propose OmniDrag, the first approach enabling both scene- and object-level motion control for accurate, high-quality omnidirectional image-to-video generation.Building on pretrained video diffusion models, we introduce an omnidirectional control module, which is jointly fine-tuned with temporal attention layers to effectively handle complex spherical motion.In addition, we develop a novel spherical motion estimator that accurately extracts motion-control signals and allows users to perform drag-style ODV generation by simply drawing handle and target points.We also present a new dataset, named Move360, addressing the scarcity of ODV data with large scene and object motions. 0.834Experiments demonstrate the significant superiority of OmniDrag in achieving holistic scene-level and fine-grained object-level control for ODV generation.The project page is available at https://lwq20020127.github.io/OmniDrag.

link

Tobacco3482 is a widely used document classification benchmark dataset.However, our manual inspection of the entire dataset uncovers widespread ontological issues, especially large amounts of annotation label problems in the dataset. 0.665We establish data label guidelines and find that 11.7% of the dataset is improperly annotated and should either have an unknown label or a corrected label, and 16.7% of samples in the dataset have multiple valid labels. 0.812We then analyze the mistakes of a top-performing model and find that 35% of the model's mistakes can be directly attributed to these label issues, highlighting the inherent problems with using a noisily labeled dataset as a benchmark. 0.747Supplementary material, including dataset annotations and code, is available at https://github.com/gordon-lim/tobacco3482-mistakes/.

link

Face recognition has made remarkable strides, driven by the expanding scale of datasets, advancements in various backbone and discriminative losses.However, face recognition performance is heavily affected by the label noise, especially closed-set noise.While numerous studies have focused on handling label noise, addressing closed-set noise still poses challenges. 0.769This paper identifies this challenge as training isn't robust to noise at the early-stage training, and necessitating an appropriate learning strategy for samples with low confidence, which are often misclassified as closed-set noise in later training phases.To address these issues, we propose a new framework to stabilize the training at early stages and split the samples into clean, ambiguous and noisy groups which are devised with separate training strategies.Initially, we employ generated auxiliary closed-set noisy samples to enable the model to identify noisy data at the early stages of training.Subsequently, we introduce how samples are split into clean, ambiguous and noisy groups by their similarity to the positive and nearest negative centers.Then we perform label fusion for ambiguous samples by incorporating accumulated model predictions. 0.602Finally, we apply label smoothing within the closed set, adjusting the label to a point between the nearest negative class and the initially assigned label. 0.624Extensive experiments validate the effectiveness of our method on mainstream face datasets, achieving state-of-the-art results.The code will be released upon acceptance.

link

Domain Generalization (DG) seeks to transfer knowledge from multiple source domains to unseen target domains, even in the presence of domain shifts.Achieving effective generalization typically requires a large and diverse set of labeled source data to learn robust representations that can generalize to new, unseen domains.However, obtaining such high-quality labeled data is often costly and labor-intensive, limiting the practical applicability of DG.To address this, we investigate a more practical and challenging problem: semi-supervised domain generalization (SSDG) under a label-efficient paradigm.In this paper, we propose a novel method, CAT, which leverages semi-supervised learning with limited labeled data to achieve competitive generalization performance under domain shifts.Our method addresses key limitations of previous approaches, such as reliance on fixed thresholds and sensitivity to noisy pseudo-labels. 0.602CAT combines adaptive thresholding with noisy label refinement techniques, creating a straightforward yet highly effective solution for SSDG tasks.Specifically, our approach uses flexible thresholding to generate high-quality pseudo-labels with higher class diversity while refining noisy pseudo-labels to improve their reliability. 0.679Extensive experiments across multiple benchmark datasets demonstrate the superior performance of our method, highlighting its effectiveness in achieving robust generalization under domain shift.

link

Model inversion attacks pose a significant privacy threat to machine learning models by reconstructing sensitive data from their outputs.While various defenses have been proposed to counteract these attacks, they often come at the cost of the classifier's utility, thus creating a challenging trade-off between privacy protection and model utility.Moreover, most existing defenses require retraining the classifier for enhanced robustness, which is impractical for large-scale, well-established models.This paper introduces a novel defense mechanism to better balance privacy and utility, particularly against adversaries who employ a machine learning model (i.e., inversion model) to reconstruct private data.Drawing inspiration from data poisoning attacks, which can compromise the performance of machine learning models, we propose a strategy that leverages data poisoning to contaminate the training data of inversion models, thereby preventing model inversion attacks. Two defense methods are presented.The first, termed label-preserving poisoning attacks for all output vectors (LPA), involves subtle perturbations to all output vectors while preserving their labels.Our findings demonstrate that these minor perturbations, introduced through a data poisoning approach, significantly increase the difficulty of data reconstruction without compromising the utility of the classifier. 0.602Subsequently, we introduce a second method, label-flipping poisoning for partial output vectors (LFP), which selectively perturbs a small subset of output vectors and alters their labels during the process.Empirical results indicate that LPA is notably effective, outperforming the current state-of-the-art defenses.Our data poisoning-based defense provides a new retraining-free defense paradigm that preserves the victim classifier's utility.

link

We introduce a new framework for analyzing classification datasets based on the ratios of reconstruction errors between autoencoders trained on individual classes.This analysis framework enables efficient characterization of datasets on the sample, class, and entire dataset levels.We define reconstruction error ratios (RERs) that probe classification difficulty and allow its decomposition into (1) finite sample size and (2) Bayes error and decision-boundary complexity.Through systematic study across 19 popular visual datasets, we find that our RER-based dataset difficulty probe strongly correlates with error rate for state-of-the-art (SOTA) classification models.By interpreting sample-level classification difficulty as a label mistakenness score, we further find that RERs achieve SOTA performance on mislabel detection tasks on hard datasets under symmetric and asymmetric label noise. 0.664Our code is publicly available at https://github.com/voxel51/reconstruction-error-ratios.

link

We study the problem of guarding the boundary of a simple polygon with a minimum number of guards such that each guard covers a contiguous portion of the boundary.First, we present a simple greedy algorithm for this problem that returns a guard set of size at most OPT + 1, where OPT is the number of guards in an optimal solution.Then, we present a polynomial-time exact algorithm.While the algorithm is not complicated, its correctness proof is rather involved. 0.604This result is interesting in the sense that guarding problems are typically NP-hard and, in particular, it is NP-hard to minimize the number of guards to see the boundary of a simple polygon, without the contiguous boundary guarding constraint. From the combinatorial point of view, we show that any $n$-vertex polygon can be guarded by at most $\lfloor \frac{n-2}{2}\rfloor$ guards.This bound is tight because there are polygons that require this many guards.

link

In the realm of Text-Based Person Search (TBPS), mainstream methods aim to explore more efficient interaction frameworks between text descriptions and visual data.However, recent approaches encounter two principal challenges.Firstly, the widely used random-based Masked Language Modeling (MLM) considers all the words in the text equally during training.However, massive semantically vacuous words ('with', 'the', etc.) be masked fail to contribute efficient interaction in the cross-modal MLM and hampers the representation alignment.Secondly, manual descriptions in TBPS datasets are tedious and inevitably contain several inaccuracies.To address these issues, we introduce an Attention-Guided Alignment (AGA) framework featuring two innovative components: Attention-Guided Mask (AGM)Modeling and Text Enrichment Module (TEM).AGM dynamically masks semantically meaningful words by aggregating the attention weight derived from the text encoding process, thereby cross-modal MLM can capture information related to the masked word from text context and images and align their representations.Meanwhile, TEM alleviates low-quality representations caused by repetitive and erroneous text descriptions by replacing those semantically meaningful words with MLM's prediction.It not only enriches text descriptions but also prevents overfitting.Extensive experiments across three challenging benchmarks demonstrate the effectiveness of our AGA, achieving new state-of-the-art results with Rank-1 accuracy reaching 78.36%, 67.31%, and 67.4% on CUHK-PEDES, ICFG-PEDES, and RSTPReid, respectively. 0.679

link

The all pairs shortest path problem is a fundamental optimization problem in graph theory.We deal with re-calculating the all-pairs shortest path (APSP) matrix after a minor modification of a weighted dense graph, e.g., adding a node, removing a node, or updating an edge.We assume the APSP matrix for the original graph is already known.The graph can be directed or undirected.A cold-start calculation of the new APSP matrix by traditional algorithms, like the Floyd-Warshall algorithm or Dijkstra's algorithm, needs $ O(n^3) $ time.We propose two algorithms for warm-start calculation of the new APSP matrix.The best case complexity for a warm-start calculation is $ O(n^2) $, the worst case complexity is $ O(n^3) $. 0.64We implemented the algorithms and tested their performance with experiments. 0.725The result shows a warm-start calculation can save a great portion of calculation time, compared with cold-start calculation. 0.685

link

Fully Homomorphic Encryption (FHE) enables operations on encrypted data, making it extremely useful for privacy-preserving applications, especially in cloud computing environments.In such contexts, operations like ranking, order statistics, and sorting are fundamental functionalities often required for database queries or as building blocks of larger protocols.However, the high computational overhead and limited native operations of FHE pose significant challenges for an efficient implementation of these tasks.These challenges are exacerbated by the fact that all these functionalities are based on comparing elements, which is a severely expensive operation under encryption. Previous solutions have typically based their designs on swap-based techniques, where two elements are conditionally swapped based on the results of their comparison. 0.611These methods aim to reduce the primary computational bottleneck: the comparison depth, which is the number of non-parallelizable homomorphic comparisons.The current state of the art solution for sorting by Lu et al. (IEEE S&P'21), for instance, achieves a comparison depth of O(log^2(N)). 0.671In this paper, we address the challenge of reducing the comparison depth by shifting away from the swap-based paradigm.We present solutions for ranking, order statistics, and sorting, that all achieve a comparison depth of O(1), making our approach highly parallelizable. 0.65Leveraging the SIMD capabilities of the CKKS FHE scheme, our approach re-encodes the input vector under encryption to allow for simultaneous comparisons of all elements with each other.The homomorphic re-encoding incurs a minimal computational overhead of O(log(N))rotations.Experimental results show that our approach ranks a 128-element vector in approximately 2.64s, computes its argmin/argmax in 14.18s, and sorts it in 21.10s.

link

The recent advancements in large language models (LLMs) have significantly improved language understanding and generation capabilities.However, it is difficult to deploy LLMs on resource-constrained edge devices due to their high computational and storage resource demands.To address this issue, we propose a novel LLM model pruning method, namely structurally-aware adaptive pruning (SAAP), to significantly reduce the computational and memory costs while maintaining model performance.We first define an adaptive importance fusion metric to evaluate the importance of all coupled structures in LLMs by considering their homoscedastic uncertainty.Then, we rank the importance of all modules to determine the specific layers that should be pruned to meet particular performance requirements.Furthermore, we develop a new group fine-tuning strategy to improve the inference efficiency of LLMs.Finally, we evaluate the proposed SAAP method on multiple LLMs across two common tasks, i.e., zero-shot classification and text generation.Experimental results show that our SAAP method outperforms several state-of-the-art baseline methods, achieving 2.17%, 2.37%, and 2.39% accuracy gains on LLaMA-7B, Vicuna-7B, and 0.738LLaMA-13B. Additionally, SAAP improves the token generation speed by 5%, showcasing its practical advantages in resource-constrained scenarios.

link

Large Language Models (LLMs) are increasingly employed in real-world applications, driving the need to evaluate the trustworthiness of their generated text.To this end, reliable uncertainty estimation is essential.Since current LLMs generate text autoregressively through a stochastic process, the same prompt can lead to varying outputs.Consequently, leading uncertainty estimation methods generate and analyze multiple output sequences to determine the LLM's uncertainty.However, generating output sequences is computationally expensive, making these methods impractical at scale.In this work, we inspect the theoretical foundations of the leading methods and explore new directions to enhance their computational efficiency. 0.633Building on the framework of proper scoring rules, we find that the negative log-likelihood of the most likely output sequence constitutes a theoretically grounded uncertainty measure.To approximate this alternative measure, we propose G-NLL, which has the advantage of being obtained using only a single output sequence generated by greedy decoding.This makes uncertainty estimation more efficient and straightforward, while preserving theoretical rigor.Empirical results demonstrate that G-NLL achieves state-of-the-art performance across various LLMs and tasks.Our work lays the foundation for efficient and reliable uncertainty estimation in natural language generation, challenging the necessity of more computationally involved methods currently leading the field.

link

Natural Language Processing and Generation systems have recently shown the potential to complement and streamline the costly and time-consuming job of professional fact-checkers.In this work, we lift several constraints of current state-of-the-art pipelines for automated fact-checking based on the Retrieval-Augmented Generation (RAG) paradigm.Our goal is to benchmark, under more realistic scenarios, RAG-based methods for the generation of verdicts - i.e., short texts discussing the veracity of a claim - evaluating them on stylistically complex claims and heterogeneous, yet reliable, knowledge bases. 0.617Our findings show a complex landscape, where, for example, LLM-based retrievers outperform other retrieval techniques, though they still struggle with heterogeneous knowledge bases; larger models excel in verdict faithfulness, while smaller models provide better context adherence, with human evaluations favouring zero-shot and one-shot approaches for informativeness, and fine-tuned models for emotional alignment.

link

Multiple-choice question (MCQ) datasets like Massive Multitask Language Understanding (MMLU) are widely used to evaluate the commonsense, understanding, and problem-solving abilities of large language models (LLMs).However, the open-source nature of these benchmarks and the broad sources of training data for LLMs have inevitably led to benchmark contamination, resulting in unreliable evaluation results.To alleviate this issue, we propose a contamination-free and more challenging MCQ benchmark called MMLU-CF.This benchmark reassesses LLMs' understanding of world knowledge by averting both unintentional and malicious data leakage.To avoid unintentional data leakage, we source data from a broader domain and design three decontamination rules.To prevent malicious data leakage, we divide the benchmark into validation and test sets with similar difficulty and subject distributions. 0.605The test set remains closed-source to ensure reliable results, while the validation set is publicly available to promote transparency and facilitate independent verification.Our evaluation of mainstream LLMs reveals that the powerful GPT-4o achieves merely a 5-shot score of 73.4% and a 0-shot score of 71.9% on the test set, which indicates the effectiveness of our approach in creating a more rigorous and contamination-free evaluation standard.The GitHub repository is available at https://github.com/microsoft/MMLU-CF and the dataset refers to https://huggingface.co/datasets/microsoft/MMLU-CF.

link

This paper targets the challenge of real-time LiDAR re-simulation in dynamic driving scenarios.Recent approaches utilize neural radiance fields combined with the physical modeling of LiDAR sensors to achieve high-fidelity re-simulation results.Unfortunately, these methods face limitations due to high computational demands in large-scale scenes and cannot perform real-time LiDAR rendering.To overcome these constraints, we propose LiDAR-RT, a novel framework that supports real-time, physically accurate LiDAR re-simulation for driving scenes.Our primary contribution is the development of an efficient and effective rendering pipeline, which integrates Gaussian primitives and hardware-accelerated ray tracing technology.Specifically, we model the physical properties of LiDAR sensors using Gaussian primitives with learnable parameters and incorporate scene graphs to handle scene dynamics.Building upon this scene representation, our framework first constructs a bounding volume hierarchy (BVH), then casts rays for each pixel and generates novel LiDAR views through a differentiable rendering algorithm.Importantly, our framework supports realistic rendering with flexible scene editing operations and various sensor configurations.Extensive experiments across multiple public benchmarks demonstrate that our method outperforms state-of-the-art methods in terms of rendering quality and efficiency. 0.708Our project page is at https://zju3dv.github.io/lidar-rt.

link

Recent advancements in large vision language models (VLMs) tailored for autonomous driving (AD) have shown strong scene understanding and reasoning capabilities, making them undeniable candidates for end-to-end driving systems.However, limited work exists on studying the trustworthiness of DriveVLMs -- a critical factor that directly impacts public transportation safety.In this paper, we introduce AutoTrust, a comprehensive trustworthiness benchmark for large vision-language models in autonomous driving (DriveVLMs), considering diverse perspectives -- including trustfulness, safety, robustness, privacy, and fairness.We constructed the largest visual question-answering dataset for investigating trustworthiness issues in driving scenarios, comprising over 10k unique scenes and 18k queries.We evaluated six publicly available VLMs, spanning from generalist to specialist, from open-source to commercial models.Our exhaustive evaluations have unveiled previously undiscovered vulnerabilities of DriveVLMs to trustworthiness threats.Specifically, we found that the general VLMs like LLaVA-v1.6 and GPT-4o-mini surprisingly outperform specialized models fine-tuned for driving in terms of overall trustworthiness.DriveVLMs like DriveLM-Agent are particularly vulnerable to disclosing sensitive information.Additionally, both generalist and specialist VLMs remain susceptible to adversarial attacks and struggle to ensure unbiased decision-making across diverse environments and populations.Our findings call for immediate and decisive action to address the trustworthiness of DriveVLMs -- an issue of critical importance to public safety and the welfare of all citizens relying on autonomous transportation systems.Our benchmark is publicly available at \url{https://github.com/taco-group/AutoTrust}, and the leaderboard is released at \url{https://taco-group.github.io/AutoTrust/}. 0.613

link

Despite significant advancements in Large Vision-Language Models (LVLMs), existing pixel-grounding models operate on single-image settings, limiting their ability to perform detailed, fine-grained comparisons across multiple images.Conversely, current multi-image understanding models lack pixel-level grounding.Our work addresses this gap by introducing the task of multi-image pixel-grounded reasoning segmentation, and PRIMA, a novel LVLM that integrates pixel-level grounding with robust multi-image reasoning capabilities to produce contextually rich, pixel-grounded explanations.Central to PRIMA is an efficient vision module that queries fine-grained visual representations across multiple images, reducing TFLOPs by $25.3\%$. To support training and evaluation, we curate $M^4Seg$, a new reasoning segmentation benchmark consisting of $\sim$224K question-answer pairs that require fine-grained visual understanding across multiple images.Experimental results demonstrate PRIMA outperforms state-of-the-art baselines. 0.759

link

We propose an unsupervised model for instruction-based image editing that eliminates the need for ground-truth edited images during training.Existing supervised methods depend on datasets containing triplets of input image, edited image, and edit instruction.These are generated by either existing editing methods or human-annotations, which introduce biases and limit their generalization ability.Our method addresses these challenges by introducing a novel editing mechanism called Cycle Edit Consistency (CEC), which applies forward and backward edits in one training step and enforces consistency in image and attention spaces.This allows us to bypass the need for ground-truth edited images and unlock training for the first time on datasets comprising either real image-caption pairs or image-caption-edit triplets.We empirically show that our unsupervised technique performs better across a broader range of edits with high fidelity and precision. 0.624By eliminating the need for pre-existing datasets of triplets, reducing biases associated with supervised methods, and proposing CEC, our work represents a significant advancement in unblocking scaling of instruction-based image editing.

link

Multimodal recommendation systems can learn users' preferences from existing user-item interactions as well as the semantics of multimodal data associated with items.Many existing methods model this through a multimodal user-item graph, approaching multimodal recommendation as a graph learning task.Graph Neural Networks (GNNs) have shown promising performance in this domain.Prior research has capitalized on GNNs' capability to capture neighborhood information within certain receptive fields (typically denoted by the number of hops, $K$) to enrich user and item semantics.We observe that the optimal receptive fields for GNNs can vary across different modalities.In this paper, we propose GNNs with Modality-Independent Receptive Fields, which employ separate GNNs with independent receptive fields for different modalities to enhance performance.Our results indicate that the optimal $K$ for certain modalities on specific datasets can be as low as 1 or 2, which may restrict the GNNs' capacity to capture global information.To address this, we introduce a Sampling-based Global Transformer, which utilizes uniform global sampling to effectively integrate global information for GNNs.We conduct comprehensive experiments that demonstrate the superiority of our approach over existing methods. 0.631Our code is publicly available at https://github.com/CrawlScript/MIG-GT.

link

Causal discovery is essential for understanding complex systems, yet traditional methods often depend on strong, untestable assumptions, making the process challenging.Large Language Models (LLMs) present a promising alternative for extracting causal insights from text-based metadata, which consolidates domain expertise.However, LLMs are prone to unreliability and hallucinations, necessitating strategies that account for their limitations.One such strategy involves leveraging a consistency measure to evaluate reliability. 0.6Additionally, most text metadata does not clearly distinguish direct causal relationships from indirect ones, further complicating the inference of causal graphs.As a result, focusing on causal orderings, rather than causal graphs, emerges as a more practical and robust approach.We propose a novel method to derive a distribution of acyclic tournaments (representing plausible causal orders) that maximizes a consistency score.Our approach begins by computing pairwise consistency scores between variables, yielding a cyclic tournament that aggregates these scores.From this structure, we identify optimal acyclic tournaments compatible with the original tournament, prioritizing those that maximize consistency across all configurations.We tested our method on both classical and well-established bechmarks, as well as real-world datasets from epidemiology and public health.Our results demonstrate the effectiveness of our approach in recovering distributions causal orders with minimal error.

link

We introduce a novel language for reasoning about agents' cognitive attitudes of both epistemic and motivational type.We interpret it by means of a computationally grounded semantics using belief bases.Our language includes five types of modal operators for implicit belief, complete attraction, complete repulsion, realistic attraction and realistic repulsion.We give an axiomatization and show that our operators are not mutually expressible and that they can be combined to represent a large variety of psychological concepts including ambivalence, indifference, being motivated, being demotivated and preference.We present a dynamic extension of the language that supports reasoning about the effects of belief change operations.Finally, we provide a succinct formulation of model checking for our languages and a PSPACE model checking algorithm relying on a reduction into TQBF.We present some experimental results for the implemented algorithm on computation time in a concrete example. 0.631

link

In this work, we consider an online robust Markov Decision Process (MDP) where we have the information of finitely many prototypes of the underlying transition kernel.We consider an adaptively updated ambiguity set of the prototypes and propose an algorithm that efficiently identifies the true underlying transition kernel while guaranteeing the performance of the corresponding robust policy.To be more specific, we provide a sublinear regret of the subsequent optimal robust policy.We also provide an early stopping mechanism and a worst-case performance bound of the value function.In numerical experiments, we demonstrate that our method outperforms existing approaches, particularly in the early stage with limited data. 0.681This work contributes to robust MDPs by considering possible prior information about the underlying transition probability and online learning, offering both theoretical insights and practical algorithms for improved decision-making under uncertainty.

link

This work develops a compute-efficient algorithm to tackle a fundamental problem in transportation: that of urban travel demand estimation.It focuses on the calibration of origin-destination travel demand input parameters for high-resolution traffic simulation models.It considers the use of abundant traffic road speed data.The travel demand calibration problem is formulated as a continuous, high-dimensional, simulation-based optimization (SO) problem with bound constraints.There is a lack of compute efficient algorithms to tackle this problem.We propose the use of an SO algorithm that relies on an efficient, analytical, differentiable, physics-based traffic model, known as a metamodel or surrogate model.We formulate a metamodel that enables the use of road speed data.Tests are performed on a Salt Lake City network.We study how the amount of data, as well as the congestion levels, impact both in-sample and out-of-sample performance.The proposed method outperforms the benchmark for both in-sample and out-of-sample performance by 84.4% and 72.2% in terms of speeds and counts, respectively. 0.729Most importantly, the proposed method yields the highest compute efficiency, identifying solutions with good performance within few simulation function evaluations (i.e., with small samples). 0.693

link

Modern challenges of robustness, fairness, and decision-making in machine learning have led to the formulation of multi-distribution learning (MDL) frameworks in which a predictor is optimized across multiple distributions.We study the calibration properties of MDL to better understand how the predictor performs uniformly across the multiple distributions.Through classical results on decomposing proper scoring losses, we first derive the Bayes optimal rule for MDL, demonstrating that it maximizes the generalized entropy of the associated loss function.Our analysis reveals that while this approach ensures minimal worst-case loss, it can lead to non-uniform calibration errors across the multiple distributions and there is an inherent calibration-refinement trade-off, even at Bayes optimality. 0.615Our results highlight a critical limitation: despite the promise of MDL, one must use caution when designing predictors tailored to multiple distributions so as to minimize disparity.

link

Despite empirical risk minimization (ERM) is widely applied in the machine learning community, its performance is limited on data with spurious correlation or subpopulation that is introduced by hidden attributes.Existing literature proposed techniques to maximize group-balanced or worst-group accuracy when such correlation presents, yet, at the cost of lower average accuracy. 0.658In addition, many existing works conduct surveys on different subpopulation methods without revealing the inherent connection between these methods, which could hinder the technology advancement in this area.In this paper, we identify important sampling as a simple yet powerful tool for solving the subpopulation problem.On the theory side, we provide a new systematic formulation of the subpopulation problem and explicitly identify the assumptions that are not clearly stated in the existing works.This helps to uncover the cause of the dropped average accuracy.We provide the first theoretical discussion on the connections of existing methods, revealing the core components that make them different.On the application side, we demonstrate a single estimator is enough to solve the subpopulation problem.In particular, we introduce the estimator in both attribute-known and -unknown scenarios in the subpopulation setup, offering flexibility in practical use cases.And empirically, we achieve state-of-the-art performance on commonly used benchmark datasets. 0.704

link

As a typical and practical application of Large Language Models (LLMs), Retrieval-Augmented Generation (RAG) techniques have gained extensive attention, particularly in vertical domains where LLMs may lack domain-specific knowledge.In this paper, we introduce an omnidirectional and automatic RAG benchmark, OmniEval, in the financial domain. 0.657Our benchmark is characterized by its multi-dimensional evaluation framework, including (1) a matrix-based RAG scenario evaluation system that categorizes queries into five task classes and 16 financial topics, leading to a structured assessment of diverse query scenarios; (2) a multi-dimensional evaluation data generation approach, which combines GPT-4-based automatic generation and human annotation, achieving an 87.47\% acceptance ratio in human evaluations on generated instances; (3) a multi-stage evaluation system that evaluates both retrieval and generation performance, result in a comprehensive evaluation on the RAG pipeline; and (4) robust evaluation metrics derived from rule-based and LLM-based ones, enhancing the reliability of assessments through manual annotations and supervised fine-tuning of an LLM evaluator.Our experiments demonstrate the comprehensiveness of OmniEval, which includes extensive test datasets and highlights the performance variations of RAG systems across diverse topics and tasks, revealing significant opportunities for RAG models to improve their capabilities in vertical domains.We open source the code of our benchmark in \href{https://github.com/RUC-NLPIR/OmniEval}{https://github.com/RUC-NLPIR/OmniEval}. 0.715

link

The deployment of machine learning models in operational contexts represents a significant investment for any organisation.Consequently, the risk of these models being misappropriated by competitors needs to be addressed.In recent years, numerous proposals have been put forth to detect instances of model stealing.However, these proposals operate under implicit and disparate data and model access assumptions; as a consequence, it remains unclear how they can be effectively compared to one another.Our evaluation shows that a simple baseline that we introduce performs on par with existing state-of-the-art fingerprints, which, on the other hand, are much more complex. 0.601To uncover the reasons behind this intriguing result, this paper introduces a systematic approach to both the creation of model fingerprinting schemes and their evaluation benchmarks.By dividing model fingerprinting into three core components -- Query, Representation and Detection (QuRD) -- we are able to identify $\sim100$ previously unexplored QuRD combinations and gain insights into their performance.Finally, we introduce a set of metrics to compare and guide the creation of more representative model stealing detection benchmarks.Our approach reveals the need for more challenging benchmarks and a sound comparison with baselines. 0.709To foster the creation of new fingerprinting schemes and benchmarks, we open-source our fingerprinting toolbox.

link

The increasing prevalence of compact UAVs has introduced significant risks to public safety, while traditional drone detection systems are often bulky and costly.To address these challenges, we present TAME, the Temporal Audio-based Mamba for Enhanced Drone Trajectory Estimation and Classification.This innovative anti-UAV detection model leverages a parallel selective state-space model to simultaneously capture and learn both the temporal and spectral features of audio, effectively analyzing propagation of sound.To further enhance temporal features, we introduce a Temporal Feature Enhancement Module, which integrates spectral features into temporal data using residual cross-attention.This enhanced temporal information is then employed for precise 3D trajectory estimation and classification.Our model sets a new standard of performance on the MMUAD benchmarks, demonstrating superior accuracy and effectiveness. 0.657The code and trained models are publicly available on GitHub \url{https://github.com/AmazingDay1/TAME}.

link

CNNs have become one of the most commonly used computational tool in the past two decades.One of the primary downsides of CNNs is that they work as a ``black box", where the user cannot necessarily know how the image data are analyzed, and therefore needs to rely on empirical evaluation to test the efficacy of a trained CNN.This can lead to hidden biases that affect the performance evaluation of neural networks, but are difficult to identify.Here we discuss examples of such hidden biases in common and widely used benchmark datasets, and propose techniques for identifying dataset biases that can affect the standard performance evaluation metrics. 0.667One effective approach to identify dataset bias is to perform image classification by using merely blank background parts of the original images.However, in some situations a blank background in the images is not available, making it more difficult to separate foreground or contextual information from the bias.To overcome this, we propose a method to identify dataset bias without the need to crop background information from the images.That method is based on applying several image transforms to the original images, including Fourier transform, wavelet transforms, median filter, and their combinations.These transforms were applied to recover background bias information that CNNs use to classify images.This transformations affect the contextual visual information in a different manner than it affects the systemic background bias.Therefore, the method can distinguish between contextual information and the bias, and alert on the presence of background bias even without the need to separate sub-images parts from the blank background of the original images.Code used in the experiments is publicly available.

link

Biometric systems are widely used for identity verification and identification, including authentication (i.e., one-to-one matching to verify a claimed identity) and identification (i.e., one-to-many matching to find a subject in a database).The matching process relies on measuring similarities or dissimilarities between a fresh biometric template and enrolled templates.The False Match Rate FMR is a key metric for assessing the accuracy and reliability of such systems. 0.616This paper analyzes biometric systems based on their FMR, with two main contributions.First, we explore untargeted attacks, where an adversary aims to impersonate any user within a database.We determine the number of trials required for an attacker to successfully impersonate a user and derive the critical population size (i.e., the maximum number of users in the database) required to maintain a given level of security.Furthermore, we compute the critical FMR value needed to ensure resistance against untargeted attacks as the database size increases.Second, we revisit the biometric birthday problem to evaluate the approximate and exact probabilities that two users in a database collide (i.e., can impersonate each other).Based on this analysis, we derive both the approximate critical population size and the critical FMR value needed to bound the likelihood of such collisions occurring with a given probability.These thresholds offer insights for designing systems that mitigate the risk of impersonation and collisions, particularly in large-scale biometric databases.Our findings indicate that current biometric systems fail to deliver sufficient accuracy to achieve an adequate security level against untargeted attacks, even in small-scale databases.Moreover, state-of-the-art systems face significant challenges in addressing the biometric birthday problem, especially as database sizes grow.

link

Evaluation plays a crucial role in the advancement of information retrieval (IR) models.However, current benchmarks, which are based on predefined domains and human-labeled data, face limitations in addressing evaluation needs for emerging domains both cost-effectively and efficiently.To address this challenge, we propose the Automated Heterogeneous Information Retrieval Benchmark (AIR-Bench).AIR-Bench is distinguished by three key features: 1) Automated.The testing data in AIR-Bench is automatically generated by large language models (LLMs) without human intervention.2) Heterogeneous.The testing data in AIR-Bench is generated with respect to diverse tasks, domains and languages.3) Dynamic.The domains and languages covered by AIR-Bench are constantly augmented to provide an increasingly comprehensive evaluation benchmark for community developers. 0.617We develop a reliable and robust data generation pipeline to automatically create diverse and high-quality evaluation datasets based on real-world corpora.Our findings demonstrate that the generated testing data in AIR-Bench aligns well with human-labeled testing data, making AIR-Bench a dependable benchmark for evaluating IR models.The resources in AIR-Bench are publicly available at https://github.com/AIR-Bench/AIR-Bench.

link

Various evaluation metrics have been proposed for Grammatical Error Correction (GEC), but many, particularly reference-free metrics, lack explainability. 0.603This lack of explainability hinders researchers from analyzing the strengths and weaknesses of GEC models and limits the ability to provide detailed feedback for users.To address this issue, we propose attributing sentence-level scores to individual edits, providing insight into how specific corrections contribute to the overall performance.For the attribution method, we use Shapley values, from cooperative game theory, to compute the contribution of each edit.Experiments with existing sentence-level metrics demonstrate high consistency across different edit granularities and show approximately 70\% alignment with human evaluations. 0.612In addition, we analyze biases in the metrics based on the attribution results, revealing trends such as the tendency to ignore orthographic edits.Our implementation is available at \url{https://github.com/naist-nlp/gec-attribute}.

link

Online planning under uncertainty remains a critical challenge in robotics and autonomous systems.While tree search techniques are commonly employed to construct partial future trajectories within computational constraints, most existing methods discard information from previous planning sessions considering continuous spaces.This study presents a novel, computationally efficient approach that leverages historical planning data in current decision-making processes.We provide theoretical foundations for our information reuse strategy and introduce an algorithm based on Monte Carlo Tree Search (MCTS) that implements this approach.Experimental results demonstrate that our method significantly reduces computation time while maintaining high performance levels. 0.695Our findings suggest that integrating historical planning information can substantially improve the efficiency of online decision-making in uncertain environments, paving the way for more responsive and adaptive autonomous systems.

link

Studies have shown that toxic behavior can cause contributors to leave, and hinder newcomers' (especially from underrepresented communities) participation in Open Source Software (OSS) projects.Thus, detection of toxic language plays a crucial role in OSS collaboration and inclusivity.Off-the-shelf toxicity detectors are ineffective when applied to OSS communications, due to the distinct nature of toxicity observed in these channels (e.g., entitlement and arrogance are more frequently observed on GitHub than on Reddit or Twitter).In this paper, we investigate a machine learning-based approach for the automatic detection of toxic communications in OSS.We leverage psycholinguistic lexicons, and Moral Foundations Theory to analyze toxicity in two types of OSS communication channels; issue comments and code reviews.Our evaluation indicates that our approach can achieve a significant performance improvement (up to 7% increase in F1 score) over the existing domain-specific toxicity detector. 0.603We found that using moral values as features is more effective than linguistic cues, resulting in 67.50% F1-measure in identifying toxic instances in code review data and 64.83% in issue comments.While the detection accuracy is far from accurate, this improvement demonstrates the potential of integrating moral and psycholinguistic features in toxicity detection models.These findings highlight the importance of context-specific models that consider the unique communication styles within OSS, where interpersonal and value-driven language dynamics differ markedly from general social media platforms.Future work could focus on refining these models to further enhance detection accuracy, possibly by incorporating community-specific norms and conversational context to better capture the nuanced expressions of toxicity in OSS environments.

link

The rapid advancement of Large Language Models (LLMs) has demonstrated remarkable progress in complex reasoning tasks.However, a significant discrepancy persists between benchmark performances and real-world applications. 0.674We identify this gap as primarily stemming from current evaluation protocols and metrics, which inadequately capture the full spectrum of LLM capabilities, particularly in complex reasoning tasks where both accuracy and consistency are crucial.This work makes two key contributions.First, we introduce G-Pass@k, a novel evaluation metric that provides a continuous assessment of model performance across multiple sampling attempts, quantifying both the model's peak performance potential and its stability. 0.628Second, we present LiveMathBench, a dynamic benchmark comprising challenging, contemporary mathematical problems designed to minimize data leakage risks during evaluation.Through extensive experiments using G-Pass@k on state-of-the-art LLMs with LiveMathBench, we provide comprehensive insights into both their maximum capabilities and operational consistency.Our findings reveal substantial room for improvement in LLMs' "realistic" reasoning capabilities, highlighting the need for more robust evaluation methods.The benchmark and detailed results are available at: https://github.com/open-compass/GPassK.

link

Robustness and generalizability in medical image segmentation are often hindered by scarcity and limited diversity of training data, which stands in contrast to the variability encountered during inference.While conventional strategies -- such as domain-specific augmentation, specialized architectures, and tailored training procedures -- can alleviate these issues, they depend on the availability and reliability of domain knowledge.When such knowledge is unavailable, misleading, or improperly applied, performance may deteriorate.In response, we introduce a novel, domain-agnostic, add-on, and data-driven strategy inspired by image stacking in image denoising.Termed ``semantic stacking,'' our method estimates a denoised semantic representation that complements the conventional segmentation loss during training.This method does not depend on domain-specific assumptions, making it broadly applicable across diverse image modalities, model architectures, and augmentation techniques.Through extensive experiments, we validate the superiority of our approach in improving segmentation performance under diverse conditions. 0.601Code is available at https://github.com/ymp5078/Semantic-Stacking.

link

In many data-driven decision-making problems, performance guarantees often depend heavily on the correctness of model assumptions, which may frequently fail in practice.We address this issue in the context of a feature-based newsvendor problem, where demand is influenced by observed features such as demographics and seasonality.To mitigate the impact of model misspecification, we propose a model-free and distribution-free framework inspired by conformal prediction.Our approach consists of two phases: a training phase, which can utilize any type of prediction method, and a calibration phase that conformalizes the model bias.To enhance predictive performance, we explore the balance between data quality and quantity, recognizing the inherent trade-off: more selective training data improves quality but reduces quantity.Importantly, we provide statistical guarantees for the conformalized critical quantile, independent of the correctness of the underlying model.Moreover, we quantify the confidence interval of the critical quantile, with its width decreasing as data quality and quantity improve.We validate our framework using both simulated data and a real-world dataset from the Capital Bikeshare program in Washington, D.C.Across these experiments, our proposed method consistently outperforms benchmark algorithms, reducing newsvendor loss by up to 40% on the simulated data and 25% on the real-world dataset. 0.657

link

With the integration of large language models (LLMs), embodied agents have strong capabilities to execute complicated instructions in natural language, paving a way for the potential deployment of embodied robots.However, a foreseeable issue is that those embodied agents can also flawlessly execute some hazardous tasks, potentially causing damages in real world.To study this issue, we present SafeAgentBench -- a new benchmark for safety-aware task planning of embodied LLM agents.SafeAgentBench includes: (1) a new dataset with 750 tasks, covering 10 potential hazards and 3 task types; (2) SafeAgentEnv, a universal embodied environment with a low-level controller, supporting multi-agent execution with 17 high-level actions for 8 state-of-the-art baselines; and (3) reliable evaluation methods from both execution and semantic perspectives.Experimental results show that the best-performing baseline gets 69% success rate for safe tasks, but only 5% rejection rate for hazardous tasks, indicating significant safety risks. 0.616More details and codes are available at https://github.com/shengyin1224/SafeAgentBench.

link

Real-time free-view human rendering from sparse-view RGB inputs is a challenging task due to the sensor scarcity and the tight time budget.To ensure efficiency, recent methods leverage 2D CNNs operating in texture space to learn rendering primitives.However, they either jointly learn geometry and appearance, or completely ignore sparse image information for geometry estimation, significantly harming visual quality and robustness to unseen body poses.To address these issues, we present Double Unprojected Textures, which at the core disentangles coarse geometric deformation estimation from appearance synthesis, enabling robust and photorealistic 4K rendering in real-time.Specifically, we first introduce a novel image-conditioned template deformation network, which estimates the coarse deformation of the human template from a first unprojected texture.This updated geometry is then used to apply a second and more accurate texture unprojection.The resulting texture map has fewer artifacts and better alignment with input views, which benefits our learning of finer-level geometry and appearance represented by Gaussian splats.We validate the effectiveness and efficiency of the proposed method in quantitative and qualitative experiments, which significantly surpasses other state-of-the-art methods. 0.676

link

Decision trees are a classic model for summarizing and classifying data.To enhance interpretability and generalization properties, it has been proposed to favor small decision trees.Accordingly, in the minimum-size decision tree training problem (MSDT), the input is a set of training examples in $\mathbb{R}^d$ with class labels and we aim to find a decision tree that classifies all training examples correctly and has a minimum number of nodes.MSDT is NP-hard and therefore presumably not solvable in polynomial time.Nevertheless, Komusiewicz et al.[ICML '23] developed a promising algorithmic paradigm called witness trees which solves MSDT efficiently if the solution tree is small.In this work, we test this paradigm empirically.We provide an implementation, augment it with extensive heuristic improvements, and scrutinize it on standard benchmark instances. 0.695The augmentations achieve a mean 324-fold (median 84-fold) speedup over the naive implementation.Compared to the state of the art they achieve a mean 32-fold (median 7-fold) speedup over the dynamic programming based MurTree solver[Demirovi\'c et al., J. Mach.Learn.Res. '22] and a mean 61-fold (median 25-fold) speedup over SAT-based implementations [Janota and Morgado, SAT '20]. 0.663As a theoretical result we obtain an improved worst-case running-time bound for MSDT.

link

The optimization-based meta-learning approach is gaining increased traction because of its unique ability to quickly adapt to a new task using only small amounts of data.However, existing optimization-based meta-learning approaches, such as MAML, ANIL and their variants, generally employ backpropagation for upper-level gradient estimation, which requires using historical lower-level parameters/gradients and thus increases computational and memory overhead in each iteration.In this paper, we propose a meta-learning algorithm that can avoid using historical parameters/gradients and significantly reduce memory costs in each iteration compared to existing optimization-based meta-learning approaches.In addition to memory reduction, we prove that our proposed algorithm converges sublinearly with the iteration number of upper-level optimization, and the convergence error decays sublinearly with the batch size of sampled tasks. 0.622In the specific case in terms of deterministic meta-learning, we also prove that our proposed algorithm converges to an exact solution.Moreover, we quantify that the computational complexity of the algorithm is on the order of $\mathcal{O}(\epsilon^{-1})$, which matches existing convergence results on meta-learning even without using any historical parameters/gradients.Experimental results on meta-learning benchmarks confirm the efficacy of our proposed algorithm. 0.682

link

Domain Generalized Semantic Segmentation (DGSS) seeks to utilize source domain data exclusively to enhance the generalization of semantic segmentation across unknown target domains.Prevailing studies predominantly concentrate on feature normalization and domain randomization, these approaches exhibit significant limitations.Feature normalization-based methods tend to confuse semantic features in the process of constraining the feature space distribution, resulting in classification misjudgment.Domain randomization-based methods frequently incorporate domain-irrelevant noise due to the uncontrollability of style transformations, resulting in segmentation ambiguity.To address these challenges, we introduce a novel framework, named SCSD for Semantic Consistency prediction and Style Diversity generalization.It comprises three pivotal components: Firstly, a Semantic Query Booster is designed to enhance the semantic awareness and discrimination capabilities of object queries in the mask decoder, enabling cross-domain semantic consistency prediction.Secondly, we develop a Text-Driven Style Transform module that utilizes domain difference text embeddings to controllably guide the style transformation of image features, thereby increasing inter-domain style diversity.Lastly, to prevent the collapse of similar domain feature spaces, we introduce a Style Synergy Optimization mechanism that fortifies the separation of inter-domain features and the aggregation of intra-domain features by synergistically weighting style contrastive loss and style aggregation loss.Extensive experiments demonstrate that the proposed SCSD significantly outperforms existing state-of-theart methods. 0.605Notably, SCSD trained on GTAV achieved an average of 49.11 mIoU on the four unseen domain datasets, surpassing the previous state-of-the-art method by +4.08 mIoU. Code is available at https://github.com/nhw649/SCSD.

link

Capturing geometric and material information from images remains a fundamental challenge in computer vision and graphics.Traditional optimization-based methods often require hours of computational time to reconstruct geometry, material properties, and environmental lighting from dense multi-view inputs, while still struggling with inherent ambiguities between lighting and material.On the other hand, learning-based approaches leverage rich material priors from existing 3D object datasets but face challenges with maintaining multi-view consistency.In this paper, we introduce IDArb, a diffusion-based model designed to perform intrinsic decomposition on an arbitrary number of images under varying illuminations.Our method achieves accurate and multi-view consistent estimation on surface normals and material properties.This is made possible through a novel cross-view, cross-domain attention module and an illumination-augmented, view-adaptive training strategy.Additionally, we introduce ARB-Objaverse, a new dataset that provides large-scale multi-view intrinsic data and renderings under diverse lighting conditions, supporting robust training.Extensive experiments demonstrate that IDArb outperforms state-of-the-art methods both qualitatively and quantitatively. 0.602Moreover, our approach facilitates a range of downstream tasks, including single-image relighting, photometric stereo, and 3D reconstruction, highlighting its broad applications in realistic 3D content creation.

link

Given the ubiquity of multi-task in practical systems, Multi-Task Learning (MTL) has found widespread application across diverse domains.In real-world scenarios, these tasks often have different priorities.For instance, In web search, relevance is often prioritized over other metrics, such as click-through rates or user engagement.Existing frameworks pay insufficient attention to the prioritization among different tasks, which typically adjust task-specific loss function weights to differentiate task priorities.However, this approach encounters challenges as the number of tasks grows, leading to exponential increases in hyper-parameter tuning complexity.Furthermore, the simultaneous optimization of multiple objectives can negatively impact the performance of high-priority tasks due to interference from lower-priority tasks. In this paper, we introduce a novel multi-task learning framework employing Lagrangian Differential Multiplier Methods for step-wise multi-task optimization.It is designed to boost the performance of high-priority tasks without interference from other tasks.Its primary advantage lies in its ability to automatically optimize multiple objectives without requiring balancing hyper-parameters for different tasks, thereby eliminating the need for manual tuning.Additionally, we provide theoretical analysis demonstrating that our method ensures optimization guarantees, enhancing the reliability of the process. 0.621We demonstrate its effectiveness through experiments on multiple public datasets and its application in Taobao search, a large-scale industrial search ranking system, resulting in significant improvements across various business metrics.

link

Determining the sustainability impact of companies is a highly complex subject which has garnered more and more attention over the past few years.Today, investors largely rely on sustainability-ratings from established rating-providers in order to analyze how responsibly a company acts.However, those ratings have recently been criticized for being hard to understand and nearly impossible to reproduce. An independent way to find out about the sustainability practices of companies lies in the rich landscape of news article data.In this paper, we explore a different approach to identify key opportunities and challenges of companies in the sustainability domain.We present a novel dataset of more than 840,000 news articles which were gathered for major German companies between January 2023 and September 2024.By applying a mixture of Natural Language Processing techniques, we first identify relevant articles, before summarizing them and extracting their sustainability-related sentiment and aspect using Large Language Models (LLMs).Furthermore, we conduct an evaluation of the obtained data and determine that the LLM-produced answers are accurate. 0.636We release both datasets at https://github.com/Bailefan/Nano-ESG.

link

Retrieve-augmented generation (RAG) frameworks have emerged as a promising solution to multi-hop question answering(QA) tasks since it enables large language models (LLMs) to incorporate external knowledge and mitigate their inherent knowledge deficiencies.Despite this progress, existing RAG frameworks, which usually follows the retrieve-then-read paradigm, often struggle with multi-hop QA with temporal information since it has difficulty retrieving and synthesizing accurate time-related information.To address the challenge, this paper proposes a novel framework called review-then-refine, which aims to enhance LLM performance in multi-hop QA scenarios with temporal information. 0.669Our approach begins with a review phase, where decomposed sub-queries are dynamically rewritten with temporal information, allowing for subsequent adaptive retrieval and reasoning process.In addition, we implement adaptive retrieval mechanism to minimize unnecessary retrievals, thus reducing the potential for hallucinations.In the subsequent refine phase, the LLM synthesizes the retrieved information from each sub-query along with its internal knowledge to formulate a coherent answer. 0.642Extensive experimental results across multiple datasets demonstrate the effectiveness of our proposed framework, highlighting its potential to significantly improve multi-hop QA capabilities in LLMs. 0.707

link

In the realm of Text-Based Person Search (TBPS), mainstream methods aim to explore more efficient interaction frameworks between text descriptions and visual data.However, recent approaches encounter two principal challenges.Firstly, the widely used random-based Masked Language Modeling (MLM) considers all the words in the text equally during training.However, massive semantically vacuous words ('with', 'the', etc.) be masked fail to contribute efficient interaction in the cross-modal MLM and hampers the representation alignment.Secondly, manual descriptions in TBPS datasets are tedious and inevitably contain several inaccuracies.To address these issues, we introduce an Attention-Guided Alignment (AGA) framework featuring two innovative components: Attention-Guided Mask (AGM)Modeling and Text Enrichment Module (TEM). 0.603AGM dynamically masks semantically meaningful words by aggregating the attention weight derived from the text encoding process, thereby cross-modal MLM can capture information related to the masked word from text context and images and align their representations.Meanwhile, TEM alleviates low-quality representations caused by repetitive and erroneous text descriptions by replacing those semantically meaningful words with MLM's prediction.It not only enriches text descriptions but also prevents overfitting.Extensive experiments across three challenging benchmarks demonstrate the effectiveness of our AGA, achieving new state-of-the-art results with Rank-1 accuracy reaching 78.36%, 67.31%, and 67.4% on CUHK-PEDES, ICFG-PEDES, and RSTPReid, respectively.

link

Large language models (LLMs) demonstrate an impressive ability to utilise information within the context of their input sequences to appropriately respond to data unseen by the LLM during its training procedure. 0.684This ability is known as in-context learning (ICL).Humans and non-human animals demonstrate similar abilities, however their neural architectures differ substantially from LLMs. 0.705Despite this, a critical component within LLMs, the attention mechanism, resembles modern associative memory models, widely used in and influenced by the computational neuroscience community to model biological memory systems. 0.68Using this connection, we introduce an associative memory model capable of performing ICL.We use this as inspiration for a novel residual stream architecture which allows information to directly flow between attention heads.We test this architecture during training within a two-layer Transformer and show its ICL abilities manifest more quickly than without this modification.We then apply our architecture in small language models with 8 million parameters, focusing on attention head values, with results also indicating improved ICL performance at this larger and more naturalistic scale.

link

In this report, we introduce Qwen2.5, a comprehensive series of large language models (LLMs) designed to meet diverse needs. 0.621Compared to previous iterations, Qwen 2.5 has been significantly improved during both the pre-training and post-training stages.In terms of pre-training, we have scaled the high-quality pre-training datasets from the previous 7 trillion tokens to 18 trillion tokens.This provides a strong foundation for common sense, expert knowledge, and reasoning capabilities.In terms of post-training, we implement intricate supervised finetuning with over 1 million samples, as well as multistage reinforcement learning.Post-training techniques enhance human preference, and notably improve long text generation, structural data analysis, and instruction following.To handle diverse and varied use cases effectively, we present Qwen2.5 LLM series in rich sizes. 0.689Open-weight offerings include base and instruction-tuned models, with quantized versions available.In addition, for hosted solutions, the proprietary models currently include two mixture-of-experts (MoE) variants: Qwen2.5-Turbo and Qwen2.5-Plus, both available from Alibaba Cloud Model Studio.Qwen2.5 has demonstrated top-tier performance on a wide range of benchmarks evaluating language understanding, reasoning, mathematics, coding, human preference alignment, etc.Specifically, the open-weight flagship Qwen2.5-72B-Instruct outperforms a number of open and proprietary models and demonstrates competitive performance to the state-of-the-art open-weight model, Llama-3-405B-Instruct, which is around 5 times larger.Qwen2.5-Turbo and Qwen2.5-Plus offer superior cost-effectiveness while performing competitively against GPT-4o-mini and GPT-4o respectively.Additionally, as the foundation, Qwen2.5 models have been instrumental in training specialized models such as Qwen2.5-Math, Qwen2.5-Coder, QwQ, and multimodal models.

link

The recent advancements in large language models (LLMs) have significantly improved language understanding and generation capabilities. 0.692However, it is difficult to deploy LLMs on resource-constrained edge devices due to their high computational and storage resource demands. 0.744To address this issue, we propose a novel LLM model pruning method, namely structurally-aware adaptive pruning (SAAP), to significantly reduce the computational and memory costs while maintaining model performance. 0.63We first define an adaptive importance fusion metric to evaluate the importance of all coupled structures in LLMs by considering their homoscedastic uncertainty. 0.645Then, we rank the importance of all modules to determine the specific layers that should be pruned to meet particular performance requirements.Furthermore, we develop a new group fine-tuning strategy to improve the inference efficiency of LLMs. 0.704Finally, we evaluate the proposed SAAP method on multiple LLMs across two common tasks, i.e., zero-shot classification and text generation. 0.678Experimental results show that our SAAP method outperforms several state-of-the-art baseline methods, achieving 2.17%, 2.37%, and 2.39% accuracy gains on LLaMA-7B, Vicuna-7B, andLLaMA-13B. Additionally, SAAP improves the token generation speed by 5%, showcasing its practical advantages in resource-constrained scenarios. 0.634

link

To manage exceptions, software relies on a key architectural guarantee, precision: that exceptions appear to execute between instructions.However, this definition, dating back over 60 years, fundamentally assumes a sequential programmers model.Modern architectures such as Arm-A with programmer-observable relaxed behaviour make such a naive definition inadequate, and it is unclear exactly what guarantees programmers have on exception entry and exit. In this paper, we clarify the concepts needed to discuss exceptions in the relaxed-memory setting -- a key aspect of precisely specifying the architectural interface between hardware and software.We explore the basic relaxed behaviour across exception boundaries, and the semantics of external aborts, using Arm-A as a representative modern architecture. 0.601We identify an important problem, present yet unexplored for decades: pinning down what it means for exceptions to be precise in a relaxed setting.We describe key phenomena that any definition should account for.We develop an axiomatic model for Arm-A precise exceptions, tooling for axiomatic model execution, and a library of tests.Finally we explore the relaxed semantics of software-generated interrupts, as used in sophisticated programming patterns, and sketch how they too could be modelled.

link

Large Language Models (LLMs) have demonstrated remarkable capabilities on various tasks, while the further evolvement is limited to the lack of high-quality training data. 0.686In addition, traditional training approaches rely too much on expert-labeled data, setting an upper limit on the performance of LLMs. 0.674To address this issue, we propose a novel paradigm that enables LLMs to train itself by autonomously generating, cleaning, reviewing, and annotating data with preference information, named LANCE. 0.737Our approach demonstrates that LLMs can serve as continuous self-evolving data engineers, significantly reducing the time and cost of the post-training data construction process. 0.708Through iterative fine-tuning on different variants of the Qwen2, we validate the effectiveness of LANCE across various tasks, showing that it can continuously improve model performance and maintain high-quality data generation.Across eight benchmark dimensions, LANCE resulted in an average score enhancement of 3.36 for Qwen2-7B and 2.70 for Qwen2-7B-Instruct.This training paradigm with autonomous data construction not only reduces the reliance on human experts or external models but also ensures that the data aligns with human values and preferences, paving the way for the development of future superintelligent systems that can exceed human capabilities.

link

Microswimmers are sub-millimeter swimming microrobots that show potential as a platform for controllable locomotion in applications including targeted cargo delivery and minimally invasive surgery. 0.605To be viable for these target applications, microswimmers will eventually need to be able to navigate in environments with dynamic fluid flows and forces. 0.634Experimental studies with microswimmers towards this goal are currently rare because of the difficulty isolating intentional microswimmer motion from environment-induced motion. 0.609In this work, we present a method for measuring microswimmer locomotion within a complex flow environment using fiducial microspheres.By tracking the particle motion of ferromagnetic and non-magnetic polystyrene fiducial microspheres, we capture the effect of fluid flow and field gradients on microswimmer trajectories.We then determine the field-driven translation of these microswimmers relative to fluid flow and demonstrate the effectiveness of this method by illustrating the motion of multiple microswimmers through different flows.

link

Text-to-video models have made remarkable advancements through optimization on high-quality text-video pairs, where the textual prompts play a pivotal role in determining quality of output videos.However, achieving the desired output often entails multiple revisions and iterative inference to refine user-provided prompts.Current automatic methods for refining prompts encounter challenges such as Modality-Inconsistency, Cost-Discrepancy, and Model-Unaware when applied to text-to-video diffusion models.To address these problem, we introduce an LLM-based prompt adaptation framework, termed as Prompt-A-Video, which excels in crafting Video-Centric, Labor-Free and Preference-Aligned prompts tailored to specific video diffusion model.Our approach involves a meticulously crafted two-stage optimization and alignment system.Initially, we conduct a reward-guided prompt evolution pipeline to automatically create optimal prompts pool and leverage them for supervised fine-tuning (SFT) of the LLM. 0.632Then multi-dimensional rewards are employed to generate pairwise data for the SFT model, followed by the direct preference optimization (DPO) algorithm to further facilitate preference alignment.Through extensive experimentation and comparative analyses, we validate the effectiveness of Prompt-A-Video across diverse generation models, highlighting its potential to push the boundaries of video generation.

link

Large Language Models (LLMs) are increasingly employed in real-world applications, driving the need to evaluate the trustworthiness of their generated text. 0.687To this end, reliable uncertainty estimation is essential.Since current LLMs generate text autoregressively through a stochastic process, the same prompt can lead to varying outputs. 0.724Consequently, leading uncertainty estimation methods generate and analyze multiple output sequences to determine the LLM's uncertainty.However, generating output sequences is computationally expensive, making these methods impractical at scale.In this work, we inspect the theoretical foundations of the leading methods and explore new directions to enhance their computational efficiency.Building on the framework of proper scoring rules, we find that the negative log-likelihood of the most likely output sequence constitutes a theoretically grounded uncertainty measure.To approximate this alternative measure, we propose G-NLL, which has the advantage of being obtained using only a single output sequence generated by greedy decoding.This makes uncertainty estimation more efficient and straightforward, while preserving theoretical rigor.Empirical results demonstrate that G-NLL achieves state-of-the-art performance across various LLMs and tasks. 0.675Our work lays the foundation for efficient and reliable uncertainty estimation in natural language generation, challenging the necessity of more computationally involved methods currently leading the field.

link

Studies have underscored how, regardless of the recent breakthrough and swift advances in AI research, even state-of-the-art Large Language models (LLMs) continue to struggle when performing logical and mathematical reasoning. 0.635The results seem to suggest that LLMs still work as (highly advanced) data pattern identifiers, scoring poorly when attempting to generalise and solve reasoning problems the models have never previously seen or that are not close to samples presented in their training data. 0.733To address this compelling concern, this paper makes use of the notion of critical questions from the literature on argumentation theory, focusing in particular on Toulmin's model of argumentation.We show that employing these critical questions can improve the reasoning capabilities of LLMs. 0.774By probing the rationale behind the models' reasoning process, the LLM can assess whether some logical mistake is occurring and correct it before providing the final reply to the user prompt. 0.721The underlying idea is drawn from the gold standard of any valid argumentative procedure: the conclusion is valid if it is entailed by accepted premises.Or, to paraphrase such Aristotelian principle in a real-world approximation, characterised by incomplete information and presumptive logic, the conclusion is valid if not proved otherwise.This approach successfully steers the models' output through a reasoning pipeline, resulting in better performance against the baseline and its Chain-of-Thought (CoT) implementation.To this end, an extensive evaluation of the proposed approach on the MT-Bench Reasoning and Math tasks across a range of LLMs is provided. 0.617

link

Large Language Model (LLM) based coding tools have been tremendously successful as software development assistants, yet they are often designed for general purpose programming tasks and perform poorly for more specialized domains such as high performance computing. 0.709Creating specialized models and tools for these domains is crucial towards gaining the benefits of LLMs in areas such as HPC. 0.723While previous work has explored HPC-specific models, LLMs still struggle to generate parallel code and it is not at all clear what hurdles are still holding back these LLMs and what must be done to overcome them. 0.734In this work, we conduct an in-depth study along the many axes of fine-tuning a specialized HPC LLM in order to better understand the challenges. 0.668Based on our findings we fine-tune and evaluate a specialized HPC LLM that is shown to be the best performing open-source code LLM for parallel code generation to date. 0.684

link

We present LlamaFusion, a framework for empowering pretrained text-only large language models (LLMs) with multimodal generative capabilities, enabling them to understand and generate both text and images in arbitrary sequences. 0.604LlamaFusion leverages existing Llama-3's weights for processing texts autoregressively while introducing additional and parallel transformer modules for processing images with diffusion. 0.609During training, the data from each modality is routed to its dedicated modules: modality-specific feedforward layers, query-key-value projections, and normalization layers process each modality independently, while the shared self-attention layers allow interactions across text and image features.By freezing the text-specific modules and only training the image-specific modules, LlamaFusion preserves the language capabilities of text-only LLMs while developing strong visual understanding and generation abilities. 0.735Compared to methods that pretrain multimodal generative models from scratch, our experiments demonstrate that, LlamaFusion improves image understanding by 20% and image generation by 3.6% using only 50% of the FLOPs while maintaining Llama-3's language capabilities. 0.618We also demonstrate that this framework can adapt existing vision-language models with multimodal generation ability.Overall, this framework not only leverages existing computational investments in text-only LLMs but also enables the parallel development of language and vision capabilities, presenting a promising direction for efficient multimodal model development. 0.651

link

Natural Language Processing and Generation systems have recently shown the potential to complement and streamline the costly and time-consuming job of professional fact-checkers.In this work, we lift several constraints of current state-of-the-art pipelines for automated fact-checking based on the Retrieval-Augmented Generation (RAG) paradigm.Our goal is to benchmark, under more realistic scenarios, RAG-based methods for the generation of verdicts - i.e., short texts discussing the veracity of a claim - evaluating them on stylistically complex claims and heterogeneous, yet reliable, knowledge bases.Our findings show a complex landscape, where, for example, LLM-based retrievers outperform other retrieval techniques, though they still struggle with heterogeneous knowledge bases; larger models excel in verdict faithfulness, while smaller models provide better context adherence, with human evaluations favouring zero-shot and one-shot approaches for informativeness, and fine-tuned models for emotional alignment. 0.626

link

Multiple-choice question (MCQ) datasets like Massive Multitask Language Understanding (MMLU) are widely used to evaluate the commonsense, understanding, and problem-solving abilities of large language models (LLMs). 0.611However, the open-source nature of these benchmarks and the broad sources of training data for LLMs have inevitably led to benchmark contamination, resulting in unreliable evaluation results. 0.642To alleviate this issue, we propose a contamination-free and more challenging MCQ benchmark called MMLU-CF.This benchmark reassesses LLMs' understanding of world knowledge by averting both unintentional and malicious data leakage. 0.669To avoid unintentional data leakage, we source data from a broader domain and design three decontamination rules.To prevent malicious data leakage, we divide the benchmark into validation and test sets with similar difficulty and subject distributions.The test set remains closed-source to ensure reliable results, while the validation set is publicly available to promote transparency and facilitate independent verification.Our evaluation of mainstream LLMs reveals that the powerful GPT-4o achieves merely a 5-shot score of 73.4% and a 0-shot score of 71.9% on the test set, which indicates the effectiveness of our approach in creating a more rigorous and contamination-free evaluation standard. 0.664The GitHub repository is available at https://github.com/microsoft/MMLU-CF and the dataset refers to https://huggingface.co/datasets/microsoft/MMLU-CF.

link

This paper introduces LongBench v2, a benchmark designed to assess the ability of LLMs to handle long-context problems requiring deep understanding and reasoning across real-world multitasks. 0.684LongBench v2 consists of 503 challenging multiple-choice questions, with contexts ranging from 8k to 2M words, across six major task categories: single-document QA, multi-document QA, long in-context learning, long-dialogue history understanding, code repository understanding, and long structured data understanding.To ensure the breadth and the practicality, we collect data from nearly 100 highly educated individuals with diverse professional backgrounds.We employ both automated and manual review processes to maintain high quality and difficulty, resulting in human experts achieving only 53.7% accuracy under a 15-minute time constraint.Our evaluation reveals that the best-performing model, when directly answers the questions, achieves only 50.1% accuracy.In contrast, the o1-preview model, which includes longer reasoning, achieves 57.7%, surpassing the human baseline by 4%.These results highlight the importance of enhanced reasoning ability and scaling inference-time compute to tackle the long-context challenges in LongBench v2.The project is available at https://longbench2.github.io.

link

Internship and industry-affiliated capstone projects are popular ways to expose students to real world experience and bridge the gap between academic training and industry requirements.However, these two approaches often require active industry collaboration and many students often struggle to find industry placements.Open-source contributions is a crucial alternative to gain real world experience, earn publicly verifiable contribution with real world impact, and learn from experienced open-source contributors.The Google Summer of Code is a global initiative that matches students or new contributors with experienced mentors to work on open-source projects.The goal of the program is to introduce the students to open-source, help gain valuable skills under the guidance of a mentor, and hopefully continue to contribute to open source development; thereby, provide a continuous pool of talented new contributors necessary for maintaining an open source project.This study presents an empirical analysis of pull requests created by interns during the Google Summer of Code program.We extracted and analysed 17,232 pull requests from 2456 interns across 1937 open-source projects.The results show that majority of the tasks involve both code-intensive tasks like adding new features and fixing bugs as well as non-code tasks like updating the documentation and restructuring the code base.The feedback from reviewers covers code functionality and programming logic, testing coverage, error handling, code readability, and adopting best practices. 0.654Finally, we discuss the implications of these results for software engineering education.

link

Visualization authoring is an iterative process requiring users to modify parameters like color schemes and data transformations to achieve desired aesthetics and effectively convey insights.Due to the complexity of these adjustments, users often create defective visualizations and require troubleshooting support. 0.609In this paper, we examine two primary approaches for visualization troubleshooting: (1) Human-assisted support via forums, where users receive advice from other individuals, and (2) AI-assisted support using large language models (LLMs).Our goal is to understand the strengths and limitations of each approach in supporting visualization troubleshooting tasks.To this end, we collected 889 Vega-Lite cases from Stack Overflow.We then conducted a comprehensive analysis to understand the types of questions users ask, the effectiveness of human and AI guidance, and the impact of supplementary resources, such as documentation and examples, on troubleshooting outcomes.Our findings reveal a striking contrast between human- and AI-assisted troubleshooting: Human-assisted troubleshooting provides tailored, context-sensitive advice but often varies in response quality, while AI-assisted troubleshooting offers rapid feedback but often requires additional contextual resources to achieve desired results.

link

Software defect prediction is a critical aspect of software quality assurance, as it enables early identification and mitigation of defects, thereby reducing the cost and impact of software failures. 0.648Over the past few years, quantum computing has risen as an exciting technology capable of transforming multiple domains; Quantum Machine Learning (QML) is one of them.QML algorithms harness the power of quantum computing to solve complex problems with better efficiency and effectiveness than their classical counterparts.However, research into its application in software engineering to predict software defects still needs to be explored.In this study, we worked to fill the research gap by comparing the performance of three QML and five classical machine learning (CML) algorithms on the 20 software defect datasets.Our investigation reports the comparative scenarios of QML vs. CML algorithms and identifies the better-performing and consistent algorithms to predict software defects.We also highlight the challenges and future directions of employing QML algorithms in real software defect datasets based on the experience we faced while performing this investigation.The findings of this study can help practitioners and researchers further progress in this research domain by making software systems reliable and bug-free.

link

Large language models (LLMs) like ChatGPT have shown the potential to assist developers with coding and debugging tasks.However, their role in collaborative issue resolution is underexplored.In this study, we analyzed 1,152 Developer-ChatGPT conversations across 1,012 issues in GitHub to examine the diverse usage of ChatGPT and reliance on its generated code.Our contributions are fourfold.First, we manually analyzed 289 conversations to understand ChatGPT's usage in the GitHub Issues.Our analysis revealed that ChatGPT is primarily utilized for ideation, whereas its usage for validation (e.g., code documentation accuracy) is minimal.Second, we applied BERTopic modeling to identify key areas of engagement on the entire dataset.We found that backend issues (e.g., API management) dominate conversations, while testing is surprisingly less covered.Third, we utilized the CPD clone detection tool to check if the code generated by ChatGPT was used to address issues.Our findings revealed that ChatGPT-generated code was used as-is to resolve only 5.83\% of the issues.Fourth, we estimated sentiment using a RoBERTa-based sentiment analysis model to determine developers' satisfaction with different usages and engagement areas.We found positive sentiment (i.e., high satisfaction) about using ChatGPT for refactoring and addressing data analytics (e.g., categorizing table data) issues.On the contrary, we observed negative sentiment when using ChatGPT to debug issues and address automation tasks (e.g., GUI interactions).Our findings show the unmet needs and growing dissatisfaction among developers. 0.634Researchers and ChatGPT developers should focus on developing task-specific solutions that help resolve diverse issues, improving user satisfaction and problem-solving efficiency in software development. 0.618

link