Daily AI Papers
   Summaries auto-generated from HuggingFace's Daily Papers using Gemini and GitHub Actions. All credits go to the research and HuggingFace communities. 🔉 You can get audio summaries via OpenAI's text-to-speech API on Telegram. Note: Authors may be listed by their HuggingFace IDs. Additionally, summaries are generated by LLM and may contain mistakes. You can see the prompt used here here. |
|
Papers for 2025-04-18
| Title | Authors | Summary |
|---|---|---|
| CLIMB: CLustering-based Iterative Data Mixture Bootstrapping for | ||
| Language Model Pre-training (Read more on arXiv or HuggingFace) | Dan Su, Xin Dong, Yonggan Fu, Yu Yang, shizhediao | CLIMB introduces an automated framework using clustering and iterative bootstrapping to optimize language model pre-training data mixtures. The main objective is to automatically discover, evaluate, and refine optimal data mixtures from large-scale corpora without manual curation or predefined domain labels to improve pre-training performance. The methodology involves embedding and clustering documents, followed by an iterative process that samples mixture configurations, trains proxy models, fits a performance predictor, and prunes the search space to find optimal weights. Primary results show that a 1B model trained continuously on 400B tokens using the CLIMB-optimized mixture (ClimbMix) surpassed the Llama-3.2-1B model by 2.0% on average across 12 reasoning benchmarks. The principal implication for AI practitioners is that CLIMB provides a data-driven, automated approach to curate high-quality pre-training datasets from unlabeled web-scale data, demonstrably improving model performance under fixed token budgets compared to baseline mixtures or random sampling, as evidenced by the released ClimbMix dataset. |
| Antidistillation Sampling (Read more on arXiv or HuggingFace) | Avi Schwarzschild, Zhili Feng, Asher Trockman, arobey1, yashsavani | This paper introduces antidistillation sampling, a technique to generate reasoning traces from large language models (LLMs) that hinder model distillation while maintaining the original model’s performance. The primary objective is to develop a sampling strategy that poisons generated data for distillation purposes, thereby protecting proprietary model capabilities, without sacrificing the utility of the model’s outputs for downstream tasks. The key methodology involves modifying the teacher model’s next-token sampling distribution by adding a penalty term proportional to an approximation of how a sampled token would increase a proxy student model’s downstream loss, calculated efficiently via a finite difference approximation of a directional derivative. Results show that for comparable teacher model accuracy on GSM8K (around 68-69%), antidistillation sampling reduced the distilled student model’s accuracy to 24.73%, significantly lower than the 51.86% achieved by a student distilled from traces generated via standard temperature sampling. For AI practitioners, this method offers a way to protect intellectual property embedded in frontier models by degrading the effectiveness of distillation when sharing model outputs, such as extended reasoning traces, while largely preserving the original model’s task performance. |
| A Strategic Coordination Framework of Small LLMs Matches Large LLMs in | ||
| Data Synthesis (Read more on arXiv or HuggingFace) | Honglin Lin, Yu Li, Zinan Tang, Qizhi Pei, GX-XinGao | A coordination framework (GRA) using multiple small LLMs achieves data synthesis quality comparable to single large LLMs. The research objective is to design a resource-efficient framework enabling small LLMs to collectively match the data synthesis capabilities of monolithic LLMs without their associated high costs and limitations. GRA employs a peer-review-inspired methodology assigning distinct Generator, Reviewer, and Adjudicator roles to multiple small LLMs for iterative data generation, evaluation, and quality control. Primary results show GRA-produced data matches or surpasses large LLM quality; data synthesized using GRA with a Qwen-2.5-7B base model outperformed Qwen-2.5-72B-Instruct distilled data by 8.83% on average across tested benchmarks. The principal implication for AI practitioners is that strategically coordinating smaller models offers a computationally efficient alternative for generating high-quality synthetic training data, reducing reliance on large models for data synthesis and distillation. |
| Packing Input Frame Context in Next-Frame Prediction Models for Video | ||
| Generation (Read more on arXiv or HuggingFace) | Maneesh Agrawala, Lvmin Zhang | This paper presents FramePack, a structure for next-frame video prediction that compresses input frames to maintain a fixed transformer context length. The primary objective is to mitigate the “forgetting” (fading memory) and “drifting” (error accumulation) problems in generating long videos. FramePack employs progressive compression using varying transformer patchify kernel sizes based on frame importance and introduces anti-drifting sampling methods like inverted temporal ordering for bi-directional context. Results show that finetuning existing models with FramePack, especially using the inverted anti-drifting sampling (e.g., f1k1_x_g9_f1k1f2k2f16k4_td configuration), achieves superior performance across multiple metrics, including the highest human assessment ELO score of 1239 in ablation studies. For AI practitioners, FramePack offers a method to train video generation models capable of handling longer sequences with significantly higher batch sizes and reduced error accumulation, potentially improving visual quality and training efficiency. |
| Generate, but Verify: Reducing Hallucination in Vision-Language Models | ||
| with Retrospective Resampling (Read more on arXiv or HuggingFace) | Trevor Darrell, Joseph E. Gonzalez, Jiaxin Ge, Heekyung Lee, tsunghanwu | This paper introduces REVERSE, a unified framework reducing visual hallucinations in Vision-Language Models (VLMs) via hallucination-aware training and retrospective resampling. The objective is to enable a single VLM to both detect and dynamically correct its own hallucinations during text generation, unifying generation adjustment and post-hoc verification. Key methodology involves fine-tuning VLMs on a new 1.3M semi-synthetic dataset annotated with confidence tokens (</CN>, </UN>) and employing inference-time retrospective resampling triggered by token uncertainty to backtrack and regenerate content. Primary results demonstrate state-of-the-art performance, achieving up to a 12% reduction in CHAIR scores on CHAIR-MSCOCO compared to previous best methods. For AI practitioners, REVERSE offers a novel technique to enhance VLM reliability by embedding self-verification and correction capabilities directly into the model, reducing reliance on external verifiers or complex multi-stage pipelines. |
| WORLDMEM: Long-term Consistent World Simulation with Memory (Read more on arXiv or HuggingFace) | Shuai Yang, Wenqi Ouyang, Yifan Zhou, Yushi Lan, Zeqi Xiao | WORLDMEM introduces a memory-augmented framework for long-term consistent world simulation, addressing temporal limitations in existing video diffusion models. The primary research objective is to mitigate the lack of long-term 3D spatial consistency in generative world simulators caused by limited temporal context windows. The methodology integrates an external memory bank (storing past frames with pose and timestamp states) into a Conditional Diffusion Transformer, using memory attention with relative state embeddings (Plücker for pose) and Diffusion Forcing to condition generation on retrieved memories. Quantitative results demonstrate improved consistency; for instance, on a Minecraft benchmark beyond the context window, WORLDMEM achieved a PSNR of 25.32 and LPIPS of 0.1429, significantly outperforming a Diffusion Forcing baseline (PSNR 18.04, LPIPS 0.4376). For AI practitioners, this approach offers a method to build more persistent and spatially coherent interactive simulations or virtual environments where maintaining state over extended periods is critical. |
| VistaDPO: Video Hierarchical Spatial-Temporal Direct Preference | ||
| Optimization for Large Video Models (Read more on arXiv or HuggingFace) | Meng Luo, Haojian Huang, scofield7419, ChocoWu, Harold328 | VistaDPO introduces a hierarchical spatial-temporal direct preference optimization framework to enhance large video models (LVMs). The primary objective is to address LVM misalignment with human intuition and video hallucination by optimizing text-video preference alignment across instance, temporal, and perceptive hierarchical levels. The key methodology involves applying this hierarchical DPO framework, termed VistaDPO, using a newly constructed VistaDPO-7k dataset (7.2K QA pairs) annotated with chosen/rejected responses and spatial-temporal grounding information. Experimental results show VistaDPO significantly improves baseline LVMs, achieving average performance gains of 26.42% over PLLaVA and 53.92% over Video-LLaVA across hallucination, QA, and captioning benchmarks. For AI practitioners, this work demonstrates that incorporating hierarchical spatial-temporal preference optimization, beyond simple instance-level DPO, is crucial for improving the reliability and reducing hallucinations in LVMs for complex video understanding tasks. |
| NoisyRollout: Reinforcing Visual Reasoning with Data Augmentation (Read more on arXiv or HuggingFace) | Chao Du, Zijian Wu, Jinjie Ni, Xiangyan Liu, dreamerdeo | NoisyRollout introduces an RL fine-tuning approach for VLMs that enhances visual reasoning by incorporating trajectories from distorted images during rollout collection. The objective is to improve policy exploration diversity and mitigate issues arising from imperfect visual perception in VLMs without additional training costs. The key methodology involves a hybrid rollout strategy within GRPO, using both clean and noise-distorted images (with noise annealing) to generate trajectories for reward calculation, while policy updates use only clean images. Using just 2.1K samples, NoisyRollout achieved state-of-the-art average accuracy of 59.2% across five out-of-domain benchmarks compared to similar open-source RL-tuned models. For AI practitioners, this work demonstrates that targeted data augmentation during RL rollouts can effectively boost VLM generalization and robustness, particularly for visual reasoning, offering a cost-effective method to enhance exploration. |
| ChartQAPro: A More Diverse and Challenging Benchmark for Chart Question | ||
| Answering (Read more on arXiv or HuggingFace) | Firoz Kabir, Aayush Bajaj, Mahir Ahmed, 38saidul, ahmed-masry | This paper introduces ChartQAPro, a diverse and challenging benchmark for Chart Question Answering (CQA). The primary objective was to address the limitations of existing CQA benchmarks, such as lack of diversity and performance saturation, and provide a more realistic evaluation of Large Vision-Language Models (LVLMs). The authors constructed ChartQAPro by collecting 1,341 charts from 157 diverse sources, including infographics and dashboards, paired with 1,948 human-verified questions covering multiple complex types like conversational and hypothetical queries. Evaluations on 21 LVLMs revealed a substantial performance decrease on ChartQAPro compared to prior benchmarks; for instance, Claude Sonnet 3.5’s accuracy dropped from 90.5% on ChartQA to 55.81% on ChartQAPro. For AI practitioners, this implies that current LVLMs struggle significantly with complex, real-world chart reasoning, and ChartQAPro serves as a more robust tool for identifying these limitations and guiding future model development. |
| Exploring Expert Failures Improves LLM Agent Tuning (Read more on arXiv or HuggingFace) | Ruochen Wang, Minhao Cheng, Andrew Bai, Li-Cheng Lan, zhoutianyi | This paper introduces Exploring Expert Failures (EEF), a fine-tuning method that improves LLM agent performance by utilizing information from failed expert trajectories. The objective is to address the limitation of Rejection Sampling Fine-Tuning (RFT), which discards failed expert trajectories, causing agents to struggle with complex, out-of-distribution subtasks where experts often fail. EEF simulates intermediate states from failed expert trajectories using the current agent policy, identifies beneficial action sequences leading to success via simulation, and selectively incorporates only these validated segments into the training data for supervised fine-tuning. The primary result shows EEF achieved a 62% win rate on the WebShop benchmark, significantly outperforming RFT (53.6%) and setting a new state-of-the-art score above 0.81. For AI practitioners, this implies that analyzing and selectively leveraging segments from failed expert demonstrations, rather than discarding them entirely, provides valuable training signals that enhance agent capabilities on complex tasks and improve overall tuning efficiency. |
| InstantCharacter: Personalize Any Characters with a Scalable Diffusion | ||
| Transformer Framework (Read more on arXiv or HuggingFace) | Yiji Cheng, Qixun Wang, Yanbing Zhang, Jiale Tao, wanghaofan | InstantCharacter presents a scalable diffusion transformer framework designed for high-fidelity, open-domain character personalization in image generation. The primary objective is to address the limited generalization, compromised image quality, and reduced textual controllability inherent in previous U-Net based or optimization-based character customization approaches, especially when applied to large Diffusion Transformers (DiTs). Methodologically, it introduces a scalable adapter with stacked transformer encoders, integrating features from SigLIP and DINOv2 via dual-stream fusion and a timestep-aware Q-former, trained progressively in three stages on a 10-million sample dataset containing paired and unpaired character images. Qualitative results demonstrate superior performance in maintaining character identity, fidelity, and text controllability compared to prior art like OminiControl, EasyControl, ACE++, and UNO, achieving comparable results to GPT4o, though specific quantitative metrics are not detailed in the provided text. For AI practitioners, this research offers a robust architecture and training strategy for adapting large foundation DiT models to specialized, controllable generation tasks like character personalization, enhancing flexibility and output quality without requiring test-time fine-tuning. |
| CCMNet: Leveraging Calibrated Color Correction Matrices for Cross-Camera | ||
| Color Constancy (Read more on arXiv or HuggingFace) | Seon Joo Kim, Michael S. Brown, Dongyun Kim, Mahmoud Afifi, dongyong2 | CCMNet introduces a lightweight framework utilizing pre-calibrated Color Correction Matrices (CCMs) for zero-shot cross-camera color constancy. The objective is to enable accurate illuminant estimation on unseen cameras without retraining or needing additional test images. The methodology involves using CCMs to map standard illuminants to the camera’s raw space, encoding this trajectory into a Camera Fingerprint Embedding (CFE) via a CNN, and using this CFE to guide a hypernetwork (based on CCC/C5) for predicting illumination from uv-histograms; imaginary camera augmentation further improves robustness. CCMNet achieves state-of-the-art results, such as a 1.68° mean angular error on Cube+, outperforming previous methods while being computationally efficient. For AI practitioners, this provides a method to achieve consistent color rendering across diverse camera hardware by leveraging readily available ISP metadata (CCMs), eliminating the need for per-camera calibration data or model fine-tuning. |
| FocusedAD: Character-centric Movie Audio Description (Read more on arXiv or HuggingFace) | Liangcheng Li, Sheng Zhou, Yiren Song, Chun Wang, Xiaojun Ye | FocusedAD introduces a novel framework for generating character-centric movie audio descriptions (AD) emphasizing narrative relevance. The main objective is to automatically produce AD for movies that explicitly identifies characters by name and focuses on plot-significant visual details, unlike generic video captioning. The methodology integrates a Character Perception Module (CPM) using an automated clustering-based query bank for character identification/tracking, a Dynamic Prior Module (DPM) injecting context via soft prompts, and a Focused Caption Module (FCM) generating descriptions from scene, character, and text tokens. FocusedAD achieves state-of-the-art performance on multiple benchmarks, including a BertScore of 57.7 on MAD-eval-Named and 64.5 on the introduced Cinepile-AD dataset, significantly outperforming prior AD methods and general MLLMs. For AI practitioners, this work provides a method for enhancing MLLM-based video understanding by incorporating specialized modules for character focus and contextual integration, leading to more narratively coherent and targeted outputs relevant for accessibility tools. |
| Retrieval-Augmented Generation with Conflicting Evidence (Read more on arXiv or HuggingFace) | Mohit Bansal, Elias Stengel-Eskin, Archiki Prasad, HanNight | This paper introduces RAMDocs, a dataset for evaluating RAG systems against simultaneous ambiguity, misinformation, and noise, and proposes MADAM-RAG, a multi-agent debate framework to handle such conflicts. The main objective is to develop and evaluate a RAG approach capable of managing diverse, concurrent sources of conflict in retrieved documents, a common challenge in real-world scenarios. The key methodology involves assigning individual documents to LLM agents who debate their validity over multiple rounds, followed by an aggregator agent synthesizing a final response based on the discussion. MADAM-RAG significantly outperforms strong RAG baselines, improving accuracy by up to 11.40% on AmbigDocs and 15.80% on FaithEval using Llama3.3-70B-Instruct, while the new RAMDocs dataset proves challenging for existing methods. For AI practitioners, this indicates that standard RAG pipelines are insufficient for handling complex, realistic conflicts, and multi-agent debate frameworks like MADAM-RAG are needed to improve the reliability and factuality of RAG outputs when facing ambiguity, misinformation, and noise simultaneously. |
| Sleep-time Compute: Beyond Inference Scaling at Test-time (Read more on arXiv or HuggingFace) | Sarah Wooders, Charles Packer, Yu Wang, Charlie Snell, Kevin Lin | This paper introduces sleep-time compute, a technique allowing LLMs to pre-process context offline to reduce test-time compute requirements. The research aims to evaluate the efficacy of sleep-time compute in improving the accuracy vs. test-time compute trade-off for stateful reasoning tasks. The methodology involves modifying reasoning datasets (GSM-Symbolic, AIME) into stateful versions where context is processed during “sleep-time” before a query arrives, comparing this to standard test-time scaling. Key results show sleep-time compute reduces the test-time compute needed for equivalent accuracy by approximately 5x on Stateful GSM-Symbolic and Stateful AIME, and scaling sleep-time compute can further improve accuracy by up to 18% on Stateful AIME. For AI practitioners, this implies that in stateful applications with available context (e.g., coding agents, document QA), implementing sleep-time compute can significantly cut test-time latency and cost while maintaining or improving accuracy, particularly when future queries are predictable. |
| Set You Straight: Auto-Steering Denoising Trajectories to Sidestep | ||
| Unwanted Concepts (Read more on arXiv or HuggingFace) | Adams Wai-Kin Kong, Yan Ren, Leyang Li, Shilin-LU | This paper introduces ANT, a finetuning framework for concept erasure in text-to-image diffusion models that automatically guides denoising trajectories away from unwanted concepts. The primary objective is to overcome limitations of prior methods by enabling precise content modification during mid-to-late denoising stages without disrupting early-stage structural integrity or relying on heuristic anchor concepts. ANT utilizes a trajectory-aware loss function that reverses the classifier-free guidance condition direction only after a specific timestep (t’) and employs an augmentation-enhanced weight saliency map to identify and finetune only the most relevant parameters for erasure. ANT achieves state-of-the-art results, reducing inappropriate image detections (e.g., NSFW content) on the I2P benchmark to 23, significantly lower than prior methods, while maintaining competitive FID and CLIP scores on MS-COCO. For AI practitioners, ANT provides a more effective and robust finetuning method to build safer generative models by removing unwanted concepts with less impact on overall generative quality and without needing manual anchor selection. |
| Perception Encoder: The best visual embeddings are not at the output of | ||
| the network (Read more on arXiv or HuggingFace) | Andrea Madotto, Jang Hyun Cho, Peize Sun, Po-Yao Huang, Daniel Bolya | Perception Encoder (PE) introduces a state-of-the-art vision encoder family achieving top performance across diverse tasks using only scaled contrastive vision-language pretraining, finding optimal embeddings within intermediate network layers. The main objective was to investigate if a single, scalable contrastive pretraining approach could generate strong, general visual embeddings suitable for classification, retrieval, language modeling, and spatial tasks without complex multi-objective training. The key methodology involved developing a robust image pretraining recipe, creating a video data engine using synthetically generated captions for video finetuning, and introducing language and spatial alignment tuning methods to extract and adapt features from specific intermediate layers. Primary results show PE models achieve state-of-the-art performance; for instance, PEcoreG obtains 86.6% average zero-shot image classification accuracy, outperforming previous models, and its intermediate features rival specialized models like AIMv2 (language) and DINOv2 (spatial) before alignment tuning. The principal implication for AI practitioners is that powerful, general-purpose visual embeddings can be learned via scaled contrastive learning alone, but optimal performance on diverse downstream tasks necessitates extracting and aligning features from intermediate layers rather than solely relying on the final network output. |
Papers for 2025-04-17
| Title | Authors | Summary |
|---|---|---|
| ColorBench: Can VLMs See and Understand the Colorful World? A | ||
| Comprehensive Benchmark for Color Perception, Reasoning, and Robustness (Read more on arXiv or HuggingFace) | zhoutianyi, jiuhai, shweta12, kweCobi, Fcr09 | This paper introduces COLORBENCH, a benchmark to evaluate Vision-Language Models’ (VLMs) capabilities in color perception, reasoning, and robustness. The research aims to assess whether and how current VLMs understand and utilize color information compared to human abilities. Methodology involved creating a benchmark with 11 distinct tasks across 3 core dimensions (Perception, Reasoning, Robustness) grounded in real-world applications, and evaluating 32 VLMs of varying sizes and architectures. Results show that while larger models generally perform better, overall performance on COLORBENCH is low (e.g., top proprietary models achieve ~53.9% overall P&R accuracy pre-CoT), performance gaps are small, and color understanding appears neglected in VLM development. The principal implication for AI practitioners is that current VLMs exhibit critical limitations in color comprehension, underscoring the need for targeted improvements in model architecture and training, using COLORBENCH as a foundational evaluation tool. |
| BitNet b1.58 2B4T Technical Report (Read more on arXiv or HuggingFace) | thegenerality, THU-CHUNXIA, buaahsh, hongyuw, shumingma | This paper introduces BitNet b1.58 2B4T, an open-source, native 1.58-bit, 2-billion parameter LLM trained on 4 trillion tokens. The primary objective was to demonstrate that a native, scaled 1-bit LLM can achieve performance comparable to similar-sized open-weight, full-precision models while being significantly more computationally efficient. Methodology involved training a modified Transformer architecture from scratch, replacing standard linear layers with BitLinear layers using 1.58-bit (ternary {-1, 0, +1}) absolute mean weight quantization and 8-bit activation quantization, followed by SFT and DPO. Results show BitNet b1.58 2B4T achieves performance on par with leading 1-2B parameter full-precision LLMs across various benchmarks (e.g., average score 54.19 vs. 55.23 for Qwen2.5 1.5B) but requires substantially less memory (0.4GB non-embedding vs 2.6GB). For AI practitioners, this work presents a highly efficient LLM that rivals full-precision counterparts in performance, enabling deployment in resource-constrained environments and offering significant reductions in memory, energy, and latency compared to both full-precision and standard post-training quantized models. |
| Cobra: Efficient Line Art COlorization with BRoAder References (Read more on arXiv or HuggingFace) | Zhaoyang Zhang, yshan2u, juxuan27, l-li, JunhaoZhuang | Cobra introduces an efficient, long-context framework for high-fidelity, reference-based line art colorization supporting over 200 references while preserving identity details. The primary objective is to address limitations in existing diffusion models regarding extensive reference handling, inference latency, and flexible control in industrial comic colorization workflows. Key methodology includes a Causal Sparse DiT architecture leveraging Localized Reusable Position Encoding for arbitrary reference image counts and Causal Sparse Attention with KV-Cache to reduce computational complexity. Results show Cobra outperforms baselines on the Cobra-bench benchmark, achieving a FID of 20.98 compared to 26.29 for ColorFlow, while Causal Sparse Attention reduces per-step inference time from 1.99s (Full Attention) to 0.35s using 24 references. For AI practitioners, Cobra offers a scalable and efficient approach for integrating extensive visual context (hundreds of images) into generative tasks like colorization with significantly reduced latency compared to standard attention mechanisms. |
| AlayaDB: The Data Foundation for Efficient and Effective Long-context | ||
| LLM Inference (Read more on arXiv or HuggingFace) | FeTieTer, YuanPeiqi, Qilong00, BenjaminXIANG, YangshenDeng | AlayaDB is a vector database system architected to enhance long-context LLM inference efficiency and effectiveness by managing KV cache and attention computation externally. The primary objective is to simultaneously reduce GPU memory consumption and inference latency (TTFT and TPOT) while maintaining or improving generation quality for long-context tasks, addressing the limitations of coupled, disaggregated, and retrieval-based sparse attention approaches. Key methodologies include decoupling KV cache/attention from the LLM inference engine, introducing a Dynamic Inner Product Range (DIPR) query to dynamically select critical tokens for sparse attention, and employing a native query optimizer with specialized index structures and computation optimizations. Experiments demonstrate that AlayaDB achieves better average generation quality (47.0) on ∞-Bench compared to baseline methods like InfLLM (43.8) and Top-k (46.7), while meeting latency SLOs and significantly reducing TTFT by 19-42x compared to LMCache for context reuse. For AI practitioners, AlayaDB offers a data foundation that can lower hardware resource requirements and simplify the development of high-performing long-context LLM applications by abstracting complex cache management and attention computation. |
| SIFT-50M: A Large-Scale Multilingual Dataset for Speech Instruction | ||
| Fine-Tuning (Read more on arXiv or HuggingFace) | Jian Xie, Rupak Vignesh Swaminathan, svinxz, vijaygirish2001, panprabh | This paper introduces SIFT-50M, a 50M-example, five-language dataset generated using LLMs from public speech corpora for speech instruction fine-tuning. The primary objective was to create a large-scale, diverse dataset to improve the instruction-following capabilities and generalization of speech-text LLMs beyond standard ASR tasks. Key methodology involved extracting detailed acoustic and content metadata from speech, mapping it to categorical values, and using LLMs (Mixtral 8x7B, Amazon Nova Pro) prompted with this metadata to generate varied instruction-response pairs, including closed-ended QA, open-ended analysis, and controllable generation prompts. The resulting SIFT-LLM model (Whisper-medium + Qwen2.5-7B), trained on SIFT-50M, achieved state-of-the-art performance on instruction-following benchmarks, notably scoring 57.4% accuracy on Dynamic-Superb (DS-1) closed-ended tasks, significantly outperforming prior models. For AI practitioners, SIFT-50M provides a substantial resource for training speech-text models that better comprehend and execute nuanced, multilingual instructions related to both speech understanding and controllable generation, alongside the EvalSIFT benchmark for systematic evaluation. |
| ReTool: Reinforcement Learning for Strategic Tool Use in LLMs (Read more on arXiv or HuggingFace) | chijx, imjcqt, YujiaHi, zhangysk, JoeYing | ReTool is a reinforcement learning framework enhancing LLM mathematical reasoning by strategically integrating real-time code interpreter execution. The research objective is to teach LLMs when and how to leverage external computational tools effectively for complex reasoning tasks where pure text-based approaches falter. The methodology uses supervised fine-tuning on synthetic code-augmented data for initialization, followed by PPO-based reinforcement learning where task outcome accuracy serves as the reward signal during policy rollouts involving real-time code execution. Primary results show ReTool significantly boosts performance and efficiency, achieving 67.0% accuracy on AIME 2024 (400k steps) versus a text-only RL baseline (40.0%, 1080k steps), and exhibits emergent capabilities like code self-correction. For AI practitioners, this work shows outcome-driven RL effectively teaches LLMs strategic tool use, yielding more capable and efficient reasoning models for computational tasks without complex reward engineering or explicit tool-use supervision. |
| REPA-E: Unlocking VAE for End-to-End Tuning with Latent Diffusion | ||
| Transformers (Read more on arXiv or HuggingFace) | liangzheng06, sainx, Zhenchang, yunzhong-hou, xingjianleng | This paper introduces REPA-E, a method enabling joint end-to-end training of VAEs and latent diffusion transformers using representation alignment loss. The main objective is to develop an effective end-to-end training scheme for both the VAE tokenizer and the diffusion model, overcoming the performance degradation observed when using standard diffusion loss for joint training. REPA-E utilizes representation alignment (REPA) loss to jointly optimize VAE and diffusion model parameters, applying standard diffusion loss only to the diffusion model via stop-gradients, and incorporates batch normalization and VAE regularization. The proposed method significantly accelerates training, achieving an FID of 4.07 on ImageNet 256x256 in 400k steps (over 17x faster than the REPA baseline) and attains a state-of-the-art FID of 1.26 with classifier-free guidance. For AI practitioners, REPA-E offers a technique to drastically reduce latent diffusion model training time while simultaneously improving the VAE’s latent structure and final generative performance through joint optimization. |
| Vivid4D: Improving 4D Reconstruction from Monocular Video by Video | ||
| Inpainting (Read more on arXiv or HuggingFace) | Yiyi Liao, BangBnag Yang, yuewenma, shengmiao, JaceyH919 | Vivid4D enhances 4D reconstruction from monocular video by reformulating view augmentation as a video inpainting task integrating geometric and generative priors. The primary research objective is to improve the quality and completeness of 4D dynamic scene reconstruction from sparse monocular video inputs. Key methodology involves warping observed views to novel viewpoints using monocular depth priors, training a video diffusion model on unposed web videos with synthetic occlusion masks to inpaint missing regions, and employing an iterative view augmentation strategy with a robust reconstruction loss. Results demonstrate improved reconstruction quality, achieving an overall PSNR of 19.45 on the HyperNeRF dataset, outperforming baselines like 4D GS (18.24) and Shape of Motion (18.82). For AI practitioners, this work presents a practical method using video inpainting to generate richer supervision signals from monocular video, thereby enhancing the fidelity of 4D scene reconstructions for applications like VR/AR content creation. |
| Robust and Fine-Grained Detection of AI Generated Texts (Read more on arXiv or HuggingFace) | ashay-sriv, jebish7, DrishtiSharma, Siddartha10, 1024m | This paper presents robust token-classification models for fine-grained detection of AI-generated text, including human-LLM co-authored content. The main objective was to create detection systems resilient to unseen generators, domains, adversarial inputs, non-native speaker text, and shorter or partially AI-generated texts. The key methodology involved training multilingual transformer models (specifically xlm-longformer) with an additional CRF layer using a token-classification approach on a new, large dataset (~2.45M samples) of human-machine co-authored texts across 23 languages and 12 LLMs. Primary results include an average word-level accuracy of 94.19% on their diverse test set and demonstrating robustness against adversarial inputs on the raid-bench benchmark, achieving an F1 score of 0.79 without specific adversarial training. The principal implication for AI practitioners is that a token-classification approach trained on varied co-authored data significantly improves robustness for detecting AI text, particularly in mixed-authorship scenarios and against unseen generators or adversarial attacks, offering a more practical method than binary text classification. |
| Syzygy of Thoughts: Improving LLM CoT with the Minimal Free Resolution (Read more on arXiv or HuggingFace) | Qigan Sun, Jiaquan Zhang, Yi Lu, Chaoning Zhang, Chenghao Li | Syzygy of Thoughts (SoT) introduces a novel framework extending Chain-of-Thought (CoT) by incorporating Minimal Free Resolution (MFR) principles to enhance LLM reasoning. The objective is to improve the robustness and structure of LLM problem-solving for complex tasks by capturing deeper logical dependencies compared to standard CoT. The methodology leverages algebraic concepts like “Module”, “Betti numbers”, and “Minimality” to systematically decompose problems into minimal, logically complete subproblems and interrelated reasoning paths. Results demonstrate that SoT matches or surpasses CoT and CoT-SC accuracy across datasets like GSM8K and MATH; for instance, using GPT-4o-mini on GSM8K, SoT achieved 96.0% accuracy versus 85.1% for CoT. For AI practitioners, SoT provides a structured, mathematically-inspired approach to prompt engineering that can yield more reliable and transparent reasoning chains for complex tasks, potentially reducing errors and improving performance without relying solely on larger models. |
Papers for 2025-04-16
| Title | Authors | Summary |
|---|---|---|
| Genius: A Generalizable and Purely Unsupervised Self-Training Framework | ||
| For Advanced Reasoning (Read more on arXiv or HuggingFace) | Haiteng Zhao, Chang Ma, Hang Yan, QiushiSun, xufangzhi | Genius is a generalizable, purely unsupervised self-training framework designed to enhance Large Language Model (LLM) reasoning capabilities without external supervision. The central research objective is to advance LLM reasoning ability using only general, unlabeled queries, bypassing the need for annotated data or auxiliary reward models. Genius employs a stepwise foresight re-sampling strategy to sample candidate reasoning steps and estimate their value by simulating future outcomes, coupled with an Advantage-Calibrated Optimization (ACO) loss function to handle estimation noise and ensure robust optimization. Using only 25K unsupervised general queries from the Magpie dataset, Genius improved the average reasoning performance of LLaMA3.1-8B-Instruct by over 7% (from 49.65% to 57.08%) across seven reasoning benchmarks. For AI practitioners, this demonstrates a promising approach to scale LLM reasoning performance by leveraging vast amounts of readily available unlabeled data, potentially reducing dependency on expensive annotations and specialized reward models. |
| xVerify: Efficient Answer Verifier for Reasoning Model Evaluations (Read more on arXiv or HuggingFace) | Bo Tang, Wentao Zhang, Pengyuan Wang, Duguce, Hush-cd | This paper introduces xVerify, an efficient LLM-based answer verifier designed for evaluating reasoning models by accurately determining answer equivalence. The research aims to address the inadequacy of existing evaluation methods in extracting final answers and performing robust equivalence checks for complex, multi-step reasoning outputs from LLMs. Methodologically, the authors constructed the VAR dataset from 19 LLMs across 24 benchmarks, used multi-round GPT-4o and human annotation for labeling, and fine-tuned various xVerify models (0.5B-32B parameters) using QLoRA. Key results show all xVerify models achieving over 95% F1 score and accuracy on the test set, with the xVerify-3B-Ib model surpassing even GPT-4o (used as a CoT judge) in overall performance (97.27% vs 96.95% accuracy). For AI practitioners, the publicly available xVerify models offer a more reliable, efficient, and cost-effective method for automatically evaluating the correctness of reasoning model outputs compared to expensive API calls or less robust rule-based frameworks. |
| Pixel-SAIL: Single Transformer For Pixel-Grounded Understanding (Read more on arXiv or HuggingFace) | Weixian Lei, Yanwei Li, Zilong Huang, Tao Zhang, LXT | Pixel-SAIL introduces a single-transformer architecture for multimodal large language models (MLLMs) targeting fine-grained, pixel-level understanding tasks. The primary research objective is to develop a highly simplified MLLM architecture for pixel-grounded understanding, eliminating the need for separate vision encoders and segmentation expert modules. Key methodologies include integrating a learnable upsampling module for refining visual tokens, a novel visual prompt injection strategy using special vocabulary tokens fused early with vision tokens, and a vision expert distillation technique. Pixel-SAIL (3B) demonstrates superior performance on referring segmentation benchmarks, outperforming larger models like GLaMM (7B) by up to 3.0% cIoU on RefCOCOg with a significantly simpler pipeline. For AI practitioners, this work shows that effective pixel-level understanding can be achieved with reduced architectural complexity using a unified transformer, potentially simplifying model development, training, and deployment. |
| Heimdall: test-time scaling on the generative verification (Read more on arXiv or HuggingFace) | Xing Jin, WesleyShi | This paper introduces Heimdall, an RL-trained long CoT verifier, and Pessimistic Verification to enhance LLM solution correctness judgment and problem-solving scaling. The main objective is to improve the weak verification capabilities of LLMs for complex reasoning tasks and leverage this improved verification to scale overall problem-solving accuracy. Key methodology involves training Heimdall via PPO reinforcement learning on filtered math problems and proposing Pessimistic Verification, an algorithm that selects solutions by balancing solver outputs and verifier judgments using a lower-confidence-bound approach. Primary results show Heimdall boosting verification accuracy from 62.5% to 94.5% on AIME2024 (97.5% with sampling), while Pessimistic Verification improved AIME2025 solving accuracy from 54.2% to 70.0% (16x compute budget with DeepSeek-R1-Distill-Qwen-32B). The principal implication for AI practitioners is that utilizing dedicated RL-trained verifiers and selection algorithms like Pessimistic Verification can significantly enhance the reliability and performance of LLMs on complex problem-solving by explicitly verifying and selecting trustworthy solutions. |
| Seedream 3.0 Technical Report (Read more on arXiv or HuggingFace) | Zhichao Lai, Xiaoxia Hou, Qiushan Guo, Lixue Gong, Yu Gao | Seedream 3.0 is presented as a high-performance Chinese-English bilingual text-to-image foundation model with significant improvements over its predecessor. The objective was to enhance alignment with complex prompts, fine-grained typography (especially Chinese text), visual aesthetics, fidelity, and native image resolution. Methodologies involved data augmentation (defect-aware training, dual-axis sampling), architectural improvements (mixed-resolution training, cross-modality RoPE, representation alignment loss), advanced post-training (aesthetic SFT, VLM reward model), and novel acceleration techniques (consistent noise expectation, importance-aware timestep sampling). Seedream 3.0 achieves superior performance, ranking first on the Artificial Analysis Leaderboard (ELO 1158), demonstrating a 94% text availability rate for Chinese characters, and enabling 4-8x inference speedup while supporting native 2K resolution. For AI practitioners, this model offers enhanced capabilities for high-fidelity, high-resolution bilingual image generation with strong text rendering and improved prompt adherence, suitable for applications demanding advanced typography and aesthetic quality. |
| How Instruction and Reasoning Data shape Post-Training: Data Quality | ||
| through the Lens of Layer-wise Gradients (Read more on arXiv or HuggingFace) | Ziyue Li, Yanhong Li, Ming Li, zhoutianyi | This paper analyzes how instruction and reasoning data quality impacts LLM post-training dynamics through the spectral properties of layer-wise gradients. The primary objective is to understand how low/high-quality instruction and reasoning data affect gradients and to unify different data quality evaluation metrics using gradient spectral characteristics. The study employs Singular Value Decomposition (SVD) on the layer-wise gradients (specifically Q, K, V, O projections) of various LLMs (Qwen2, Llama3, Gemma2 families) finetuned on datasets partitioned by quality metrics (IFD, InsTag, Difficulty, Reward) and compares instruction-following versus reasoning data. Results consistently show that higher-quality data, for both instruction and reasoning types, leads to lower nuclear norms and significantly higher effective ranks of the gradients; for instance, high-quality reasoning data (s1.1) yielded substantially higher effective ranks than high-quality instruction data across models (e.g., Table 2, Qwen2.5-7B K-projection high-quality reasoning rank 361.2 vs. instruction rank 153.3). The principal implication for AI practitioners is that the effective rank of layer-wise gradients offers a unified, robust metric to evaluate data quality, potentially guiding more effective data selection or synthesis strategies for stable LLM post-training, particularly for developing complex reasoning abilities. |
| TextArena (Read more on arXiv or HuggingFace) | Leshem Choshen, Benjamin-eecs, simonycl, bobbycxy, LeonGuertler | TextArena introduces an open-source framework leveraging 74+ competitive text-based games for evaluating and training agentic capabilities in LLMs via a dynamic TrueSkill leaderboard. The objective is to provide a scalable, relative benchmark assessing LLM skills like strategic planning, theory of mind, and deception, often missed by static benchmarks, through competitive gameplay. Methodologically, TextArena employs diverse text-based games (single/two/multi-player) within a Gym-compatible interface, evaluating models online (model-vs-model/human) and tracking performance using TrueSkill ratings across 10 specific soft skills. Primary results include relative model rankings and granular skill profiles; preliminary data shows frontier models achieving TrueSkill scores in the 30-38 range in certain games, demonstrating capabilities relative to a collective human baseline, though performance varies significantly across tasks (Figure 2). For AI practitioners, TextArena offers a platform to benchmark complex agentic behaviors without human preference bias, diagnose specific model skill gaps (e.g., Persuasion vs. Spatial Thinking), and potentially generate diverse interaction data for RL-based agent training. |
| The Scalability of Simplicity: Empirical Analysis of Vision-Language | ||
| Learning with a Single Transformer (Read more on arXiv or HuggingFace) | Jun Hao Liew, Haochen Wang, Jiacong Wang, Weixian Lei, LXT | This paper introduces and empirically analyzes SAIL, a single-transformer architecture for joint vision-language processing, comparing its properties to modular designs. The research objective is to evaluate the scalability, cross-modal information flow patterns, and visual representation capabilities of this unified approach against modular Multimodal Large Language Models (MLLMs) that use separate vision encoders. SAIL employs a single transformer with mixed attention (bidirectional for image patches, causal for text) and multimodal rotary position embeddings (M-RoPE) to process raw pixels and text, evaluated via scaling experiments and performance on vision-language/vision benchmarks. Key results show SAIL exhibits superior data scalability compared to modular models (Fig 1A) and achieves strong vision task performance, including 84.95% Top-1 accuracy on ImageNet-1K classification, demonstrating effective visual feature learning without a pre-trained encoder. For AI practitioners, this indicates that unified single-transformer architectures are a viable, potentially more scalable alternative to complex modular designs, simplifying the model stack while achieving competitive performance, especially with large datasets. |
| Efficient Process Reward Model Training via Active Learning (Read more on arXiv or HuggingFace) | Tianyu Pang, Xin Mao, Zichen Liu, Keyu Duan, dreamerdeo | This paper proposes ACTPRM, an active learning framework to efficiently train Process Reward Models (PRMs) for large language models. The primary objective is to reduce the prohibitive annotation costs required for obtaining step-level supervision needed to train PRMs. ACTPRM employs an ensemble PRM to estimate both aleatoric and epistemic uncertainty at each reasoning step, selectively forwarding only the most uncertain samples to a capable reasoning LLM for annotation, and then training the PRM exclusively on this subset. ACTPRM achieved state-of-the-art performance (75.0% average F1) on ProcessBench while requiring only 20% of the estimated annotation cost compared to the prior SOTA model, UniversalPRM. For AI practitioners, this methodology offers a significantly more cost-effective approach to training PRMs, enabling scalable development of LLMs with improved reasoning capabilities through fine-grained process supervision. |
| Efficient Generative Model Training via Embedded Representation Warmup (Read more on arXiv or HuggingFace) | Tao Lin, Xufeng Li, Peng Sun, SempraETY | This paper introduces Embedded Representation Warmup (ERW) to accelerate diffusion model training by initializing early layers with pretrained representations. The primary objective is to improve training efficiency and representation quality by decoupling the representation learning phase from the generation phase in diffusion models. ERW employs a two-phase training strategy: first, a warmup phase aligns the initial layers (Latent-to-Representation circuit) with features from a pretrained model (e.g., Dinov2) using an alignment loss; second, standard diffusion training proceeds with a decaying alignment guidance term. Empirically, ERW demonstrates a 40x acceleration in training speed compared to the REPA baseline, achieving an FID of 6.0 on ImageNet-1k (SiT-XL/2, no CFG) within 100k iterations. For AI practitioners, ERW offers a plug-and-play method to significantly reduce computational costs and training time for large diffusion models by leveraging existing pretrained representation encoders, making state-of-the-art generative modeling more accessible. |
| NormalCrafter: Learning Temporally Consistent Normals from Video | ||
| Diffusion Priors (Read more on arXiv or HuggingFace) | Bing Wang, Xinya Chen, Haoyuan Wang, Yanrui Bin, wbhu-tc | NormalCrafter introduces a novel method leveraging video diffusion priors to generate temporally consistent and detailed surface normals from open-world videos. The main objective is to address the challenge of maintaining both high spatial fidelity and temporal coherence in video-based normal estimation, which existing methods often fail to achieve simultaneously. Key methodology includes adapting a pre-trained video diffusion model (SVD), proposing Semantic Feature Regularization (SFR) to align internal features with semantic representations (from DINO), and utilizing a two-stage training protocol optimizing first in latent space for temporal context and then in pixel space for spatial accuracy. Primary results demonstrate superior performance on video benchmarks, achieving a 1.6° reduction in mean angular error on the Sintel dataset compared to the prior state-of-the-art, alongside improved temporal consistency. For AI practitioners, this research provides a framework for adapting large video generative models for downstream perception tasks, showcasing how diffusion priors combined with specific regularization and training strategies can yield high-fidelity, temporally stable outputs for video understanding applications like 3D reconstruction or editing. |
| A Minimalist Approach to LLM Reasoning: from Rejection Sampling to | ||
| Reinforce (Read more on arXiv or HuggingFace) | Lei Wang, Bo Pang, Yuhui Xu, Jiarui Yao, Wei Xiong | This paper analyzes simplified reinforcement learning algorithms for fine-tuning large language models (LLMs) on reasoning tasks, demonstrating the strong performance of rejection sampling. The primary objective is to understand the sources of effectiveness in complex RL algorithms like GRPO and identify minimal yet performant alternatives. Key methodologies include empirical comparisons of RAFT (rejection sampling), vanilla Reinforce, GRPO, and PPO on mathematical reasoning benchmarks, alongside ablation studies isolating components like reward normalization and sample filtering, leading to a proposed variant, Reinforce-Rej. The primary result shows that RAFT achieves competitive performance (e.g., 49.9% average accuracy on Qwen2.5-Math-7B-base) compared to GRPO (53.9%) and PPO (51.8%), with GRPO’s advantage largely attributed to filtering prompts with only incorrect responses, not reward normalization. The principal implication for AI practitioners is that simpler, computationally lighter methods like RAFT and the proposed Reinforce-Rej can be highly effective alternatives to complex RL algorithms for reward-based LLM fine-tuning, highlighting the crucial role of selective sample filtering over intricate algorithmic designs. |
| DeepMath-103K: A Large-Scale, Challenging, Decontaminated, and | ||
| Verifiable Mathematical Dataset for Advancing Reasoning (Read more on arXiv or HuggingFace) | Xingyu Chen, Qiuzhi Liu, Jiahao Xu, Tian Liang, Zhiwei He | This paper introduces DeepMath-103K, a large-scale, challenging, decontaminated, and verifiable mathematical dataset designed for advancing AI reasoning via reinforcement learning. The primary objective was to create a dataset overcoming limitations of existing resources, namely insufficient difficulty, lack of verifiable answers for RL, benchmark contamination, and inadequate scale for highly challenging problems. The methodology involved a rigorous curation pipeline including source analysis, semantic decontamination against multiple benchmarks using LLM-judges, difficulty filtering focusing on levels 5-9, and answer verification through consistency checks across three distinct R1-generated solutions for each of the 103K problems. Models trained using RL-Zero on DeepMath-103K demonstrated significant performance improvements, with DeepMath-Zero-7B achieving 85.5% pass@1 accuracy on MATH500, substantially outperforming baseline and models trained on other RL datasets. For AI practitioners, DeepMath-103K provides a crucial, publicly available resource enabling the development and evaluation of more powerful reasoning systems, particularly through rule-based RL paradigms demanding verifiable answers and high problem complexity. |
| Diffusion Distillation With Direct Preference Optimization For Efficient | ||
| 3D LiDAR Scene Completion (Read more on arXiv or HuggingFace) | Jiale Wu, Zejian Li, Ling Yang, Shengyuan Zhang, An Zhaol | This paper proposes Distillation-DPO, a novel framework integrating diffusion distillation with direct preference optimization for efficient and high-quality 3D LiDAR scene completion. The primary objective is to accelerate the slow sampling speed of diffusion models for LiDAR completion while mitigating performance degradation typically associated with distillation. Distillation-DPO generates paired completion samples using a student model with varied initial noise, constructs win/lose pairs based on non-differentiable LiDAR metrics (used as preference), and optimizes the student by minimizing the difference in score functions between teacher and student models on these pairs, facilitated by two teaching assistant models. Experiments demonstrate that Distillation-DPO achieves superior completion quality (e.g., 0.354 refined CD compared to the SOTA LiDiff’s 0.375) while accelerating inference speed by over 5-fold (3.38s vs 17.87s). For AI practitioners, this method offers a way to significantly enhance the efficiency of diffusion models for 3D scene completion tasks, making them more viable for real-world applications by effectively using preference data to guide distillation without requiring differentiable reward functions. |
| PVUW 2025 Challenge Report: Advances in Pixel-level Understanding of | ||
| Complex Videos in the Wild (Read more on arXiv or HuggingFace) | Shuting He, Nikhila Ravi, Chang Liu, LXT, HenghuiDing | This report summarizes the 4th Pixel-level Video Understanding in the Wild (PVUW) Challenge, focusing on methods and results for complex video segmentation tasks. The primary objective was to benchmark and advance algorithms for complex video object segmentation (MOSE track) and motion/language-guided video segmentation (MeViS track) using new, challenging real-world datasets. Key methodologies employed by top teams included fine-tuning large foundation models like SAM2, utilizing multi-model ensembles, adaptive pseudo-labeling (e.g., PGMR), and integrating Large Multimodal Models (LMMs) like Sa2VA, evaluated via J&F scores on confidential test sets. The top team on the MOSE track achieved a J&F score of 87.26%, while the MeViS track winner reached 61.98%, showcasing the effectiveness of these advanced techniques. For AI practitioners, the principal implication is the demonstrated benefit of adapting large pre-trained vision and multimodal models (SAM2, LMMs) and using ensemble strategies to improve robustness and accuracy in complex, dynamic video understanding tasks. |
| ReZero: Enhancing LLM search ability by trying one-more-time (Read more on arXiv or HuggingFace) | Thinh Le, alandao | ReZero introduces a reinforcement learning framework to enhance LLM search persistence within Retrieval-Augmented Generation (RAG) by rewarding query retries. The main objective is to improve LLM robustness in information retrieval by explicitly incentivizing the model to attempt subsequent searches if the initial one fails. The key methodology utilizes Group Relative Policy Optimization (GRPO) to fine-tune an LLM, incorporating a specific reward_retry function that rewards additional search attempts conditional on generating a correct final answer. The primary result showed the ReZero model achieved 46.88% peak accuracy on the evaluation dataset, nearly doubling the 25.00% peak accuracy of a baseline model trained without the retry incentive. For AI practitioners, this implies that designing RL rewards to explicitly encourage persistence can significantly improve RAG system performance, especially for tasks where initial information retrieval attempts are likely insufficient. |
| AI-University: An LLM-based platform for instructional alignment to | ||
| scientific classrooms (Read more on arXiv or HuggingFace) | Rahul Gulati, Mostafa Faghih Shojaei, garikipati, Dinzhenzhenzhu, simocimolato | This paper introduces AI-University (AI-U), a framework using fine-tuned LLMs and Retrieval-Augmented Generation (RAG) to generate instructor-aligned responses for scientific courses. The objective was to develop and evaluate a platform that adapts an LLM (Llama-3.2-11B) to a specific graduate-level Finite Element Method (FEM) course’s content and teaching style using lecture transcripts, notes, and textbooks. Key methodology involved systematic question-answer pair generation for LoRA-based fine-tuning (creating LLaMA-TOMMI-1.0), followed by RAG synthesis for contextualized, referenced answers, evaluated via cosine similarity and LLM-as-a-judge. The fine-tuned LLaMA-TOMMI-1.0 model achieved higher cosine similarity to ground-truth answers than the base model on 86% of test cases and was preferred approximately four times more often by an LLM judge. The principal implication for AI practitioners is that this combined approach of systematic data generation for fine-tuning and RAG offers a robust method for developing domain-specific LLMs that exhibit strong alignment with specialized technical content and style, providing traceable and accurate AI assistance. |
| Adaptive Computation Pruning for the Forgetting Transformer (Read more on arXiv or HuggingFace) | Aaron Courville, Johan Obando-Ceron, Zhixuan Lin, littleowen | This paper proposes Adaptive Computation Pruning (ACP) to accelerate the Forgetting Transformer (FoX) by dynamically skipping computations based on forget gate decay. The objective is to determine if dynamically pruning FoX attention computations based on decay strength can improve training throughput without performance loss. ACP employs a dynamic pruning threshold, calculated based on attention logit bounds and sequence length, to identify and skip negligible input-output dependency computations within a modified FlashAttention framework. Results demonstrate that ACP consistently reduces FLOPs in softmax attention by ~70% across different model sizes (125M-760M) and context lengths (4k-16k), resulting in 10%-35% faster training throughput without performance degradation on language modeling or downstream tasks. For AI practitioners, ACP provides a technique to significantly decrease computational costs and improve training efficiency for FoX models, particularly those with long contexts, while maintaining accuracy. |
| Multimodal Long Video Modeling Based on Temporal Dynamic Context (Read more on arXiv or HuggingFace) | Xiangyu Yue, Yiyuan Zhang, Jiaming Han, Hoar012 | This paper introduces Temporal Dynamic Context (TDC), a method for multimodal long video understanding integrating static features and dynamic context compression. The research aims to address MLLM context length limitations and suboptimal multimodal integration (vision, audio) in long video processing. TDC segments videos by inter-frame similarity, encodes static keyframes fully, and uses a Q-Former to compress subsequent visual/audio tokens based on temporal differences relative to the static frame; a Long Video Chain-of-Thought (LVCoT) strategy handles extremely long videos without training. TDC demonstrates strong performance, outperforming the audio-visual VideoLLaMA2 model by 15.6% on the long-video MLVU benchmark. For AI practitioners, TDC provides an effective technique for encoding dense multimodal video data more efficiently, enabling MLLMs to process longer videos by compressing dynamic context while preserving key static details, reducing information loss compared to sparse sampling or purely visual compression methods. |
| Summarization of Multimodal Presentations with Vision-Language Models: | ||
| Study of the Effect of Modalities and Structure (Read more on arXiv or HuggingFace) | Frédéric Dufaux, Camille Guinaudeau, gigant | This paper analyzes how input modality and structure affect Vision-Language Model (VLM) performance for summarizing multimodal presentations. The primary objective is to evaluate the cost-performance tradeoffs of various input representations (raw video, extracted slides, transcript, structured/unstructured combinations) and suggest effective strategies. Using Qwen2-VL and other VLMs on a benchmark derived from the TIB dataset, the study measured performance with metrics like ROUGE and Importance-based Relevance (IbR). Results demonstrate that a structured representation using interleaved slides and transcript yields the best performance (e.g., Qwen2-VL 2B achieved ROUGE-1 of 27.1 and overall IbR of 33.4), significantly outperforming raw video or unstructured inputs. For AI practitioners, the key implication is that preprocessing presentations into structured, interleaved slide-transcript sequences offers the most effective input for VLM summarization, balancing computational cost and summary quality, especially for inputs exceeding approximately 6k tokens. |
| D^2iT: Dynamic Diffusion Transformer for Accurate Image Generation (Read more on arXiv or HuggingFace) | Zhendong Mao, Lei Zhang, Nan Chen, Mengqi Huang, Weinan Jia | This paper introduces D²iT, a Diffusion Transformer using dynamic compression based on regional information density to improve image generation accuracy. The main objective is to overcome the limitations of fixed spatial compression in standard Diffusion Transformers (DiTs) which disregard varying information densities across image regions. The methodology employs a two-stage framework: first, a Dynamic VAE (DVAE) uses a hierarchical encoder and information density estimation (Shannon entropy) to create multi-grained latent codes; second, the Dynamic Diffusion Transformer (D²iT) predicts corresponding multi-grained noise using novel Dynamic Grain and Content Transformers. Primary results demonstrate a significant quality improvement, achieving a 1.73 FID score on class-conditional ImageNet 256x256 generation, a 23.8% improvement over the baseline DiT’s 2.27 FID, using only 57.1% of the training resources. For AI practitioners, this research implies that dynamically adapting compression and computational effort based on input complexity, rather than using fixed approaches, can yield substantial gains in both the performance and efficiency of generative models like DiTs. |
| Change State Space Models for Remote Sensing Change Detection (Read more on arXiv or HuggingFace) | Erchan Aptoula, ElmanGhazaei | This paper introduces the Change State Space Model (CSSM), a computationally efficient Mamba-based architecture tailored for remote sensing change detection. The research objective is to develop a specialized state-space model that focuses exclusively on relevant bi-temporal changes, improving efficiency and accuracy over existing ConvNet, ViT, and general Mamba approaches for change detection. CSSM utilizes a lightweight CNN encoder-decoder framework incorporating a modified state space model block that employs an L1 distance mechanism on projected inputs to isolate and process only changed features between pre- and post-event images. Evaluated on benchmark datasets like LEVIR-CD+, CSSM achieved state-of-the-art performance, attaining an F1-score of 92.39 while requiring only 4.34M parameters and 5.10 GFLOPs, significantly less than comparable models. For AI practitioners, CSSM presents a highly resource-efficient architecture delivering state-of-the-art accuracy in change detection, making it suitable for large-scale analysis or deployment in computationally constrained environments. |
| LazyReview A Dataset for Uncovering Lazy Thinking in NLP Peer Reviews (Read more on arXiv or HuggingFace) | Iryna Gurevych, Lizhen Qu, Anne Lauscher, Zhuang Li, sukannya | This paper introduces LAZYREVIEW, a dataset annotated with fine-grained categories to detect ‘lazy thinking’ heuristics in NLP peer reviews. The primary objective was to create this resource and evaluate the ability of Large Language Models (LLMs) to automatically identify such instances. The methodology involved iteratively developing annotation guidelines over three rounds using ARR-22 reviews, annotating 500 expert and 1276 silver review segments, and evaluating LLMs using zero-shot, few-shot in-context learning, and instruction fine-tuning. Key results show that while LLMs struggle in zero-shot detection, instruction fine-tuning on LAZYREVIEW significantly boosts performance by 10-20 accuracy points (e.g., instruction-tuned Qwen achieved 59.4% string-matching accuracy for fine-grained classification). For AI practitioners, this provides a validated dataset and methodology for building automated tools to flag superficial review arguments, potentially improving review quality assessment systems and reviewer training. |
Papers for 2025-04-15
| Title | Authors | Summary |
|---|---|---|
| InternVL3: Exploring Advanced Training and Test-Time Recipes for | ||
| Open-Source Multimodal Models (Read more on arXiv or HuggingFace) | jackroos, duanyuchen, gulixin0922, Yeshenglong, Weiyun1025 | InternVL3 presents an open-source Multimodal Large Language Model (MLLM) series developed via native multimodal pre-training and advanced training/test-time techniques. The research objective was to improve MLLM performance and training efficiency by jointly learning multimodal and linguistic capabilities within a single pre-training stage, circumventing typical post-hoc adaptation of text-only LLMs. Key methodologies employed include unified pre-training on mixed text and multimodal corpora, Variable Visual Position Encoding (V2PE), supervised fine-tuning (SFT), and Mixed Preference Optimization (MPO) post-training, alongside test-time scaling. The primary result shows InternVL3-78B achieving a state-of-the-art score of 72.2% on the MMMU benchmark among open-source MLLMs, demonstrating strong capabilities competitive with proprietary models like ChatGPT-4o and Gemini 2.5 Pro. For AI practitioners, this work provides evidence that native multimodal pre-training yields powerful open-source MLLMs, and the released models and data offer a strong foundation for developing advanced multimodal applications without relying solely on closed systems. |
| PRIMA.CPP: Speeding Up 70B-Scale LLM Inference on Low-Resource Everyday | ||
| Home Clusters (Read more on arXiv or HuggingFace) | Hongfang Yu, Mohsen Guizani, NeuronNomad, LiPhilip, LIKirin | PRIMA.CPP introduces a distributed system for running 70B-scale LLMs on heterogeneous, low-resource home device clusters. The objective is to minimize inference latency while managing limited and diverse resources (CPU/GPU, RAM/VRAM, disk, OS, network). It employs piped-ring parallelism with prefetching to hide disk I/O latency from memory-mapped weights and uses the Halda algorithm to optimally assign model layers based on a detailed heterogeneity model. Evaluations on a four-node home cluster show prima.cpp is 15x faster than llama.cpp for 70B models, achieving ~600 ms/token with memory pressure under 6%. This enables AI practitioners to deploy state-of-the-art 30B-70B models locally on clusters of everyday consumer devices, expanding accessibility beyond high-end hardware or cloud services. |
| VL-Rethinker: Incentivizing Self-Reflection of Vision-Language Models | ||
| with Reinforcement Learning (Read more on arXiv or HuggingFace) | Wei Chu, Chao Qu, wenhu, zuminghuang, JasperHaozhe | VL-Rethinker improves multimodal reasoning by incentivizing self-reflection in vision-language models through reinforcement learning. The research aims to enhance slow-thinking capabilities in VLMs for complex multimodal tasks. It uses Group Relative Policy Optimization (GRPO) with Selective Sample Replay (SSR) and Forced Rethinking to train the model. VL-Rethinker achieves state-of-the-art scores on MathVista (80.3%), MathVerse (61.8%), and MathVision (43.9%). The method provides AI practitioners with an RL approach for enhancing VLM reasoning without reliance on distillation, offering techniques such as SSR to stabilize training and Forced Rethinking to promote self-reflection. |
| FUSION: Fully Integration of Vision-Language Representations for Deep | ||
| Cross-Modal Understanding (Read more on arXiv or HuggingFace) | Jingzhou Chen, conghui, jingwei-xu-00, Balalauuoo, starriver030515 | i) The paper introduces FUSION, a family of multimodal large language models (MLLMs) designed for deep, dynamic integration of vision and language. ii) The research aims to enhance cross-modal understanding by achieving a fully vision-language aligned and integrated paradigm within MLLMs. iii) The methodology incorporates Text-Guided Unified Vision Encoding, Context-Aware Recursive Alignment Decoding, and a Dual-Supervised Semantic Mapping Loss. iv) Experiments show FUSION 3B outperforms Cambrian-1 8B and Florence-VL 8B on most benchmarks, even when limited to 300 vision tokens. v) FUSION’s approach provides AI practitioners with a strategy for significantly improving MLLM performance with fewer vision tokens by focusing on deep modality integration. |
| Iterative Self-Training for Code Generation via Reinforced Re-Ranking (Read more on arXiv or HuggingFace) | Valentin Malykh, Ivan Sedykh, Nikita Sorokin | Iterative self-training is used to refine code generation through reinforced re-ranking with Proximal Policy Optimization (PPO). The research aims to improve code generation quality and re-ranking accuracy of decoder-based models through iterative self-training using PPO to optimize a reward/re-ranking model. The methodology involves supervised fine-tuning, reward model training, PPO-based code generation, and iterative refinement using hard negative mining. Results demonstrate a 13.4B parameter model outperforming a 33B parameter model on the MultiPL-E dataset in code generation quality and reaching comparable to GPT-4 performance in code generation, while being three times faster. For AI practitioners, the study presents a method for developing more efficient code generation models by focusing on a robust reward mechanism within a self-training framework. |
| Mavors: Multi-granularity Video Representation for Multimodal Large | ||
| Language Model (Read more on arXiv or HuggingFace) | kugwzk, zhenhuawu, UnnamedWatcher, CheeryLJH, DogNeverSleep | Mavors introduces a multi-granularity video representation framework for multimodal large language models (MLLMs) aimed at efficient long-context video understanding. The main objective is to balance computational efficiency with the retention of fine-grained spatio-temporal patterns, addressing information loss from methods like sparse sampling or token compression. Mavors uses an Intra-chunk Vision Encoder (IVE) for high-resolution spatial features within video segments and an Inter-chunk Feature Aggregator (IFA) with chunk-level rotary position embeddings (C-ROPE) for temporal coherence across segments. Results demonstrate Mavors-7B’s strong performance, achieving a score of 39.4 on the DREAM-1K video captioning benchmark, significantly outperforming many comparable 7B models on tasks requiring fine-grained spatio-temporal reasoning. For AI practitioners, Mavors offers an approach to enhance MLLM capabilities for long video analysis by preserving detailed spatio-temporal information more effectively than common sampling or compression strategies, crucial for applications needing nuanced video understanding. |
| AgentRewardBench: Evaluating Automatic Evaluations of Web Agent | ||
| Trajectories (Read more on arXiv or HuggingFace) | dongchans, arkilpatel, ncmeade, kazemnejad, xhluca | This paper introduces AGENTREWARDBENCH, a benchmark designed to evaluate the automatic evaluation of web agent trajectories by LLM judges. The main objective is to assess the effectiveness of LLMs in judging web agent success compared to expert human annotations, addressing limitations of rule-based and manual evaluations. The methodology involved collecting 1302 trajectories from 4 LLMs across 5 web environments, annotating each by experts for success, side effects, and repetition, and then using this dataset to evaluate 12 different LLM judges and existing rule-based methods. Primary results indicate that no single LLM judge performs best across all benchmarks, the best judges achieve less than 70% precision against expert labels, and official rule-based methods significantly underestimate agent success rates (55.9% recall). The principal implication for AI practitioners is that current automatic evaluation methods, including LLM judges, are not yet reliable enough for high-fidelity assessment or reward modeling, necessitating the development of more accurate automatic evaluation techniques for web agents. |
| S1-Bench: A Simple Benchmark for Evaluating System 1 Thinking Capability | ||
| of Large Reasoning Models (Read more on arXiv or HuggingFace) | Tingwen Liu, Xinghua Zhang, Starrrrrry, ShuaiyiNie, WYRipple | i) S1-Bench is introduced as a benchmark to evaluate Large Reasoning Models’ (LRMs) system 1 thinking capabilities, contrasting with their prevalent system 2 reliance. ii) The research aims to assess LRMs’ performance on simple, intuitive tasks better suited for system 1 processing to understand the effects of over-reliance on system 2. iii) The methodology involves constructing a dataset of simple, diverse questions across multiple domains and languages and evaluating 22 LRMs on this benchmark. iv) Results indicate that LRMs exhibit lower efficiency tendencies, generating outputs averaging 15.5 times longer than traditional small LLMs, and accuracy degradation on simple questions. v) This highlights the need for substantial development in LRMs to achieve balanced dual-system thinking capabilities adaptable to task complexity for AI practitioners. |
| Have we unified image generation and understanding yet? An empirical | ||
| study of GPT-4o’s image generation ability (Read more on arXiv or HuggingFace) | Ning Li, cuijiaxing, zhangjingran | i) This paper empirically evaluates GPT-4o’s image generation capabilities across global instruction adherence, fine-grained editing precision, and post-generation reasoning. ii) The main objective is to assess whether GPT-4o achieves world knowledge-informed semantic synthesis during image generation. iii) The methodology involves designing three types of prompts: global instruction, fine-grained editing, and post-generation reasoning, to test specific aspects of image generation. iv) Results show GPT-4o defaults to literal interpretations, inconsistently applies knowledge constraints, and struggles with conditional reasoning tasks. v) The principal implication is that GPT-4o has significant limitations in dynamically integrating knowledge into its image generation process, necessitating more robust benchmarks for reasoning-aware multimodal generation. |
| DUMP: Automated Distribution-Level Curriculum Learning for RL-based LLM | ||
| Post-training (Read more on arXiv or HuggingFace) | zwt123home123, timecuriosity, gfcui, ztwang | i) The paper introduces DUMP, an automated distribution-level curriculum learning framework for reinforcement learning-based post-training of large language models. ii) The research aims to dynamically schedule training across heterogeneous data distributions to optimize learning efficiency in LLMs. iii) The methodology employs Upper Confidence Bound (UCB) scores based on expected absolute advantage to adaptively adjust sampling probabilities for different distributions. iv) Experiments on logic reasoning datasets show that DUMP significantly improves convergence speed and final performance, achieving a reward of over 0.5 in the 9-character K&K puzzles distribution, while the uniform sampling baseline remained below 0.0. v) The principal implication is that AI practitioners can utilize DUMP to improve the efficiency and effectiveness of RL-based LLM post-training by dynamically prioritizing learnable data distributions. |
| SocioVerse: A World Model for Social Simulation Powered by LLM Agents | ||
| and A Pool of 10 Million Real-World Users (Read more on arXiv or HuggingFace) | milesz7777, tangshiping, SimingChen, libo-ca, Lishi0905 | i) SocioVerse is presented as an LLM-agent-driven world model for social simulation. ii) The research aims to address alignment challenges in social simulation across environment, users, interaction, and behavior. iii) The methodology involves a framework with four alignment components and a user pool of 10 million real individuals derived from social media data. iv) Experiments across politics, news, and economics domains demonstrated SocioVerse’s ability to reflect population dynamics, with presidential election prediction achieving over 90% accuracy in state voting results. v) The study indicates a need for careful selection of underlying LLMs to optimize simulation precision across different social scenarios for AI practitioners. |
| Breaking the Data Barrier – Building GUI Agents Through Task | ||
| Generalization (Read more on arXiv or HuggingFace) | jxhe, QiushiSun, changma, heroding77, leoozy | i) This paper investigates the effectiveness of mid-training Vision Language Models (VLMs) on reasoning-intensive tasks for improved generalization in GUI agent planning. ii) The research aims to determine how incorporating various instruction-tuning tasks during the mid-training phase of VLMs facilitates generalization to GUI planning scenarios, addressing the scarcity of high-quality trajectory data. iii) The methodology involves training VLMs on a range of readily available instruction-tuning datasets, including GUI perception, multimodal reasoning, and textual reasoning, followed by fine-tuning on GUI trajectory data. iv) The primary results indicate that task generalization proves highly effective, with multimodal mathematical reasoning enhancing performance on AndroidWorld by an absolute 6.3%; text-only mathematical data significantly boosts GUI web agent performance, achieving a 5.6% improvement on WebArena and a 5.4% improvement on AndroidWorld. v) The principal implication for AI practitioners is that incorporating specific, readily available reasoning tasks into the mid-training of VLMs can substantially improve the performance and generalization capabilities of GUI agents, offering a practical approach to addressing data scarcity challenges in this domain; The work also identifies an optimized dataset mixture called GUIMid which achieves absolute gains of 8.0% on WebArena and 12.2% on AndroidWorld. |
| TinyLLaVA-Video-R1: Towards Smaller LMMs for Video Reasoning (Read more on arXiv or HuggingFace) | Lei Huang, Wenjun Wu, wenzz1, Zhang199 | TinyLLaVA-Video-R1 explores reasoning in small vision-language models (VLMs) for video understanding. The research investigates how reinforcement learning (RL) can improve reasoning capabilities in smaller VLMs using general Video-QA datasets. The GRPO algorithm was applied to TinyLLaVA-Video with modifications to the reward structure, including a continuous length reward and penalties for incorrect answers. TinyLLaVA-Video-R1 achieves 49.5 on MVBench, improving reasoning with fewer parameters. The work demonstrates that RL can elicit emergent reasoning abilities like self-verification in small-scale VLMs, suggesting avenues for improving video reasoning with limited computational resources. |
| LLM-SRBench: A New Benchmark for Scientific Equation Discovery with | ||
| Large Language Models (Read more on arXiv or HuggingFace) | Khoa D Doan, Amir Barati Farimani, Ngoc-Hieu Nguyen, mkmeidani, parshinsh | i) LLM-SRBench, a new benchmark, is introduced for evaluating scientific equation discovery using Large Language Models (LLMs). ii) The research aims to provide a rigorous benchmark that avoids memorization effects and properly assesses the equation discovery capabilities of LLMs. iii) The methodology involves creating a dataset with 239 challenging problems across four scientific domains, utilizing both LSR-Transform (alternative mathematical representations) and LSR-Synth (synthetic problems) categories. iv) Experimental results demonstrate that the best-performing system achieves only 31.5% symbolic accuracy across the benchmark. v) This benchmark highlights the limitations of current LLMs in scientific equation discovery, suggesting AI practitioners need to develop more robust methods to leverage LLMs for complex scientific reasoning tasks that go beyond memorization. |
| EmoAgent: Assessing and Safeguarding Human-AI Interaction for Mental | ||
| Health Safety (Read more on arXiv or HuggingFace) | Edify-Kd2024, yaozixin, YimingWang, ChrisJuan, yinghuihe | i) EmoAgent is a multi-agent AI framework for evaluating and mitigating mental health risks in human-AI interactions within character-based chatbots. ii) The research aims to assess and safeguard human-AI interactions for mental health safety, particularly for vulnerable users. iii) EmoAgent employs a simulated environment (EmoEval) using clinically validated psychological assessment tools and a real-time safeguard agent (EmoGuard) that monitors and provides corrective feedback. iv) Experiments show that emotionally engaging dialogues can lead to mental state deterioration in vulnerable users in more than 34.4% of simulations; EmoGuard reduces these deterioration rates significantly. v) AI practitioners should be aware that emotionally engaging AI dialogues can lead to mental state deterioration in vulnerable users; and real-time monitoring and corrective feedback are crucial for ensuring safety in AI-human interactions. |
| The AI Scientist-v2: Workshop-Level Automated Scientific Discovery via | ||
| Agentic Tree Search (Read more on arXiv or HuggingFace) | Chris Lu, Shengran Hu, Robert Tjarko Lange, conglu, yyamada | i) This paper introduces THE AI SCIENTIST-v2, an AI agentic system for automated scientific discovery, improving upon its predecessor. ii) The research aims to develop an end-to-end system capable of autonomously producing scientific manuscripts acceptable for peer review. iii) The methodology involved agentic tree search managed by an experiment manager agent, Vision-Language Model (VLM) feedback loops, and parallel experiment execution. iv) The system generated a manuscript that achieved an average reviewer score of 6.33 at an ICLR workshop, exceeding the average human acceptance threshold. v) This work demonstrates the potential for AI to conduct all aspects of scientific research, enabling unprecedented scalability in research productivity. |
| Executable Functional Abstractions: Inferring Generative Programs for | ||
| Advanced Math Problems (Read more on arXiv or HuggingFace) | Zaid Khan, mohitbansal, j-min, archiki, esteng | i) The paper introduces EFAGen, a framework for automatically constructing Executable Functional Abstractions (EFAs) for advanced math problems by inferring generative programs from static examples. ii) The research aims to automate the construction of EFAs for advanced math problems, operationalizing this as a program synthesis task. iii) EFAGen conditions a large language model (LLM) on a seed math problem and its solution to generate candidate EFA programs, using executable unit tests as verifiable rewards to train the LLM. iv) Experiments show that EFAs constructed by EFAGen remain faithful to seed problems, produce learnable problem variations, infer EFAs across multiple diverse sources of competition-level math problems, and EFA-based augmentation yields consistent improvements on MATH-500, where Pass@1 improves by +1.9 in the 33% seed setting. v) The principal implication is a scalable approach for generating diverse and verifiable math problem variants, aiding in data augmentation, model stress-testing, and curriculum learning for improving mathematical reasoning in AI systems. |
| How new data permeates LLM knowledge and how to dilute it (Read more on arXiv or HuggingFace) | Nolan Andrew Miller, Andrey Zhmoginov, Chen Sun, gozzo87, mendor | i) This paper investigates how individual text samples update LLM knowledge, introducing a “priming” effect where new facts inappropriately generalize to unrelated contexts. ii) The research aims to understand and predict how new information propagates through an LLM’s knowledge base, leading to both generalization and problematic hallucination. iii) The methodology involves a novel dataset, “Outlandish”, composed of 1320 diverse text samples designed to systematically probe knowledge permeation, along with measuring token probabilities before and after learning. iv) The study found that the degree of priming can be predicted by measuring the token probability of key words before learning, and developed two techniques, “stepping-stone” text augmentation and “ignore-k” update pruning, reducing priming effects by 50-95%. v) The findings offer AI practitioners empirical insights and practical tools for improving the specificity of knowledge insertion in language models and reducing undesirable knowledge permeation. |
| VisuoThink: Empowering LVLM Reasoning with Multimodal Tree Search (Read more on arXiv or HuggingFace) | QipengGuo, alphadl, ngc7293, sinwang, LibraTree | VisuoThink introduces a multimodal tree search framework to enhance Large Vision-Language Model (LVLM) reasoning by interleaving visual and textual information dynamically. The research aims to improve LVLM performance on complex reasoning tasks by integrating visual aids and step-by-step thinking through a predictive rollout search mechanism. The methodology involves a vision-text interleaved reasoning framework coupled with a look-ahead tree search algorithm that explores multiple reasoning paths. Experiments show VisuoThink achieves an accuracy of 48.5% on Geomeverse, a 21.8% improvement over the state-of-the-art baseline without fine-tuning, particularly excelling in problems requiring multi-step visual reasoning. This framework offers AI practitioners an effective method for improving reasoning capabilities in vision-language models without requiring model retraining. |
| M1: Towards Scalable Test-Time Compute with Mamba Reasoning Models (Read more on arXiv or HuggingFace) | Daniele Paliotta, tridao, voidptr74, xu3kev, JunxiongWang | i) The paper introduces M1, a hybrid Mamba-based reasoning model, that exhibits efficient test-time compute scaling. ii) The research aims to develop a scalable reasoning model that can leverage increased test-time computation for improved performance on mathematical tasks. iii) The methodology includes distilling a Transformer model into a Mamba architecture, followed by supervised fine-tuning on math datasets and reinforcement learning training with GRPO. iv) M1 achieves performance comparable to DeepSeek-R1-Distill-Qwen-1.5B on MATH500 (82) and AIME25 (22) benchmarks, while demonstrating over 3x faster inference throughput compared to similarly-sized transformer models using vLLM. v) M1 offers AI practitioners an efficient alternative to Transformers for reasoning tasks, enabling greater test-time compute scaling through faster inference and potentially improving performance via self-consistency or chain-of-thought approaches under fixed time budgets. |
| LLM Can be a Dangerous Persuader: Empirical Study of Persuasion Safety | ||
| in Large Language Models (Read more on arXiv or HuggingFace) | Xinyi Zhang, sarvech123, aneverfull, Zhiyang03, mqliu | i) This paper introduces PERSUSAFETY, a framework for assessing persuasion safety in Large Language Models (LLMs). ii) The primary objective is to investigate whether LLMs reject unethical persuasion tasks and avoid unethical strategies, considering influencing factors like personality traits and external pressures. iii) The methodology involves creating persuasion scenes, simulating persuasive conversations between LLMs, and assessing safety via refusal rates and unethical strategy usage. iv) Experiments across 8 LLMs revealed that most models fail to consistently refuse harmful persuasion tasks and employ unethical strategies; Claude-3.5-Sonnet, while exhibiting strong refusal rates, showed high unethical strategy usage when engaged. v) AI practitioners should be aware that current safety alignment techniques in LLMs may not prevent the use of unethical strategies once the model is engaged, necessitating further research into safety alignment in goal-driven conversations. |
| DeepSeek vs. o3-mini: How Well can Reasoning LLMs Evaluate MT and | ||
| Summarization? (Read more on arXiv or HuggingFace) | Christoph Leiter, Yanran Chen, Ran Zhang, Sotaro Takeshita, Daniil Larionov | i) The paper systematically compares the performance of reasoning-enabled LLMs against non-reasoning counterparts in evaluating machine translation (MT) and text summarization (TS) tasks. ii) The main research questions are whether reasoning models improve upon conventional models in NLG evaluation and how effectively distillation preserves evaluation capabilities while reducing computational costs. iii) The methodology involves evaluating eight different models, including reasoning-based LLMs, distilled variants, and conventional LLMs, using GEMBA-MQM for MT evaluation and G-Eval for TS evaluation, across the WMT23 and SummEval benchmarks. iv) Primary results indicate that OpenAI’s o3-mini models show performance improvements with increased reasoning intensity, achieving the highest overall Eval4NLP scores of 0.644 and 0.645, while DeepSeek-R1 generally underperforms compared to its non-reasoning variant. v) A principal implication for AI practitioners is that the efficacy of reasoning capabilities for NLG evaluation is highly architecture-dependent, and distillation of reasoning capabilities maintains reasonable performance in medium-sized models but degrades substantially in smaller variants, requiring careful consideration of model architecture and task alignment. |
| MDK12-Bench: A Multi-Discipline Benchmark for Evaluating Reasoning in | ||
| Multimodal Large Language Models (Read more on arXiv or HuggingFace) | Jiaxin Ai, Zhaopan Xu, Xiaopeng Peng, Fanrui Zhang, Pengfei Zhou | i) MDK12-Bench is introduced as a new multi-disciplinary benchmark for evaluating multimodal reasoning in large language models (MLLMs) using K-12 level examinations. ii) The research aims to address the limitations of existing benchmarks by providing a more comprehensive evaluation of MLLMs’ reasoning capabilities across multiple disciplines. iii) The methodology involves curating a dataset of 140K reasoning instances spanning six disciplines, annotating instances with knowledge points, and developing a dynamic evaluation framework to mitigate data contamination through bootstrapped unseen data. iv) Experiments showed that Gemini2-thinking achieves the highest overall accuracy of 59.4% on the MDK12-Mini dataset, and models demonstrate sensitivity to combined textual and visual bootstrapping. v) AI practitioners can utilize MDK12-Bench to identify specific knowledge gaps in MLLMs, facilitating targeted improvements in multimodal reasoning capabilities, particularly in areas such as contextual comprehension and resistance to data contamination. |
Papers for 2025-04-14
| Title | Authors | Summary |
|---|---|---|
| Seaweed-7B: Cost-Effective Training of Video Generation Foundation Model (Read more on arXiv or HuggingFace) | Zhijie Lin, Ceyuan Yang, Team Seawead, zhenheny, lingff | This paper details a cost-effective strategy for training Seaweed-7B, a 7-billion parameter video generation foundation model using moderate compute. The primary objective was to demonstrate that a medium-sized video generation model can achieve competitive performance compared to much larger models trained with significantly greater computational resources. Key methodologies involved training a novel 64x compression Variational Autoencoder (VAE) and a hybrid-stream Diffusion Transformer (DiT) from scratch on curated data using 665,000 H100 GPU hours, employing multi-stage training, SFT, DPO, and infrastructure optimizations like 3D parallelism and Multi-Level Activation Checkpointing (MLAC). Seaweed-7B achieved competitive performance, ranking second in image-to-video generation Elo ratings (1047 Elo, 58% win rate) against models like Sora and Wan 2.1, and its VAE obtained state-of-the-art reconstruction (e.g., 0.0391 LPIPS on UCF-101). Its distilled version requires only 12 NFEs for inference, 62 times faster than Wan 2.1 (100 NFEs). For AI practitioners, this work implies that careful design choices in data curation, VAE/DiT architecture, and training/inference optimization enable the development of highly competitive, cost-effective video generation models without necessarily resorting to massive parameter counts. |
| GigaTok: Scaling Visual Tokenizers to 3 Billion Parameters for | ||
| Autoregressive Image Generation (Read more on arXiv or HuggingFace) | Jiashi Feng, Zilong Huang, Jun Hao Liew, XihuiLiu, YuuTennYi | GigaTok introduces a 3 billion parameter visual tokenizer for autoregressive image generation that improves reconstruction, generation, and representation quality simultaneously during scaling. The research aims to overcome the common dilemma where scaling visual tokenizers improves reconstruction but degrades downstream generation performance. Key methods involve semantic regularization using features from a pre-trained DINOv2 model, employing 1D Q-Former based tokenizers, prioritizing decoder scaling in an asymmetric architecture, and using entropy loss for billion-scale training stability. The proposed 2.9B parameter GigaTok, when paired with a 1.4B AR model, achieves state-of-the-art autoregressive generation performance with a gFID of 1.98* on ImageNet 256x256. AI practitioners can apply semantic regularization and the identified scaling practices (1D tokenizers, asymmetric scaling, entropy loss) to develop larger, more effective visual tokenizers for generative models without sacrificing downstream performance due to increased latent space complexity. |
| MineWorld: a Real-Time and Open-Source Interactive World Model on | ||
| Minecraft (Read more on arXiv or HuggingFace) | Yushu Jiang, Haoyu Wu, Tianyu He, Yang Ye, Junliang Guo | MineWorld introduces a real-time, open-source, interactive world model for Minecraft based on an autoregressive Transformer. The primary objective is to develop an efficient and controllable world model capable of real-time interaction by predicting future game states conditioned on past states and actions. Key methodology involves tokenizing visual game states and player actions, feeding them interleaved into a Transformer trained via next-token prediction, and employing a novel parallel decoding algorithm for inference acceleration. Results demonstrate the model’s efficacy, with the 1.2B parameter version achieving 3.01 FPS, a discrete action F1 score of 0.73, and camera control L1 loss of 1.02, significantly outperforming diffusion-based baselines while the parallel decoding provides over 3x speedup. For AI practitioners, MineWorld offers a validated open-source framework and an efficient parallel decoding technique for building fast, interactive simulators essential for agent training and human-AI interaction in complex environments. |
| PixelFlow: Pixel-Space Generative Models with Flow (Read more on arXiv or HuggingFace) | Ping Luo, Peize Sun, Shilong Zhang, Chongjian Ge, Shoufa Chen | i) PixelFlow, a novel image generation model, performs image generation directly in raw pixel space through cascade flow modeling. ii) The research aims to develop an end-to-end trainable image generation model operating directly in pixel space, avoiding the need for pre-trained VAEs and cascaded upsampling. iii) PixelFlow employs a cascade flow modeling strategy, operating on multi-scale samples across cascading resolutions and using Flow Matching for velocity prediction. iv) PixelFlow achieves an FID of 1.98 on the 256x256 ImageNet class-conditional image generation benchmark. v) The PixelFlow framework provides AI practitioners with a simpler, end-to-end trainable alternative to latent-space diffusion models, enabling efficient pixel-space image generation with competitive performance. |
| SQL-R1: Training Natural Language to SQL Reasoning Model By | ||
| Reinforcement Learning (Read more on arXiv or HuggingFace) | Ran Chen, Xuhui Jiang, Chengjin Xu, Peixian Ma, ZhuangXialie | i) This paper introduces SQL-R1, an NL2SQL reasoning model trained via reinforcement learning to improve performance in complex scenarios. ii) The research aims to enhance NL2SQL inference performance in complex database scenarios using reinforcement learning. iii) The methodology involves training a NL2SQL model using reinforcement learning with a specialized reward function and a cold start strategy based on supervised fine-tuning. iv) SQL-R1 achieves execution accuracy of 88.6% on the Spider benchmark and 66.6% on the BIRD benchmark using a 7B base model. v) AI practitioners can leverage the SQL-R1 model to achieve competitive accuracy in NL2SQL tasks with limited data and improved reasoning capabilities, demonstrating the potential of RL in optimizing NL2SQL performance. |
| FlexIP: Dynamic Control of Preservation and Personality for Customized | ||
| Image Generation (Read more on arXiv or HuggingFace) | Kaiwen Xiao, Yanning Zhou, Haonan Lin, DevLinyan | FlexIP is introduced as a novel framework for decoupling identity preservation and personalized editing in image generation. The research aims to enable flexible, parameterized control during inference through dynamic tuning of the weight adapter in generative models. FlexIP uses a dual-adapter architecture comprising a Personalization Adapter and a Preservation Adapter, coupled with a dynamic weight gating mechanism to balance identity retention and stylistic variation. Experiments demonstrate that FlexIP achieves a 61.4% controllability (Flex score) and 76.8% ID-Pres score. The framework offers AI practitioners a robust and flexible solution for subject-driven image generation by enabling continuous parametric control of the preservation-editability trade-off. |
| In-2-4D: Inbetweening from Two Single-View Images to 4D Generation (Read more on arXiv or HuggingFace) | Ali Mahdavi-Amiri, Hao Zhang, Daniel Cohen-Or, Sauradip Nag | i) This paper introduces In-2-4D, a method for generating 4D (3D object + motion) interpolations from two single-view images. ii) The primary objective is to generate and reconstruct a smooth 4D motion sequence given only start and end state images of an object. iii) The method uses a hierarchical approach involving video interpolation models, keyframe selection based on motion and appearance analysis, 3D Gaussian Splatting for static 3D representation, and dynamic Gaussian generation via a deformation field optimized with multi-view diffusion priors. iv) The method achieves improved performance on a newly introduced I4D-15 benchmark, outperforming baselines in terms of appearance (LPIPS: 0.103, FVD: 679.23) and geometry (SI-CD: 22.67, CD: 0.59), with user studies indicating a preference for the generated 4D motion quality (1.29 rating). v) The approach provides AI practitioners with a method for generating dynamic 3D content from minimal input, enabling applications in content creation and animation by requiring less data and allowing for diverse motion synthesis. |
| ModernBERT or DeBERTaV3? Examining Architecture and Data Influence on | ||
| Transformer Encoder Models Performance (Read more on arXiv or HuggingFace) | Djamé Seddah, Benoît Sagot, Wissam Antoun | This paper conducts a controlled comparison of ModernBERT and DeBERTaV3 architectures by pretraining them on identical French datasets. The objective is to disentangle architectural advantages from training data differences in explaining performance variations between these transformer encoder models. The methodology involves pretraining French ModernBERT on the same 275B token dataset as CamemBERTaV2 (a French DeBERTaV3 model) and evaluating on French NER, QA, and classification tasks. Results show DeBERTaV3 (CamemBERTaV2) achieves superior benchmark performance (e.g., 83.04 F1 QA vs. 81.34 F1 for ModernBERT-CV2) and sample efficiency when data is controlled, while ModernBERT offers faster training/inference speeds. For AI practitioners, this implies a trade-off: DeBERTaV3 yields higher accuracy, whereas ModernBERT provides better computational efficiency, highlighting the need to evaluate models under shared data conditions for fair comparison. |
| Pangu Ultra: Pushing the Limits of Dense Large Language Models on Ascend | ||
| NPUs (Read more on arXiv or HuggingFace) | Xueyu Wu, Yehui Tang, Kaikai Song, Wenyong Huang, Yichun Yin | Pangu Ultra is a 135B parameter dense Transformer LLM trained on 13.2 trillion tokens using 8,192 Ascend NPUs. The primary objective was to explore the performance limits of large-scale dense LLMs and address the associated training stability and system efficiency challenges on Ascend hardware. Methodology involved proposing depth-scaled sandwich normalization and TinyInit for stable training of the 94-layer model, alongside system optimizations like NPU Fusion Attention (NFA) and MC2 for efficient training, achieving over 50% MFU. Results show Pangu Ultra significantly outperforms comparable dense models like Llama 3.1 405B (e.g., 90.3% vs 72.5% on C-Eval) and achieves competitive results against larger sparse MoE models such as DeepSeek-R1. For AI practitioners, this work validates the capability of Ascend NPUs for efficiently training >100B parameter dense models and demonstrates that optimized dense architectures can achieve state-of-the-art performance comparable to sparse models, potentially offering simpler inference deployment. |
| SAEs Can Improve Unlearning: Dynamic Sparse Autoencoder | ||
| Guardrails for Precision Unlearning in LLMs (Read more on arXiv or HuggingFace) | Virginia Smith, Mona Diab, Jacopo Bonato, Aashiq Muhamed | i) This paper introduces Dynamic SAE Guardrails (DSG), an activation-based method using Sparse Autoencoders (SAEs) that significantly improves precision unlearning in LLMs compared to gradient-based approaches. ii) The primary objective is to develop an unlearning technique that effectively removes targeted knowledge from LLMs while preserving general utility, addressing limitations of existing methods like high cost, instability, and poor data efficiency. iii) DSG employs principled feature selection using Fisher Information approximation via squared SAE activations to identify forget-relevant features and uses a dynamic, input-dependent classifier with a statistically determined threshold to conditionally clamp these features during inference. iv) Experiments demonstrate DSG substantially outperforms baseline methods, achieving a superior forget-utility trade-off by reducing WMDP-Bio accuracy to 29.64% (vs. 50.00% for the next best, RMU) while maintaining high MMLU accuracy (99.34%) and offering better computational efficiency, hyperparameter stability, and sequential unlearning performance. v) For AI practitioners, DSG provides a more computationally efficient, stable, interpretable, and data-efficient mechanism for targeted knowledge removal, enhancing LLM safety, privacy, and maintenance capabilities without requiring gradient computations during intervention. |
| Do PhD-level LLMs Truly Grasp Elementary Addition? Probing Rule Learning | ||
| vs. Memorization in Large Language Models (Read more on arXiv or HuggingFace) | Zhenzhong Lan, Renjun Xu, Yu Lu, Yang Yan | This paper probes whether Large Language Models genuinely understand elementary addition principles or rely on pattern memorization. The research investigates if LLMs learn generalizable arithmetic rules or merely exploit statistical patterns when performing two-integer addition. Methodology involves evaluating LLMs on addition tasks using standard digits versus isomorphic symbolic mappings, testing commutativity (A+B vs B+A), and analyzing performance scaling with digit count. Results show that while models achieve high numerical accuracy (73.8-99.8%), performance collapses to ≤7.5% under symbolic mapping, indicating a failure to generalize learned rules beyond familiar patterns. The principal implication for AI practitioners is that current LLMs heavily rely on memorization over true rule learning for arithmetic, necessitating more rigorous evaluation methods to assess genuine mathematical reasoning capabilities before deployment. |
| CoRAG: Collaborative Retrieval-Augmented Generation (Read more on arXiv or HuggingFace) | Virginia Smith, Mona Diab, Aashiq Muhamed | i) The paper introduces CoRAG, a framework for collaborative retrieval-augmented generation. ii) The research investigates how to effectively train RAG models in collaborative settings with shared passage stores, addressing the challenges of data heterogeneity and client incentives. iii) The methodology involves developing a novel benchmark, CRAB, for homogeneous open-domain question answering and comparing CoRAG against parametric collaborative learning and local RAG baselines using FedAvg. iv) Experiments on CRAB show CoRAG consistently outperforms baselines in few-shot settings, achieving a 33.8% improvement over local RAG at 16-shot; further analysis reveals that relevant passages are crucial, hard negatives are detrimental, while irrelevant passages can even be beneficial for model performance. v) AI practitioners can leverage CoRAG to improve model performance in low-resource, collaborative knowledge-intensive tasks by careful curation of the shared passage store, balancing the inclusion of relevant and irrelevant passages while minimizing hard negatives. |
| InteractVLM: 3D Interaction Reasoning from 2D Foundational Models (Read more on arXiv or HuggingFace) | Cordelia Schmid, Omid Taheri, Shashank Tripathi, Dimitrije Antić, saidwivedi | i) InteractVLM estimates 3D human-object contact points from single images by leveraging 2D vision-language models. ii) The research objective is to accurately estimate 3D contact points between humans and objects from in-the-wild 2D images to improve joint reconstruction without relying on extensive 3D contact annotations. iii) The methodology involves a “Render-Localize-Lift” module using multi-view rendering, a novel multi-view localization model (MV-Loc), and fine-tuning a VLM with limited 3D contact data. iv) InteractVLM achieves a 20.6% improvement in F1 score over existing methods for binary human contact prediction on the DAMON dataset. v) InteractVLM enables AI practitioners to improve 3D human-object interaction reconstruction from 2D images using predicted contact points and minimal 3D annotation, improving the realism and accuracy of HOI reconstruction. |
Papers for 2025-04-11
| Title | Authors | Summary |
|---|---|---|
| Kimi-VL Technical Report (Read more on arXiv or HuggingFace) | dongliangwang, congcongwang, DuChenZhuang, tzzcl, xingbowei | Kimi-VL is presented as an efficient open-source Mixture-of-Experts (MoE) vision-language model (VLM). The objective is to develop a VLM offering advanced multimodal reasoning, long-context understanding (128K), and strong agent capabilities while activating only 2.8B parameters in its language decoder. Methodology involves pairing a native-resolution MoonViT vision encoder with an MoE language model (Moonlight), trained through multi-stage pre-training, joint supervised fine-tuning (SFT), and enhanced with long-CoT SFT and reinforcement learning (RL) for the Kimi-VL-Thinking variant. Primary results show Kimi-VL competes effectively with larger VLMs across various benchmarks, while the Kimi-VL-Thinking variant achieves 61.7 on MMMU and 36.8 on MathVision, demonstrating strong long-horizon reasoning with its compact 2.8B activated LLM parameters. For AI practitioners, this research indicates the viability of using MoE architectures and native-resolution vision encoders to create parameter-efficient VLMs capable of complex multimodal reasoning, long-context processing, and agentic behavior. |
| VCR-Bench: A Comprehensive Evaluation Framework for Video | ||
| Chain-of-Thought Reasoning (Read more on arXiv or HuggingFace) | lovesnowbest, Lin-Chen, Osilly, ChthollyTree, yukunqi | VCR-Bench introduces a novel benchmark for comprehensively evaluating video Chain-of-Thought (CoT) reasoning capabilities in Large Vision-Language Models (LVLMs). The primary objective is to rigorously assess the entire reasoning process, differentiating failures originating from perception versus reasoning deficits, which current benchmarks inadequately address. Methodology involves a new dataset (VCR-Bench) with 859 videos and 1,034 QA pairs, featuring manually annotated, stepwise CoT rationales tagged for perception/reasoning, and a CoT score derived from recall/precision evaluation of these steps. Experiments reveal significant limitations in existing LVLMs, with the top-performing model achieving only a 62.8% CoT score and 56.7% accuracy, and most models exhibiting lower performance on perception steps compared to reasoning steps. For AI practitioners, VCR-Bench offers a standardized framework to identify specific weaknesses, particularly in temporal-spatial perception, providing actionable insights for improving LVLMs on complex video reasoning tasks. |
| MM-IFEngine: Towards Multimodal Instruction Following (Read more on arXiv or HuggingFace) | yhcao, sweetFruit, KennyUTC, yuhangzang, ChrisDing1105 | MM-IFEngine introduces a pipeline for generating multimodal instruction-following data and the MM-IFEval benchmark for evaluation. The research objective is to address the scarcity of high-quality training data and the limitations of existing benchmarks for evaluating multimodal instruction following (IF) in MLLMs. Key methodology involves the MM-IFEngine pipeline using LLMs (GPT-4o) for image filtering, task generation, and integrating 32 constraint categories to create the MM-IFInstruct-23k (SFT) and MM-IFDPO-23k (DPO) datasets, alongside the MM-IFEval benchmark featuring hybrid evaluation. Primary results show fine-tuning Qwen2-VL-7B on MM-IFDPO-23k significantly improves IF performance, achieving gains of +10.2% on MM-IFEval and +7.6% on MIA-Bench, while maintaining comparable VQA capabilities. For AI practitioners, this work provides datasets (MM-IFInstruct-23k, MM-IFDPO-23k) and a benchmark (MM-IFEval) to train and rigorously evaluate MLLMs for enhanced instruction adherence, crucial for applications needing precise, constrained multimodal outputs. |
| VisualCloze: A Universal Image Generation Framework via Visual | ||
| In-Context Learning (Read more on arXiv or HuggingFace) | mingming8688, cosumosu25, JonsonYan, RuoyiDu, lzyhha | VisualCloze presents a universal image generation framework leveraging visual in-context learning (ICL) to perform diverse tasks using a unified infilling model approach. Its primary objective is to overcome limitations of language-based instructions and task sparsity by enabling a model to understand and generalize visual tasks from examples. The key methodology involves formulating generation tasks as infilling problems on a grid of concatenated visual prompts and targets, fine-tuning the FLUX.1-Fill-dev model with LoRA on the proposed dense Graph200K dataset. Results demonstrate strong performance on in-domain tasks, generalization to unseen tasks, and task unification, with ICL quantitatively improving results (e.g., reducing Depth-to-Image RMSE from 10.31 to 9.68 using two in-context examples). For AI practitioners, this work implies that visual ICL combined with pre-trained infilling models offers a promising, unified paradigm for building versatile image generation systems that can learn complex visual relationships and adapt to new tasks with fewer explicit instructions compared to purely language-guided or task-specific models. |
| DeepSeek-R1 Thoughtology: Let’s |
parishadbehnam, miladink, vaibhavad, arkilpatel, spaidartaigar | This paper introduces “Thoughtology,” a systematic analysis of the internal reasoning chains (“thoughts”) produced by the Large Reasoning Model (LRM) DeepSeek-R1. The main objective is to characterize DeepSeek-R1’s reasoning patterns, evaluate the impact of thought length and context on performance, and assess its safety and cognitive parallels. Key methodologies include developing a taxonomy for reasoning steps, quantitative evaluation on math (AIME-24, GSM8k, multiplication), long-context (Needle-in-a-Haystack, CHASE-QA/Code), safety (HarmBench), and cognitive/cultural benchmarks. Primary results indicate a consistent reasoning structure but reveal an optimal thought length ‘sweet spot’ beyond which performance declines; notably, DeepSeek-R1 also exhibits significant safety vulnerabilities, responding harmfully to 30.0% of direct HarmBench requests. For AI practitioners, this implies that controlling LRM thought length is crucial for performance and efficiency, yet DeepSeek-R1 lacks inherent mechanisms for this, and its reasoning capabilities introduce new safety risks requiring specific mitigation strategies beyond standard LLM alignment. |
| HoloPart: Generative 3D Part Amodal Segmentation (Read more on arXiv or HuggingFace) | Lp256, zouzx, KevinHuang, bennyguo, yhyang-myron | HoloPart introduces a generative approach for 3D part amodal segmentation, decomposing shapes into complete semantic parts, including occluded geometry. The primary objective is to address the limitations of standard 3D part segmentation by inferring and completing hidden part geometry while maintaining global shape consistency. The key methodology employs a two-stage approach: leveraging existing segmentation for initial surface patches, followed by HoloPart, a novel diffusion-based model using specialized local and context-aware attention mechanisms, to complete these patches into full parts. HoloPart significantly outperforms existing shape completion methods, achieving a mean instance IoU of 0.764 on the ABO benchmark compared to 0.565 for the next best baseline (Finetune-VAE). For AI practitioners, this work offers a tool to generate complete, semantically meaningful 3D parts from potentially incomplete data, enabling more robust downstream applications in 3D content creation, editing, and analysis. |
| C3PO: Critical-Layer, Core-Expert, Collaborative Pathway Optimization | ||
| for Test-Time Expert Re-Mixing (Read more on arXiv or HuggingFace) | Ziyue Li, zhoutianyi, Lzy01241010 | C3PO dynamically optimizes sub-optimal expert pathways in MoE LLMs at test-time to boost performance without retraining. The objective is to improve individual test sample predictions by re-mixing expert routing weights based on pathways from successful reference samples. C3PO employs collaborative optimization using neighbors in an embedding space to define a surrogate objective, focusing optimization on core experts within critical layers using methods like Neighborhood Gradient Descent (NGD). Results show C3PO improves base MoE accuracy by 7-15%; NGD on OLMoE-1B-7B achieved a 9.3% average accuracy increase (69.9% to 79.2%) across six benchmarks, enabling it to outperform 7-9B parameter dense models. AI practitioners can apply C3PO to enhance deployed MoE LLM performance on specific tasks or samples, potentially achieving higher accuracy with smaller models and reduced computational cost during inference. |
| MOSAIC: Modeling Social AI for Content Dissemination and Regulation in | ||
| Multi-Agent Simulations (Read more on arXiv or HuggingFace) | Marzyeh Ghassemi, saadia, elisakreiss, salmannyu, genglinliu | MOSAIC is an open-source multi-agent simulation framework using LLM agents to model social network content diffusion, user engagement, and moderation effects. The primary objective is to analyze LLM agent interactions, model misinformation propagation, and evaluate the efficacy of different content moderation strategies within a simulated social environment. The methodology employs LLM-driven agents (GPT-4o) assigned diverse personas who interact on a directed social graph, with their engagement patterns compared against human participants and tested under no-fact-checking, community-based, third-party, and hybrid moderation conditions. Key results indicate that simulated misinformation does not spread faster than factual content (unlike observed human behavior), and a hybrid fact-checking approach yielded the best balance of precision and recall (F1 score = 0.612) while enhancing factual content engagement. For AI practitioners, this suggests agent-based simulations can test moderation systems, but results must be critically evaluated as agent behavior, potentially influenced by safety training or simulation design, may deviate significantly from human patterns, impacting the direct applicability of findings to real-world platform governance. |
| Scaling Laws for Native Multimodal Models Scaling Laws for Native | ||
| Multimodal Models (Read more on arXiv or HuggingFace) | Joshua Susskind, Matthieu Cord, Victor Guilherme Turrisi da Costa, Enrico Fini, Mustafa Shukor | This paper investigates the scaling laws of native multimodal models (NMMs) trained from scratch, comparing early-fusion, late-fusion, and sparse architectures. The primary objective is to determine if commonly used late-fusion architectures hold an inherent advantage over early-fusion for NMMs and to characterize their scaling properties. The methodology involves training and evaluating 457 NMMs with varying architectures and training mixtures, deriving scaling laws by fitting power-law relationships between validation loss, compute (FLOPs), model parameters (N), and training tokens (D). Results indicate no inherent advantage for late-fusion; early-fusion performs comparably (loss L ∝ C^-0.049 for both) while being more parameter-efficient for compute-optimal models, and sparse Mixture-of-Experts (MoE) significantly improve early-fusion performance. For AI practitioners, this suggests early-fusion NMMs, trained natively and potentially enhanced with MoEs, offer a viable and efficient alternative to late-fusion approaches that rely on separate pre-trained vision encoders, especially at lower parameter counts. |
| SoTA with Less: MCTS-Guided Sample Selection for Data-Efficient Visual | ||
| Reasoning Self-Improvement (Read more on arXiv or HuggingFace) | furongh-lab, kevinlin311tw, linjieli222, zyang39, russwang | This paper presents an MCTS-guided data selection method for efficient visual reasoning self-improvement in VLMs using less data and no knowledge distillation. The main objective is to enhance VLM reasoning capabilities through reinforcement fine-tuning (RFT) using a minimal set of appropriately challenging training samples identified based on difficulty. The key methodology involves repurposing Monte Carlo Tree Search (MCTS) to quantify sample difficulty by measuring the iterations required for the base VLM (Qwen2.5-VL-7B-Instruct) to solve each problem, filtering 70k samples down to 11k. The resulting model, ThinkLite-VL-7B, trained on only 11k samples, achieves 75.1 accuracy on MathVista, surpassing larger models and improving the average benchmark performance of the base VLM by 7% (from 59.69 to 63.89). For AI practitioners, this demonstrates that strategically selecting challenging training data using MCTS for RFT can yield state-of-the-art reasoning performance in VLMs with significantly reduced data requirements, optimizing resource utilization. |
| Towards Visual Text Grounding of Multimodal Large Language Model (Read more on arXiv or HuggingFace) | Franck-Dernoncourt, YfZ, JoshuaGu, zhangry868, MingLiiii | This paper introduces TRIG, a novel task, benchmark (TRIG-Bench), and dataset to evaluate and improve the visual text grounding capabilities of Multimodal Large Language Models (MLLMs) on text-rich document images. The main research objective is to address the poor performance of existing MLLMs in localizing specific text regions within documents that support their generated answers for question-answering tasks. Methodology involved creating the TRIG-Bench benchmark (800 manually verified QA pairs) and a 90k synthetic instruction dataset using an OCR-LLM-human interaction pipeline, and proposing instruction-tuning and embedding-based grounding methods. Evaluation revealed significant limitations in current models on TRIG-Bench (e.g., GPT-4o achieved only 5.28% average pixel-level IoU in the OCR-free setting), while the proposed instruction-tuning method improved performance considerably to 29.98% average IoU after fine-tuning. For AI practitioners, this research provides a standardized benchmark and effective fine-tuning methods to assess and enhance MLLMs’ ability to ground answers in documents, crucial for building more trustworthy and verifiable document understanding systems. |
| MonoPlace3D: Learning 3D-Aware Object Placement for 3D Monocular | ||
| Detection (Read more on arXiv or HuggingFace) | R. Venkatesh Babu, Jogendra Kundu, Sarthak Vora, Srinjay Sarkar, RishubhPar | MonoPlace3D learns realistic, scene-aware 3D object placement to generate effective data augmentations for improving monocular 3D object detection. The main objective is to automatically determine plausible 3D bounding box parameters (position, dimensions, orientation) for inserting synthetic objects into real scenes, addressing a key limitation of prior augmentation methods focused mainly on appearance. The methodology involves training a Scene-Aware Placement Network (SA-PlaceNet) on inpainted scenes to predict a distribution over plausible 3D boxes, then sampling from this distribution and rendering realistic objects using synthetic assets refined by ControlNet. MonoPlace3D significantly improves detection accuracy across multiple detectors and datasets; for example, on KITTI (easy, AP40@IOU=0.7) with MonoDLE, it boosted AP from 17.45% to 22.49% and achieved performance comparable to using the full dataset with only 50% of the data. For AI practitioners, this work demonstrates that focusing on learning physically plausible object placement is crucial for creating highly effective 3D data augmentations, leading to substantial gains in detector performance and data efficiency. |
| Compass Control: Multi Object Orientation Control for Text-to-Image | ||
| Generation (Read more on arXiv or HuggingFace) | R. Venkatesh Babu, Vaibhav Agrawal, sachi1, RishubhPar | Compass Control introduces a method for precise, explicit 3D orientation control of individual objects within text-to-image diffusion models. The primary objective is to enable users to specify the desired 3D orientation for multiple objects in a scene alongside a text prompt, overcoming the limitations and imprecision of text-only control. Key methodology involves predicting orientation-aware ‘compass tokens’ via a lightweight encoder, prepending them to object tokens in the text prompt, and using ‘Coupled Attention Localization (CALL)’ to constrain the cross-attention maps of compass and object tokens to corresponding 2D bounding box regions. The approach achieves superior orientation control, yielding a significantly lower angular error (0.198 radians for single objects, 0.215 for multiple) compared to baselines like LooseControl (0.385 and 0.372 respectively), and generalizes effectively to unseen objects and scenes with more than two objects. For AI practitioners, this provides a user-friendly interface for granular 3D orientation control in generative models using only orientation angles and coarse 2D boxes, enhancing predictability and streamlining creative workflows without requiring dense 3D data. |
| TAPNext: Tracking Any Point (TAP) as Next Token Prediction (Read more on arXiv or HuggingFace) | rgoroshin, apsarath, msajjadi, skoppula, artemZholus | TAPNext reformulates Tracking Any Point (TAP) in video as a sequential masked token decoding problem for online, low-latency tracking. The primary objective is to develop a simpler, more scalable TAP model by removing complex tracking-specific inductive biases and heuristics found in prior work. It employs a causal architecture combining ViT and SSM layers (TRecViT) to jointly process image patch tokens and masked point coordinate tokens, predicting trajectories via token imputation using a classification-based coordinate head. The method achieves state-of-the-art online tracking performance, with BootsTAPNext-B reaching 78.5 Average Jaccard (AJ) on DAVIS First at 256x256 resolution, outperforming previous frame-latency methods while operating purely online. For AI practitioners, TAPNext demonstrates that general-purpose sequence models with minimal task-specific components can achieve SOTA performance in complex correspondence tasks like point tracking, offering a potentially more scalable and easily adaptable approach for applications requiring online video understanding. |
Papers for 2025-04-10
| Title | Authors | Summary |
|---|---|---|
| DDT: Decoupled Diffusion Transformer (Read more on arXiv or HuggingFace) | Weilin Huang, Zhi Tian, lmwang, wangsssssss | This paper introduces the Decoupled Diffusion Transformer (DDT), separating semantic encoding and high-frequency detail decoding. The objective is to resolve the inherent optimization conflict in standard diffusion transformers, thereby accelerating training convergence and improving generation quality. DDT utilizes a distinct condition encoder for semantic extraction and a velocity decoder for detail generation, incorporating representation alignment and trained via linear flow matching. Key results show DDT-XL/2 achieves a state-of-the-art 1.31 FID on ImageNet 256x256 in 256 epochs, indicating approximately 4x faster convergence than prior diffusion transformers like REPA. For AI practitioners, DDT offers a significantly more efficient architecture for training high-fidelity diffusion models and introduces a statistical dynamic programming approach to accelerate inference by sharing encoder computations between steps with minimal performance loss. |
| GenDoP: Auto-regressive Camera Trajectory Generation as a Director of | ||
| Photography (Read more on arXiv or HuggingFace) | lindahua, wetzste1, liuziwei7, jingtan, Dubhe-zmc | This paper introduces GenDoP, an auto-regressive model, and DataDoP, a large-scale dataset, for generating artistic camera trajectories. The research aims to generate controllable, expressive camera trajectories based on multi-modal inputs (text, optional RGBD), addressing limitations in existing methods lacking directorial intent alignment or suffering from instability. The methodology involves creating the DataDoP dataset (29K shots, 11M frames) with detailed motion/directorial captions and developing GenDoP, a decoder-only Transformer that tokenizes camera parameters and generates trajectories auto-regressively. GenDoP significantly outperforms prior methods in text-trajectory alignment, achieving a CLaTr-CLIP score of 36.179 compared to 31.689 for a retrained baseline (Director3D) on the Motion caption task, and also shows superior user-rated alignment, quality, and complexity. For AI practitioners, this work provides a method for generating complex, instruction-following camera paths, enhancing controllability in camera-controlled video generation systems for applications like filmmaking and virtual cinematography. |
| OLMoTrace: Tracing Language Model Outputs Back to Trillions of Training | ||
| Tokens (Read more on arXiv or HuggingFace) | Yensung, sewon, yanaiela, taylorb, liujch1998 | OLMOTRACE is a system that traces language model (LM) outputs back to their training data to understand LM behavior. The research question is how to efficiently trace LM outputs to their full multi-trillion-token training data in real time. The methodology uses an extended version of infini-gram to index the training data and a parallel algorithm to compute matching spans. The system traces LM responses (average 450 tokens) to the training data in 4.5 seconds on average. OLMOTRACE enables AI practitioners to explore the relationship between LM outputs and training data for fact-checking, creativity analysis, and understanding math capabilities. |
| A Unified Agentic Framework for Evaluating Conditional Image Generation (Read more on arXiv or HuggingFace) | Yiyu Wang, Longyue Wang, Xue Yang, Jifang Wang, imryanxu | i) The paper introduces CIGEVAL, a unified agentic framework leveraging large multimodal models (LMMs) for evaluating conditional image generation tasks. ii) The research aims to develop a task-agnostic, reliable, and explainable evaluation metric for conditional image generation. iii) CIGEVAL employs LMMs with a multi-functional toolbox and a fine-grained evaluation framework, synthesizing evaluation trajectories for fine-tuning smaller LMMs. iv) Experiments across seven conditional image generation tasks show CIGEVAL (GPT-40 version) achieves a Spearman correlation of 0.4625 with human assessments. v) CIGEVAL offers AI practitioners a more human-aligned and explainable method for automated evaluation of conditional image generation models, especially in tasks involving multiple conditions, and a pathway for fine-tuning smaller LMMs using synthesized evaluation trajectories for improved performance. |
| Missing Premise exacerbates Overthinking: Are Reasoning Models losing | ||
| Critical Thinking Skill? (Read more on arXiv or HuggingFace) | Ming Li, zhoutianyi, sunlichao137, Fcr09 | i) The paper investigates the effect of missing premises in questions on the response behavior of reasoning Large Language Models (LLMs). ii) The study aims to quantify and analyze the extent to which LLMs exhibit “MiP-Overthinking”, characterized by increased response length and ineffective reasoning on ill-posed questions with missing premises. iii) The methodology involves curating MiP datasets across varying difficulty levels, evaluating LLMs’ response length, accuracy, and abstain rate, and analyzing step-level similarities in reasoning chains. iv) Reasoning models generate responses 2x-4x longer for MiP questions compared to well-defined questions, contradicting test-time scaling law, while non-reasoning models generate responses of similar lengths for both. v) AI practitioners should be aware that current training paradigms for reasoning LLMs insufficiently promote efficient thinking, potentially resulting in resource inefficiencies and the abuse of reasoning patterns when faced with ambiguous input. It is unclear how in-process suspicion metrics are calculated in the paper. |
| FantasyTalking: Realistic Talking Portrait Generation via Coherent | ||
| Motion Synthesis (Read more on arXiv or HuggingFace) | Yunpeng Zhang, Yaqi Fan, Mengchao Wang, fanjiang, wangqiang9 | i) FantasyTalking generates realistic talking portraits from a single image via a dual-stage audio-visual alignment strategy. ii) The research aims to generate high-fidelity and coherent talking portraits with controllable motion dynamics from a static image. iii) The method utilizes a video diffusion transformer model with clip-level and frame-level audio-visual alignment and a facial-focused cross-attention module for identity preservation. iv) The proposed approach achieves state-of-the-art performance, demonstrating improved video quality, temporal consistency, and motion diversity, and achieves an aesthetic score of 0.6183 on the wild talking head dataset. v) AI practitioners can leverage this method for creating more realistic and controllable avatar animations, enhancing applications in gaming, filmmaking, and virtual reality. |
| A Sober Look at Progress in Language Model Reasoning: Pitfalls and Paths | ||
| to Reproducibility (Read more on arXiv or HuggingFace) | AmeyaPrabhu, albanie, vishaal27, hrdkbhatnagar, libeanim | i) This paper analyzes the reproducibility of recent advances in language model (LM) reasoning, identifying sensitivities to implementation choices and proposing a standardized evaluation framework. ii) The research investigates whether reported performance gains in mathematical reasoning benchmarks are robust to variations in decoding parameters, random seeds, prompt formatting, and hardware configurations. iii) The methodology involves a comprehensive empirical study re-evaluating recent methods using a standardized framework and assessing variance across multiple seeds and varying hyperparameters. iv) The study found reinforcement learning approaches yield only modest improvements and are prone to overfitting, while supervised finetuning shows consistently stronger generalization; Pass@1 values show standard deviations ranging from 5 to 15 percentage points across seeds. v) AI practitioners should adopt rigorous, multi-seed evaluation protocols and standardized testing frameworks to ensure the reliability and generalizability of LM reasoning enhancements before integrating them into applications. |
| OmniCaptioner: One Captioner to Rule Them All (Read more on arXiv or HuggingFace) | Cxxs, Wayne-lc, Dakerqi, JiakangYuan, yeeeeeyy | OmniCaptioner introduces a unified visual captioning framework for diverse domains. The main objective is to generate fine-grained textual descriptions for natural images, visual text (posters, UIs), and structured visuals (tables, charts, math) using a single model. The methodology involves a two-stage captioning pipeline (Seed-Caption Generation with GPT-40, Caption Extension with Qwen LLMs) trained on a 21M multi-domain dataset, initializing from Qwen2-VL-Instruct. Primary results show that integrating OmniCaptioner’s detailed captions with LLMs (e.g., DS-R1-Distill-Qwen-7B) significantly improves visual reasoning, achieving 40.5 on MathVerse without MLLM fine-tuning, enhances text-to-image generation (+2.97 on GenEval for SANA-1.0), and enables more efficient SFT (reaching comparable performance to LLaVA-OV-7B with ~1/3 of the SFT data). The principal implication for AI practitioners is the ability to leverage a single, versatile captioner to generate rich, domain-specific descriptions that directly enhance downstream visual reasoning systems, improve text-to-image generation quality, and accelerate supervised fine-tuning for various multimodal tasks. |
| Are We Done with Object-Centric Learning? (Read more on arXiv or HuggingFace) | Matthias Bethge, coallaoh, AmeyaPrabhu, arubique | i) This paper explores the limits of current object-centric learning (OCL) methods. ii) The main objective is to assess whether advances in OCL provide practical benefits beyond unsupervised object discovery, particularly in out-of-distribution (OOD) generalization scenarios. iii) The methodology involves introducing Object-Centric Classification with Applied Masks (OCCAM), a probe using sample-efficient segmentation models to generate object-centric representations and evaluate downstream classification tasks with spurious backgrounds. iv) The primary result shows that segmentation-based encoding of individual objects significantly outperforms slot-based OCL methods in robust zero-shot image classification, achieving up to 78.5% accuracy on ImageNet-D with HQES masks and SigLip models, which is superior to baseline LLAVA 1.5 (73.3%) and FT-Dinosaur (71.5%). v) The principal implication for AI practitioners is that utilizing foundational segmentation models for generating object-centric representations offers a more scalable and effective approach for robust classification tasks compared to traditional slot-centric OCL methods. |
| Self-Steering Language Models (Read more on arXiv or HuggingFace) | Jacob Andreas, Vikash K. Mansinghka, Joshua B. Tenenbaum, Gabriel Grand, alexanderlew | i) This paper introduces DISCIPL, a self-steering framework for language models (LMs) that decouples planning from execution by generating task-specific inference programs. ii) The main research question is how to enable LMs to perform complex reasoning tasks more efficiently and verifiably without extensive fine-tuning. iii) The methodology involves using a Planner LM to generate an inference program, which is then executed by a population of Follower LMs via Sequential Monte Carlo (SMC). iv) Experiments on constrained generation tasks show that DISCIPL, with a 1B Follower, matches or outperforms GPT-40 and 01 models and achieves 0.81 pass@1 on COLLIE sentence-level tasks. v) DISCIPL offers AI practitioners a method to automate the creation of highly parallelized Monte Carlo inference strategies for LMs, improving performance on challenging generation tasks. |
| RuOpinionNE-2024: Extraction of Opinion Tuples from Russian News Texts (Read more on arXiv or HuggingFace) | Anna Lapanitsyna, Natalia Tkachenko, Natalia Loukachevitch, nicolay-r, RefalMachine | i) The paper introduces the RuOpinionNE-2024 shared task for extracting structured opinion tuples from Russian news texts. ii) The primary objective is to extract tuples composed of a sentiment holder, target, expression, and polarity for a given sentence. iii) The methodology involved participants experimenting with large language models using zero-shot, few-shot, and fine-tuning techniques. iv) The best result on the test set was achieved through fine-tuning a large language model with an F1 score of 0.41. v) The principal implication for AI practitioners is the benchmark dataset and performance metrics for structured opinion extraction in Russian, enabling development and evaluation of models for Russian sentiment analysis. |
| Masked Scene Modeling: Narrowing the Gap Between Supervised and | ||
| Self-Supervised Learning in 3D Scene Understanding (Read more on arXiv or HuggingFace) | Leon Sick, Christian Stippel, phermosilla | i) The paper introduces a novel self-supervised approach, Masked Scene Modeling, for learning 3D scene representations. ii) The research aims to develop a self-supervised model for 3D scene understanding that can achieve performance comparable to supervised models when using off-the-shelf features. iii) The methodology involves a bottom-up hierarchical masking approach with a novel reconstruction objective tailored to hierarchical 3D models, reconstructing deep features of masked patches. iv) Experiments demonstrate that the proposed model achieves competitive performance in semantic segmentation (68.7 mIoU on ScanNet using linear probing) compared to supervised models, surpassing existing self-supervised methods. v) The principal implication is that the proposed self-supervised pre-training approach provides AI practitioners with a method to extract features from 3D scenes that perform comparably to supervised approaches, reducing the need for labeled data. |
| DiTaiListener: Controllable High Fidelity Listener Video Generation with | ||
| Diffusion (Read more on arXiv or HuggingFace) | chaubeyG, hongkung, minhtran, Boese0601, havent-invented | DiTaiListener is a video generation model for synthesizing high-fidelity listener head portraits conditioned on speaker audio, facial motions, and optional text prompts. The paper aims to generate controllable and temporally consistent listener behavior in video by adapting Diffusion Transformer (DiT) architecture. The method introduces a Causal Temporal Multimodal Adapter (CTM-Adapter) to process speaker audio and visual cues and DiTaiListener-Edit for refining transitional frames between video segments. DiTaiListener achieves a 73.8% improvement in FID score on RealTalk dataset and a 6.1% improvement on VICO dataset, signifying enhanced photorealism and motion representation. This work provides AI practitioners with an approach for generating realistic and customizable listener videos for applications in virtual avatars and human-computer interaction. |
| VideoChat-R1: Enhancing Spatio-Temporal Perception via Reinforcement | ||
| Fine-Tuning (Read more on arXiv or HuggingFace) | Lanxingxuan, donglu, desenmeng, Aurorana, xinhaoli | VideoChat-R1 enhances spatio-temporal perception in video MLLMs via reinforcement fine-tuning. The research aims to improve spatio-temporal perception in video MLLMs while preserving general capabilities. It employs Reinforcement Fine-Tuning (RFT) with Group Relative Policy Optimization (GRPO) on spatio-temporal objectives using limited data samples. VideoChat-R1 achieves state-of-the-art performance, improving temporal grounding by +31.8 and object tracking by +31.2 compared to Qwen2.5-VL-7B. RFT offers a data-efficient approach for specialized task enhancement in video MLLMs without sacrificing general capabilities, relevant to AI engineers developing video understanding systems. |
| WildGS-SLAM: Monocular Gaussian Splatting SLAM in Dynamic Environments (Read more on arXiv or HuggingFace) | Songyou Peng, Marc Pollefeys, Valentin Bieri, Zihan Zhu, Jianhao Zheng | WildGS-SLAM is presented as a monocular SLAM system using 3D Gaussian Splatting robust to dynamic environments. The research aims to achieve accurate camera tracking and scene reconstruction in dynamic environments using only monocular RGB input. An uncertainty map derived from DINOv2 features is used to guide dynamic object removal within tracking and mapping pipelines. Evaluation on the Wild-SLAM MoCap dataset shows the system achieves an ATE RMSE of 0.46 cm, outperforming existing dynamic SLAM methods. Practitioners can leverage this method for improved SLAM performance in real-world applications with dynamically changing elements without explicit depth or semantic information. |
| RobustDexGrasp: Robust Dexterous Grasping of General Objects from | ||
| Single-view Perception (Read more on arXiv or HuggingFace) | Jie Song, Sammy Christen, Linyi Huang, Zijian Wu, ethHuiZhang | i) This paper introduces a reinforcement-learning-based framework for robust, zero-shot dynamic dexterous grasping of unseen objects from single-view perception. ii) The main objective is to enable a robot to grasp a wide range of previously unseen objects with a dexterous hand using only a single-view camera while adapting to external disturbances. iii) The methodology involves a mixed curriculum learning strategy that combines imitation learning from a teacher policy trained with privileged information and reinforcement learning for adaptation to disturbances, utilizing a hand-centric object representation. iv) The primary result is a grasping success rate of 97.0% across 247,786 simulated objects and 94.6% across 512 real objects without prior knowledge or object-specific training. v) The principal implication for AI practitioners is the demonstrated effectiveness of sparse hand-centric object representation and mixed curriculum learning for training robust dexterous grasping policies that generalize to unseen objects from limited observations, suggesting a path toward more adaptable and general-purpose robotic manipulation systems. |
Papers for 2025-04-09
| Title | Authors | Summary |
|---|---|---|
| OmniSVG: A Unified Scalable Vector Graphics Generation Model (Read more on arXiv or HuggingFace) | Jiaxu Zhang, Xianfang Zeng, Yiying Yang, CH3COOK, wchengad | OmniSVG is a unified framework leveraging pre-trained Vision-Language Models (VLMs) for end-to-end multimodal Scalable Vector Graphics (SVG) generation. The main objective is to produce high-quality, complex, and editable SVGs across diverse modalities (Text-to-SVG, Image-to-SVG, Character-Reference SVG), addressing the limitations of existing methods in handling complexity and structure. The key methodology involves parameterizing SVG commands and coordinates into discrete tokens using a dedicated SVG tokenizer and training a VLM (Qwen2.5-VL) on a large-scale dataset (MMSVG-2M) with a next-token prediction objective. Primary results demonstrate superior performance over existing methods; for instance, on the MMSVG-Illustration text-to-SVG task, OmniSVG(7B) achieved a FID score of 66.91, outperforming SVGDreamer (75.31 on MMSVG-Icon) and other baselines, while handling complex SVGs with token lengths up to 30k. For AI practitioners, OmniSVG offers a versatile, end-to-end solution for generating complex and editable vector graphics from multimodal inputs, potentially integrating into professional design workflows and overcoming the limitations of previous optimization-based or simpler auto-regressive approaches. |
| Skywork R1V: Pioneering Multimodal Reasoning with Chain-of-Thought (Read more on arXiv or HuggingFace) | Jiangbo Pei, Yichen Wei, Xiaokun Wang, Chris, Yi Peng | This paper introduces Skywork R1V, a 38B parameter multimodal model enhancing LLM reasoning for visual tasks using Chain-of-Thought. The primary objective is to efficiently transfer the reasoning capabilities of the text-based R1-series LLM to handle multimodal inputs without retraining the base LLM or vision encoder. Key methodologies include an efficient multimodal transfer via a lightweight MLP visual projector, a hybrid optimization framework combining Iterative SFT and GRPO, and an Adaptive-Length Chain-of-Thought distillation for data generation. Skywork R1V achieves competitive performance, notably scoring 69.0 on the MMMU benchmark and 94.0 on the text-based MATH500 benchmark. For AI practitioners, this work presents an open-source model and methodology demonstrating how to effectively build capable multimodal reasoning systems by efficiently adapting existing strong LLMs, offering a practical approach to enhance VLM reasoning without prohibitive retraining costs. |
| An Empirical Study of GPT-4o Image Generation Capabilities (Read more on arXiv or HuggingFace) | Zhuoran Zhao, Sixiang Chen, donghao-zhou, QingyuShi, BryanW | This paper empirically benchmarks GPT-4o’s image generation, revealing strengths like text rendering but limitations like inconsistency. The objective is to assess GPT-4o’s image generation capabilities by qualitatively benchmarking it against models like Gemini 2.0 Flash Experimental and domain-SOTA methods across >20 tasks (text-to-image, image-to-image, image-to-3D, image-to-X). Methodology relies on structured visual evaluation and error analysis (detailed qualitatively in Table 1) due to the lack of API access and unpublished architecture. Primary results show GPT-4o excels in exceptional text rendering, compositional prompt following, spatial reasoning, and image transformation, often surpassing benchmarks qualitatively, but exhibits limitations in inconsistent generation, hallucination, and data bias (e.g., non-Latin scripts); the study explicitly notes the qualitative nature and lack of quantitative metrics. For AI practitioners, GPT-4o’s notably strong text rendering capability demonstrates potential for unified models requiring precise visual-textual alignment, although current reliability issues (inconsistency, bias) warrant caution for direct deployment. |
| Hogwild! Inference: Parallel LLM Generation via Concurrent Attention (Read more on arXiv or HuggingFace) | Vage Egiazarian, George Yakushev, Alina Shutova, Roman Garipov, Gleb Rodionov | Hogwild! Inference: Parallel LLM Generation via Concurrent Attention This paper proposes Hogwild! Inference, a method enabling multiple instances of the same LLM to generate text in parallel while sharing and concurrently updating a common Key-Value attention cache. The main objective is to explore if LLMs can develop dynamic collaboration strategies for problem-solving without pre-defined frameworks, leveraging immediate access to each other’s partial progress. The key methodology involves running parallel LLM “workers” with a shared KV cache, utilizing Rotary Position Embeddings (RoPE) to efficiently manage positional information across workers and testing three cache layouts: contiguous, interleaved, and combined. Preliminary results on LIMO mathematical reasoning tasks show that the Hogwild! Combined layout allows multiple workers (e.g., 2 workers) to achieve higher accuracy faster than single-threaded baselines or independent parallel workers, reaching approximately 89% accuracy with an 8192 max forward pass budget, surpassing other methods at equivalent budgets. For AI practitioners, the principal implication is that existing reasoning-capable LLMs can potentially leverage shared KV caches for parallel, collaborative inference out-of-the-box to improve efficiency, without requiring model fine-tuning or explicit coordination protocols. |
| COIG-P: A High-Quality and Large-Scale Chinese Preference Dataset for | ||
| Alignment with Human Values (Read more on arXiv or HuggingFace) | Siwei Wu, M-A-P Team, Liam-Liu, aaabiao, JinChengRen | This paper introduces COIG-P, a large-scale (1,006k pairs), high-quality Chinese preference dataset generated via an LLM-based pipeline for human value alignment. The primary objective was to overcome limitations of existing Chinese preference datasets, such as small scale, narrow domains, lack of validation, and the scalability issues of human annotation. The methodology involved crawling and filtering 92k Chinese queries, using 15 LLMs to generate responses, and employing 8 LLMs to score and create chosen-rejected pairs without human intervention, alongside training an 8B Chinese Reward Model (CRM) and creating a Chinese Reward Benchmark (CRBench). Results show COIG-P significantly improves LLM performance on AlignBench, yielding gains of 2% to 12% for Qwen2/2.5 and Infinity-Instruct-3M-0625 models compared to training without it, and the developed CRM demonstrates scoring capabilities comparable to GPT-40 on a test split filtering task. For AI practitioners, COIG-P provides a valuable resource for aligning Chinese LLMs using methods like DPO, while the LLM-based annotation pipeline and the CRM offer scalable, cost-effective alternatives to manual annotation or reliance on expensive large models for data curation. |
| Less-to-More Generalization: Unlocking More Controllability by | ||
| In-Context Generation (Read more on arXiv or HuggingFace) | Fei Ding, Yufeng Cheng, Mengqi Huang, wuwx, fenfan | i) This paper introduces UNO, a universal customization framework enabling less-to-more generalization for controllable single-to-multi-subject image generation using in-context generation. ii) The research aims to develop a stable and scalable paradigm for subject-driven image generation that enhances controllability and consistency, particularly for multi-subject scenarios, while overcoming data limitations. iii) The key methodology is a model-data co-evolution approach, featuring a progressive synthetic data curation pipeline leveraging diffusion transformers’ in-context generation and the UNO model, which incorporates progressive cross-modal alignment and Universal Rotary Position Embedding (UnoPE) into a DiT architecture. iv) UNO demonstrates state-of-the-art results, achieving the highest DINO (0.760) and CLIP-I (0.835) scores on the DreamBench single-subject benchmark among tuning-free methods evaluated. v) For AI practitioners, UNO provides a tuning-free framework capable of generating high-fidelity images with strong subject similarity and text controllability for both single and multiple subjects, directly applicable to customization tasks without per-subject optimization. |
| Generative Evaluation of Complex Reasoning in Large Language Models (Read more on arXiv or HuggingFace) | Baizhou Huang, Ruilin Yan, Xiangyu Wang, YitaoLiang, pkuHaowei | This paper introduces KUMO, a generative evaluation framework combining LLMs and symbolic engines to dynamically create complex, contamination-resistant reasoning tasks for assessing large language models. The primary objective is to reliably evaluate genuine LLM reasoning capabilities, distinguishing it from memorization resulting from training data contamination of static benchmarks. KUMO employs a neural-symbolic pipeline utilizing LLMs for domain generation and SAT-based engines for task instantiation, creating partially observable, multi-turn reasoning games across numerous domains with adjustable difficulty, evaluated via success rate and relative action count. Key results from evaluating 23 LLMs on 5,000 tasks across 100 domains show reasoning-scaled models achieve university-level performance on complex tasks, and KUMO performance correlates strongly (Pearson correlation > 0.9 on hard setting vs MMLU-Pro/LiveBench-Reason) with recent real-world benchmarks, while experiments demonstrate resistance to overfitting. For AI practitioners, KUMO provides a scalable, dynamic, and contamination-resistant benchmark methodology for assessing the true reasoning progress of LLMs, facilitating more reliable model evaluation and development efforts compared to potentially saturated static datasets. |
| Tuning-Free Image Editing with Fidelity and Editability via Unified | ||
| Latent Diffusion Model (Read more on arXiv or HuggingFace) | Ming-Hsuan Yang, Mike Zheng Shou, Yuchao Gu, Lan Chen, Qi Mao | i) The paper introduces UnifyEdit, a tuning-free method for text-based image editing that balances fidelity and editability using a unified latent diffusion optimization framework. ii) The research aims to enable a balanced integration of fidelity and editability in text-based image editing without extensive retraining, addressing issues of over- or under-editing. iii) UnifyEdit employs self-attention preservation and cross-attention alignment constraints, along with an adaptive time-step scheduler, to guide diffusion latent optimization. iv) Experiments show UnifyEdit outperforms existing methods, demonstrating superior structure preservation and text alignment across various editing tasks, with user studies showing a 66%-84% preference for fidelity compared to baseline approaches. v) AI practitioners can utilize UnifyEdit for more robust and adaptable text-based image editing, achieving a better balance between preserving original image structure and accurately reflecting text-based modifications. |
| V-MAGE: A Game Evaluation Framework for Assessing Visual-Centric | ||
| Capabilities in Multimodal Large Language Models (Read more on arXiv or HuggingFace) | Alex Jinpeng Wang, Ping Yu, Zhengyuan Yang, Linjie Li, Fengx1nn | i) V-MAGE is introduced as a game-based framework to evaluate the visual reasoning capabilities of multimodal large language models (MLLMs). ii) The research aims to address limitations in current game-based benchmarks by providing visually-centric tasks that assess diverse reasoning skills. iii) The methodology involves evaluating leading MLLMs across five games with 30+ levels, using an adaptive Elo-based ranking system for performance comparison. iv) Results show a substantial performance gap between top-performing MLLMs and humans, with GPT-40 scoring 1.93/10 versus a human score of ≈10/10 in FlappyBird Level 6, while Qwen2VL-72B achieved 0.61/10 on the same task. v) V-MAGE highlights limitations in MLLMs’ visual perception and reasoning, suggesting a need to refine agent strategies and address perceptual inaccuracies from an agent-centric perspective for AI improvement. |
| CrossWordBench: Evaluating the Reasoning Capabilities of LLMs and LVLMs | ||
| with Controllable Puzzle Generation (Read more on arXiv or HuggingFace) | William W. Cohen, Bill Yuchen Lin, Langlin Huang, Chengsong Huang, Jixuan Leng | This paper introduces CrossWordBench, a benchmark using controllable crossword puzzles to evaluate the multimodal reasoning of LLMs and Large Vision-Language Models (LVLMs). The main objective is to assess model capabilities in handling tasks requiring simultaneous adherence to semantic constraints from text clues and structural constraints from visual grids. Methodologically, it utilizes a controllable puzzle generation framework creating text and image formats from diverse sources and evaluates over 20 models using zero-shot Chain-of-Thought and interactive modes. Results show reasoning LLMs significantly outperform non-reasoning models by leveraging crossing-letter constraints (achieving an 89% relative increase in Intersection Consistency Rate), while LVLMs perform poorly, with puzzle-solving performance strongly correlating (r=0.94) with grid-parsing accuracy. For AI practitioners, this highlights current LVLMs’ limitations in integrating visual-structural information with textual reasoning for constrained tasks and suggests the benchmark’s potential for developing and evaluating models with better spatial-textual grounding. |
| Accelerate Parallelizable Reasoning via Parallel Decoding within One | ||
| Sequence (Read more on arXiv or HuggingFace) | Yijiong Yu | The paper introduces a parallel decoding method, “Parallel Decoding in One Sequence,” for accelerating reasoning in Large Language Models (LLMs). The research aims to address the inefficiency of autoregressive decoding for tasks with parallelizable steps. The methodology involves identifying parallelizable steps, decoding them in parallel using a modified attention mask and position IDs within a single sequence, and then concatenating the results. Experiments demonstrate over 100% speedup in decoding time on a retrieval task with a context of 10 items while maintaining answer quality. This method enables AI practitioners to accelerate LLM reasoning on parallelizable tasks without additional memory usage or KV cache recomputation. |
| HiFlow: Training-free High-Resolution Image Generation with Flow-Aligned | ||
| Guidance (Read more on arXiv or HuggingFace) | Tong Wu, Pan Zhang, Yujie Zhou, Pengyang Ling, Jiazi Bu | HiFlow introduces a training-free, model-agnostic framework for high-resolution text-to-image generation using pre-trained rectified flow models. The research aims to enhance image quality in high-resolution synthesis by establishing a virtual reference flow and aligning it with the high-resolution sampling flow through initialization, direction, and acceleration alignment. HiFlow achieves superior high-resolution image quality over state-of-the-art methods, demonstrating, for example, a FID score of 52.55 for 4096x4096 image generation. The flow-aligned guidance approach offers AI practitioners a method for improving image fidelity and detail in high-resolution T2I tasks without requiring model retraining. The paper does not provide information about the compute resources required or its use with large language models. |
| Leanabell-Prover: Posttraining Scaling in Formal Reasoning (Read more on arXiv or HuggingFace) | Yang Yue, Yahui Liu, Xingguang Ji, Qi Wang, Jingyuan Zhang | Leanabell-Prover improves automated theorem proving (ATP) through posttraining scaling of large language models using Lean 4 code. This research investigates posttraining techniques for ATP with the aim of achieving breakthroughs similar to those seen in natural language reasoning models. The study utilizes a hybrid dataset for continual training and GRPO for reinforcement learning, incorporating cognitive behaviors. Results show a 59.8% pass rate (pass@32) on the MiniF2F test after employing RL training, surpassing DeepSeek-Prover-v1.5-RL and Goedel-Prover-SFT. AI practitioners can leverage the proposed methods to enhance formal provers, leading to state-of-the-art performance in whole-proof generation. |
Papers for 2025-04-08
| Title | Authors | Summary |
|---|---|---|
| One-Minute Video Generation with Test-Time Training (Read more on arXiv or HuggingFace) | guestrin, zhaoyue-zephyrus, GashonHussein, koceja, karansdalal | This paper introduces Test-Time Training (TTT) layers integrated into a Diffusion Transformer to generate coherent one-minute videos from text storyboards. The main objective is to address the inefficiency of self-attention and the limited expressiveness of standard RNN hidden states for generating long videos with complex narratives. The key methodology involves adding TTT layers, whose hidden states are neural networks (specifically two-layer MLPs) updated via test-time gradient descent on a self-supervised reconstruction task, to a pre-trained CogVideo-X 5B model and fine-tuning on a curated Tom and Jerry dataset. The primary result shows that TTT layers significantly improve video coherence and storytelling for one-minute videos compared to baselines like Mamba 2 and Gated DeltaNet, leading by 34 Elo points in human evaluations, although some artifacts persist and efficiency needs improvement. For AI practitioners, this demonstrates TTT layers as a viable approach to enhance temporal consistency in long video generation, offering a mechanism to handle extended contexts beyond typical attention or RNN limitations, but requiring consideration of current efficiency trade-offs. |
| SmolVLM: Redefining small and efficient multimodal models (Read more on arXiv or HuggingFace) | eliebak, mervenoyan, mfarre, orrzohar, andito | SmolVLM introduces a family of compact, efficient Vision-Language Models (VLMs) designed for resource-constrained inference on edge devices. The primary objective was to engineer small VLMs by systematically exploring architectural configurations, tokenization strategies, and data curation optimized for low computational overhead and minimal memory footprints. Key methodologies included investigating encoder-LM parameter balance, optimizing context length and pixel shuffling for token reduction, evaluating learned versus string positional tokens, using image splitting, and carefully curating training data mixes (including CoT and video duration). Results show the smallest model (SmolVLM-256M) achieves a 44.0% average score across benchmarks using less than 1GB GPU RAM, outperforming significantly larger models, while the 2.2B variant rivals state-of-the-art models requiring double the GPU memory. For AI practitioners, the principal implication is that strategic architectural optimizations, aggressive tokenization, and curated data enable high-performance multimodal capabilities at much smaller scales, facilitating practical deployment on edge devices. |
| T1: Tool-integrated Self-verification for Test-time Compute Scaling in | ||
| Small Language Models (Read more on arXiv or HuggingFace) | Jaewoong Cho, Jongwon Jeong, Nardien | This paper introduces Tool-integrated Self-verification (T1) to enhance small language model (sLM) self-verification during test-time compute scaling by using external tools. The main research objective is to investigate if sLMs can reliably perform self-verification for test-time scaling, particularly for memorization-heavy tasks, and to improve this capability without resorting to larger models. The key methodology involves T1, a two-stage process combining a tool-based verifier (ToolV) leveraging external tools (e.g., code interpreter) for filtering, and a reward model (RM)-based verifier for scoring, with both components enhanced via knowledge distillation from larger teacher models. Primary results demonstrate that T1 significantly boosts sLM performance; specifically, a Llama-3.2 1B model using T1 under test-time scaling outperformed a significantly larger Llama-3.1 8B model on the MATH benchmark. The principal implication for AI practitioners is that integrating external tools via methods like T1 can substantially improve the reasoning and verification capabilities of computationally cheaper sLMs, enabling them to tackle complex tasks more effectively and potentially match larger model performance in specific domains. |
| URECA: Unique Region Caption Anything (Read more on arXiv or HuggingFace) | Heeji Yoon, seungryong, crepejung00, junwann, SammyLim | URECA introduces a large-scale dataset and novel model for generating unique captions for image regions at multiple granularities. The primary objective is to address the limitation of existing methods that struggle to produce distinctive descriptions for regions across varying levels of detail, especially distinguishing visually similar regions. The methodology involves a four-stage automated data curation pipeline utilizing mask trees and MLLMs to generate unique captions, and a captioning model featuring a dynamic mask encoder that preserves spatial properties for multi-granularity inputs. The proposed URECA model achieves state-of-the-art performance on the new dataset, attaining a BERTScore of 75.11, and demonstrates strong zero-shot generalization on benchmarks like Visual Genome with a METEOR score of 18.4. For AI practitioners, this work provides a robust dataset and model architecture enabling the generation of precise, context-aware natural language descriptions for arbitrarily selected image regions, enhancing detailed visual understanding applications. |
| Quantization Hurts Reasoning? An Empirical Study on Quantized Reasoning | ||
| Models (Read more on arXiv or HuggingFace) | Yuxuan Sun, Tiezheng, baihaoli, manyi2024, ruikangliu | This paper empirically investigates the impact of quantization on the reasoning abilities of large language models. The primary objective is to systematically evaluate how weight-only, KV cache, and weight-activation quantization affect reasoning performance across various model families, sizes, and tasks. The study quantizes DeepSeek-R1-Distilled Qwen/LLaMA families (1.5B-70B) and QwQ-32B using state-of-the-art algorithms (e.g., AWQ, QuaRot, FlatQuant) and evaluates them on mathematical, scientific, and programming reasoning benchmarks. Key findings reveal that W8A8 weight-activation or W4A16 weight-only/KV cache quantization can achieve near-lossless performance (≤1% accuracy drop), whereas lower bit-widths introduce significant risks, influenced by model size, origin (distilled vs. RL), and task difficulty. For AI practitioners, this implies that while 8-bit or selective 4-bit quantization can preserve reasoning with minimal loss, aggressive low-bit quantization requires careful consideration of the specific model and task, with FlatQuant and AWQ/QuaRot being preferred algorithms for weight-activation and weight-only/KV cache respectively. |
| Concept Lancet: Image Editing with Compositional Representation | ||
| Transplant (Read more on arXiv or HuggingFace) | Hancheng Min, Tianjiao Ding, CCB, ryanckh, peterljq | Concept Lancet (CoLan) introduces a zero-shot, plug-and-play framework for diffusion-based image editing using sparse concept decomposition and transplant in latent space. The research aims to solve the challenge of accurately determining the required edit strength for concept manipulation in images, avoiding over/under-editing without costly trial-and-error. CoLan employs a large curated concept dictionary (CoLan-150K), VLM-based parsing for task-specific concepts, and sparse coding to decompose the source latent vector (text embedding or diffusion score), allowing targeted replacement (transplant) of concept vectors. Equipping editing backbones like P2P-Zero with CoLan significantly improved consistency preservation, reducing LPIPS by nearly 50% (from 273.8/142.4 to 120.3/68.43 x10^-3 on whole image/background) while enhancing edit effectiveness on the PIE-Bench dataset. AI practitioners can integrate CoLan into diffusion editing pipelines to achieve more precise and consistent edits automatically by estimating and applying appropriate concept-specific magnitudes, eliminating the need for manual edit strength tuning per image. |
| LiveVQA: Live Visual Knowledge Seeking (Read more on arXiv or HuggingFace) | Yao Wan, Mingyang Fu, shuaishuaicdp, Tim666, Ayiirep | This paper introduces LIVEVQA, a benchmark dataset automatically collected from recent news to evaluate Multimodal Large Language Models (MLLMs) on live visual knowledge seeking. The research objective is to assess the capability of current MLLMs to answer questions demanding understanding of up-to-date visual knowledge synthesized from internet news content. Methodology involved creating the LIVEVQA dataset (3,602 single- and multi-hop visual questions from 1,233 news instances across 14 categories) and evaluating 15 MLLMs (e.g., GPT-4o, Gemma-3, Qwen-2.5-VL) with and without search tool integration. Primary results demonstrate that while stronger models perform better overall, significant performance gaps persist, particularly for complex multi-hop questions requiring recent visual knowledge; Gemini-2.0-Flash achieved the highest accuracy at 24.93% without search integration. The principal implication for AI practitioners is that current MLLMs, even sophisticated ones, struggle significantly with visual questions requiring timely, real-world knowledge and complex reasoning, highlighting a critical need for improved visual grounding and knowledge integration mechanisms. |
| Are You Getting What You Pay For? Auditing Model Substitution in LLM | ||
| APIs (Read more on arXiv or HuggingFace) | Tianneng Shi, Will Cai, dawnsong, Xuandong | This paper evaluates methods for detecting undisclosed model substitution in black-box Large Language Model (LLM) APIs. The objective is to formalize the API auditing problem and assess the robustness of software-based verification techniques (text classification, MMD, benchmarks, log probability analysis) and hardware solutions (TEEs) against adversarial attacks like quantization and randomized substitution. Methodology involves empirical evaluation of these techniques using various LLMs (Llama, Gemma, Mistral, Qwen2) under different attack scenarios, including comparing outputs, benchmark scores, and log probabilities. Primary results indicate that text-output-based methods are ineffective against subtle changes like quantization (e.g., text classifiers achieve only ~50% accuracy distinguishing original vs. quantized models) and randomized substitution (MMD test power drops significantly), while log probability analysis is more sensitive but relies on often unavailable API features; TEEs show promise with low performance overhead (<3% throughput impact under load). The principal implication for AI practitioners is that relying solely on current software-based verification for API model identity is unreliable, highlighting the need for enhanced provider transparency or hardware-attested environments like TEEs to ensure model integrity in critical applications and benchmarking. |
| Gaussian Mixture Flow Matching Models (Read more on arXiv or HuggingFace) | saibi, wetzste1, luanfujun, zexiangxu, Lakonik | GMFlow introduces a novel flow matching model predicting Gaussian mixture (GM) parameters instead of just the mean velocity to enhance generative modeling. The primary objective is to overcome the limitations of discretization errors in few-step sampling and color over-saturation issues associated with classifier-free guidance (CFG) in existing diffusion and flow matching models. Key methodology involves parameterizing the flow velocity as a GM, training with a KL divergence loss, deriving novel GM-SDE/ODE solvers that leverage analytic distributions, and introducing a probabilistic guidance mechanism for CFG reweighting rather than extrapolation. GMFlow demonstrates superior performance, achieving a Precision of 0.942 with only 6 sampling steps and a state-of-the-art Precision of 0.950 with 32 steps on ImageNet 256x256, significantly outperforming baselines, especially in few-step scenarios. For AI practitioners, this provides a framework for developing generative models capable of faster, higher-fidelity sampling with reduced CFG-induced saturation artifacts. |
| DiaTool-DPO: Multi-Turn Direct Preference Optimization for | ||
| Tool-Augmented Large Language Models (Read more on arXiv or HuggingFace) | Donghun Lee, dsindex, junrae, gaeunseo, hash2430 | This paper introduces DiaTool-DPO, a Direct Preference Optimization method enhancing Tool-Augmented LLMs’ multi-turn dialogue control for information gathering and tool rejection. The primary objective was to improve TA-LLM handling of incomplete or out-of-scope user queries by adapting DPO without requiring new expert demonstrations. Key methodology involves modeling interactions as a Markov Decision Process, automatically constructing paired chosen/rejected dialogue trajectory datasets based on defined query types, and applying a specialized DiaTool-DPO objective loss with turn-length normalization and reward gap margins. Experiments showed DiaTool-DPO significantly improved LLaMA3-8B-Instruct’s performance over SFT-only baselines, achieving 91.7% slot-filling accuracy (a 44% improvement) and 91.3% relevance accuracy (a 9.6% improvement), nearing GPT-4o performance. For AI practitioners, this method offers a way to train more robust TA-LLMs capable of managing ambiguous requests and unavailable tools using automatically generated preference data, reducing problematic tool calls without manual labeling. |
| VAPO: Efficient and Reliable Reinforcement Learning for Advanced | ||
| Reasoning Tasks (Read more on arXiv or HuggingFace) | Ruofei Zhu, Xiaochen Zuo, Qiying Yu, Yufeng Yuan, YuYue | This paper introduces VAPO, a value-based reinforcement learning framework designed to enhance the performance and efficiency of large language models on advanced reasoning tasks requiring long chain-of-thought. The primary objective is to overcome limitations inherent in value-based RL for long-CoT, specifically value model bias, handling heterogeneous sequence lengths, and sparse reward signals, aiming to surpass existing value-free methods. VAPO employs a modified Proximal Policy Optimization (PPO) approach incorporating seven key techniques, including Value-Pretraining, Decoupled and Length-Adaptive Generalized Advantage Estimation (GAE), Token-Level Loss, Clip-Higher clipping, Positive Example LM Loss, and Group-Sampling. Benchmarked on AIME 2024 using a Qwen-32B model, VAPO achieved a state-of-the-art score of 60.4 within 5,000 training steps, significantly outperforming the prior SOTA value-free method DAPO by over 10 points while demonstrating greater training stability and efficiency. For AI practitioners, VAPO presents a robust and efficient value-based RL alternative for training high-performance reasoning models, offering improved stability and potentially higher accuracy ceilings compared to value-free methods on complex, long-CoT tasks. |
| Mamba as a Bridge: Where Vision Foundation Models Meet Vision Language | ||
| Models for Domain-Generalized Semantic Segmentation (Read more on arXiv or HuggingFace) | robbytan, XinNUS | This paper introduces MFuser, a Mamba-based framework to efficiently fuse Vision Foundation Models (VFMs) and Vision-Language Models (VLMs) for Domain-Generalized Semantic Segmentation (DGSS). The primary objective is to combine the complementary strengths of VFMs (fine-grained features) and VLMs (robust text alignment) while overcoming the challenges of long-sequence modeling and computational cost associated with integrating large models. The key methodology involves two components: MVFuser, a Mamba-based co-adapter for joint parameter-efficient fine-tuning of VFM and VLM visual features, and MTEnhancer, a hybrid attention-Mamba module to refine VLM text embeddings using visual priors. MFuser significantly outperforms existing DGSS methods, achieving a state-of-the-art 68.20 mIoU on the synthetic-to-real benchmark (G→{C, B, M} average) using DINOv2 and EVA02-CLIP. For AI practitioners, this work presents a computationally efficient Mamba-based adapter approach (MVFuser) to synergistically combine diverse foundation models, enhancing generalization for semantic segmentation tasks without requiring full fine-tuning of the base models. |
| BOP Challenge 2024 on Model-Based and Model-Free 6D Object Pose | ||
| Estimation (Read more on arXiv or HuggingFace) | taeyeop, anas-gouda, mfourmy, swtyree, nv-nguyen | The BOP Challenge 2024 advanced the state-of-the-art in 6D object pose estimation by introducing model-free tasks, new high-resolution datasets (BOP-H3), and a practical 6D detection task. The main objective was to shift evaluation from lab-like setups towards real-world applicability, notably by requiring methods to onboard unseen objects from reference videos without CAD models in model-free tracks. Key methodology involved evaluating methods across seven tracks defined by task (6D localization, 6D detection, 2D detection), onboarding setup (model-based, model-free), and dataset group (BOP-Classic-Core, BOP-H3) using established metrics like Average Recall (AR) and Average Precision (AP). Primary results showed significant progress: the best model-based 6D localization method for unseen objects (FreeZeV2.1) achieved 82.1 AR on BOP-Classic-Core, 22% higher than the 2023 best, though 2D detection for unseen objects still lags significantly (-53% behind seen objects), indicating it’s the main pipeline bottleneck. For AI practitioners, this highlights substantial improvements in unseen object pose estimation accuracy but underscores the critical need to advance 2D detection capabilities for robust real-world system deployment. |
| Clinical ModernBERT: An efficient and long context encoder for | ||
| biomedical text (Read more on arXiv or HuggingFace) | Jeffrey N. Chiang, Anthony Wu, Simonlee711 | This paper introduces Clinical ModernBERT, an efficient transformer encoder adapted for long-context biomedical and clinical text processing. The main objective is to leverage ModernBERT’s architectural improvements (RoPE, Flash Attention, GeGLU, 8192 token context) and adapt them via domain-specific pretraining for enhanced clinical language understanding. Methodology involved continued pretraining of a ModernBERT-base model on a 13-billion-token corpus comprising PubMed abstracts, MIMIC-IV clinical notes, and structured medical ontologies using masked language modeling with token-aware masking. Primary results demonstrate strong performance on clinical NLP benchmarks, achieving a state-of-the-art 0.9769 AUROC on EHR classification and superior runtime efficiency compared to BioClinicalBERT, processing data ~1.6x faster at higher volumes. The principal implication for AI practitioners is the availability of a performant, efficient, and publicly released encoder backbone specifically optimized for long clinical sequences and medical code semantics, suitable for replacing older BERT variants in clinical applications. |
| JailDAM: Jailbreak Detection with Adaptive Memory for Vision-Language | ||
| Model (Read more on arXiv or HuggingFace) | Li Li, Yi Nian, yuehanqi, yuehanqi, Chouoftears | i) JailDAM is a novel framework for detecting and mitigating jailbreak attacks on Vision-Language Models (VLMs) using an adaptive memory mechanism. ii) The research aims to develop a robust and efficient jailbreak detection method for VLMs, addressing the limitations of existing approaches, such as reliance on model internals or expensive computations. iii) The methodology involves a memory-based approach, using policy-driven unsafe knowledge representations, test-time adaptation to refine the memory with emerging unsafe variations, and an autoencoder-based detection pipeline. iv) Experiments on VLM jailbreak benchmarks demonstrate that JailDAM delivers state-of-the-art performance in harmful content detection, improving both accuracy and speed by an average of 0.10 AUROC compared to the second-best method. v) JAILDAM offers AI practitioners a black-box compatible and computationally efficient solution for detecting jailbreak attempts in VLMs, adaptable to new attack strategies without requiring extensive harmful data or model retraining, enhancing the safety and robustness of VLM deployments. |
| GlotEval: A Test Suite for Massively Multilingual Evaluation of Large | ||
| Language Models (Read more on arXiv or HuggingFace) | Ona de Gibert, Sawal Devkota, Joseph Attieh, Zihao Li, zuenmin | i) GlotEval is introduced as a lightweight massively multilingual evaluation framework for Large Language Models (LLMs). ii) The research aims to address the challenge of evaluating LLMs in diverse linguistic environments, especially low-resource languages, by providing a consistent and flexible evaluation framework. iii) The methodology involves integrating 20+ existing multilingual benchmarks across seven key tasks including machine translation, text classification, and summarization, standardizing language codes, and incorporating language-specific prompt templates with optional Microsoft Translator integration. iv) Experiments with Qwen2-1.5B model show throughput variances across languages and hardware setups, with Nvidia A100 generally achieving higher throughput than AMD MI250X; for example, French translation achieved 969.55 tokens/s on Nvidia A100. v) GlotEval offers AI practitioners a tool for fine-grained diagnostics of model strengths and weaknesses across a wide array of languages, facilitating the development of more inclusive and robust multilingual language technologies. |
| Rethinking Multilingual Continual Pretraining: Data Mixing for Adapting | ||
| LLMs Across Languages and Resources (Read more on arXiv or HuggingFace) | Jörg Tiedemann, Hengyu Luo, Shaoxiong Ji, Zihao Li | i) This paper investigates data mixing strategies in multilingual continual pretraining (CPT) for adapting large language models (LLMs) across languages and resources. ii) The main objective is to evaluate the relative effectiveness of monolingual, bilingual, and code-augmented data strategies in multilingual CPT. iii) The study systematically evaluates 36 CPT configurations involving three multilingual base models across 30+ languages categorized as altruistic, selfish, and stagnant. iv) The findings reveal that bilingual CPT improves multilingual classification but often causes language mixing, while code data inclusion enhances classification but introduces generation trade-offs, with Llama-3.1-8B achieving only 7.47 BLEU with bilingual CPT versus 25.52 with monolingual CPT for high-resource languages. v) The principal implication for AI practitioners is the need for adaptive CPT methods that balance classification improvements and generation quality due to the complex interactions between language characteristics and data mixing strategies. |
Papers for 2025-04-07
| Title | Authors | Summary |
|---|---|---|
| Multi-SWE-bench: A Multilingual Benchmark for Issue Resolving (Read more on arXiv or HuggingFace) | Linhao Zhang, Hanwu Chen, Wei Liu, Zhirong Huang, Daoguang Zan | This paper introduces Multi-SWE-bench, a multilingual benchmark for evaluating Large Language Models (LLMs) on software issue resolving tasks across diverse programming languages. The main objective is to overcome the limitations of existing Python-centric benchmarks like SWE-bench by providing a comprehensive evaluation framework for Java, TypeScript, JavaScript, Go, Rust, C, and C++. The methodology involved a five-phase pipeline including repository selection, pull request crawling, environment determination, automated filtering based on test outcomes, and rigorous manual verification by 68 experts, resulting in 1,632 high-quality instances; state-of-the-art LLMs were then evaluated using Agentless, SWE-agent, and OpenHands methods. Primary results show existing models struggle to generalize beyond Python, with performance significantly decreasing on complex tasks; for instance, resolved rates drop sharply when fix patches exceed 600 tokens or involve multiple files, indicating weaknesses in long-context retention and multi-file reasoning. For AI practitioners, Multi-SWE-bench offers a robust tool for assessing LLM capabilities in realistic, multilingual software engineering scenarios, revealing current limitations and guiding future development, alongside releasing initial datasets and infrastructure for reinforcement learning (Multi-SWE-RL) in this domain. |
| Agentic Knowledgeable Self-awareness (Read more on arXiv or HuggingFace) | Xiangyuan Ru, Xiaobin Wang, Baochang Ren, Zhisong Qiu, Shuofei Qiao | This paper introduces agentic knowledgeable self-awareness, enabling LLM agents to autonomously regulate knowledge utilization based on situational difficulty. The research objective is to overcome the limitations of traditional “flood irrigation” methods by allowing agents to decide when to use internal capabilities, reflect, or seek external knowledge. The proposed method, KnowSelf, employs a heuristic situation judgment criterion on self-explored trajectories and a two-stage (SFT + RPO) training process using special tokens to signify different cognitive states (fast, slow, knowledgeable thinking). Experiments demonstrate KnowSelf achieves superior performance with minimal knowledge; for instance, on ALFWorld using Llama-8B, it attained an 84.33% average reward while using external knowledge for only 15.01% of actions, outperforming baselines. For AI practitioners, this implies a method to train more efficient agents that dynamically manage computational resources (like reflection or knowledge retrieval) based on assessed task complexity, potentially reducing inference costs and improving robustness. |
| MegaMath: Pushing the Limits of Open Math Corpora (Read more on arXiv or HuggingFace) | Liping Tang, Zhoujun Cheng, Nikhil Ranjan, Zengzhi Wang, Fan Zhou | MegaMath introduces a large-scale, 371B token open dataset specifically curated for math-centric LLM pre-training. The primary objective was to address the lack of open, high-quality, large-scale corpora tailored for mathematical reasoning in LLMs. Methodology involved re-extracting and filtering Common Crawl data with math-specific optimizations, recalling math-relevant code from Stack-V2, and synthesizing QA, translated code, and interleaved text-code data. Key results demonstrate MegaMath’s scale and quality, with subsets like MegaMath-Web-Pro (15.1B tokens) outperforming existing open math corpora like FineMath-4+ by ≥ 4% in comparative pre-training evaluations, and boosting Llama-3 CoT performance by 15-20%. For AI practitioners, MegaMath provides a high-quality, large-scale open resource enabling the pre-training of more capable mathematical reasoning LLMs, previously hindered by the scarcity of suitable open datasets. |
| SynWorld: Virtual Scenario Synthesis for Agentic Action Knowledge | ||
| Refinement (Read more on arXiv or HuggingFace) | Jialong Wu, Shuofei Qiao, Yuan Liang, Xiaobin Wang, Runnan Fang | SynWorld introduces a framework for LLM-based agents to refine action knowledge by synthesizing virtual scenarios and using Monte Carlo Tree Search (MCTS) for exploration. The primary objective is to enable agents to autonomously enhance their understanding of actions and optimize workflows in novel or complex environments. The methodology involves synthesizing multi-step task scenarios conditioned on tool subsets and applying iterative MCTS optimization to refine action descriptions and cognitive workflows based on simulated environmental feedback. Key results demonstrate SynWorld’s effectiveness, achieving a 59.33 PASS score on ToolBench using GPT-4-turbo, outperforming several baseline methods. For AI practitioners, this implies a viable approach to automatically adapt agents to new tools and environments, improving planning and execution capabilities through simulated experience, thereby reducing reliance on manual annotation for action knowledge refinement. |
| MME-Unify: A Comprehensive Benchmark for Unified Multimodal | ||
| Understanding and Generation Models (Read more on arXiv or HuggingFace) | Bingyan Nie, Yang Shi, Chaoyou Fu, Yi-Fan Zhang, Wulin Xie | This paper introduces MME-Unify (MME-U), a comprehensive benchmark to evaluate Unified Multimodal Large Language Models (U-MLLMs) across understanding, generation, and novel unified tasks. The primary objective was to create a standardized evaluation framework addressing the lack of unified standards and benchmarks for mixed-modality generation capabilities in U-MLLMs. The methodology involved curating tasks from 12 datasets, standardizing formats (e.g., multiple-choice QA, normalized scores), and designing five new ‘unify’ tasks (e.g., Visual CoT, Image Editing & Explaining) requiring synergistic understanding and generation. Evaluations of 12 U-MLLMs revealed significant room for improvement, especially in instruction following and unified tasks, with the top model Gemini2.0-flash-exp achieving an MME-U score of 45.57, while many models struggled significantly on complex unified tasks. For AI practitioners, this highlights current U-MLLM limitations in reliably performing complex, integrated multimodal reasoning and generation, underscoring the need for improved model architectures and training strategies for robust real-world deployment. |
| VARGPT-v1.1: Improve Visual Autoregressive Large Unified Model via | ||
| Iterative Instruction Tuning and Reinforcement Learning (Read more on arXiv or HuggingFace) | Liming Liang, Dongchao Yang, Yufan Deng, Yuxin Xie, Xianwei Zhuang | VARGPT-v1.1 presents an improved unified visual autoregressive model for enhanced understanding and generation tasks. The objective is to advance the VARGPT framework by improving instruction-following, generation quality, and overall multimodal performance through enhanced training strategies and data scaling. Key methodology combines iterative visual instruction tuning (SFT) on an expanded 8.3M visual-generative instruction pair corpus with Direct Preference Optimization (DPO) reinforcement learning, upgrades the LLM backbone to Qwen2-7B, increases generation resolution, and enables editing capabilities via SFT. The model achieves state-of-the-art results on multimodal understanding benchmarks, such as 81.01 on MMBench, significantly improving comprehension and generation metrics over its predecessor and comparable models. For AI practitioners, this work demonstrates that iterative SFT and DPO-based RL within a purely visual autoregressive framework can yield highly capable unified multimodal systems, offering an alternative architecture to diffusion-based or separate component approaches. |
| APIGen-MT: Agentic Pipeline for Multi-Turn Data Generation via Simulated | ||
| Agent-Human Interplay (Read more on arXiv or HuggingFace) | Ming Zhu, Jianguo Zhang, Weiran Yao, Zuxin Liu, Akshara Prabhakar | This paper introduces APIGen-MT, a two-phase framework for generating verifiable multi-turn agent interaction data via simulated agent-human interplay. The primary objective was to overcome the scarcity of high-quality, realistic multi-turn data needed for training capable AI agents. The methodology involves first generating verified task blueprints using an agentic pipeline with LLM reviewers and feedback, followed by simulating human-agent interactions based on these blueprints to create full trajectories. Key results show models trained on this data (xLAM-2-fc-r series) outperform strong baselines; for instance, the 70B model achieved 78.19% accuracy on BFCL v3, surpassing GPT-40, with smaller models also demonstrating superior multi-turn consistency. For AI practitioners, this work provides open-source, high-quality synthetic data and models enabling the development of more reliable agents for complex, multi-turn interactions, potentially allowing smaller models to achieve performance comparable to larger ones. |
| HumanDreamer-X: Photorealistic Single-image Human Avatars Reconstruction | ||
| via Gaussian Restoration (Read more on arXiv or HuggingFace) | Guosheng Zhao, Xiaofeng Wang, Runqi Ouyang, Boyuan Wang, ZhengZhu | HumanDreamer-X introduces a unified pipeline for photorealistic single-image 3D human avatar reconstruction by integrating multi-view generation and Gaussian restoration. The primary objective is to overcome geometric inconsistencies and visual artifacts like fragmented limbs common in decoupled generation-then-reconstruction approaches for single-view inputs. The methodology involves initial coarse avatar reconstruction using 3D Gaussian Splatting (3DGS), rendering multi-view video frames, refining these frames with a video restoration model named HumanFixer which incorporates an attention modulation strategy, and subsequently using the restored video to enhance the 3DGS model. Key results show significant improvements over existing methods, achieving up to 25.62 dB PSNR in reconstruction quality, a 12.65% increase compared to prior SOTA on CustomHumans. For AI practitioners, this work demonstrates a technique combining explicit 3D representation (3DGS) with generative video restoration and attention modulation to create higher-quality, consistent digital humans from minimal input, applicable to virtual avatar creation and animation. |
| TransMamba: Flexibly Switching between Transformer and Mamba (Read more on arXiv or HuggingFace) | Shuaipeng Li, Xingwu Sun, Ruobing Xie, andyyang, Yixinglee | This paper proposes TransMamba, a framework unifying Transformer and Mamba using shared parameters to switch dynamically between attention and state space model (SSM) mechanisms. The objective is to leverage the strengths of both Transformer (short context efficiency) and Mamba (long context efficiency) within a single flexible architecture, overcoming static hybrid model limitations. TransMamba utilizes shared QKV/CBx parameters and introduces a “Memory Converter” for lossless state transfer at designated sequence positions (“TransPoints”), with a scheduling strategy determining the switch points across layers. Experiments show TransMamba achieves superior efficiency (e.g., 0.75 relative training time vs. 1.00 for Transformer at 1.5B parameters) and performance on benchmarks like LongBench-v2 (38.76 overall score vs. 31.61 for Transformer-1.5B) compared to baseline Transformer, Mamba2, and static Hybrid models. For AI practitioners, TransMamba presents a scalable architecture potentially offering improved training/inference efficiency and performance, especially for applications involving variable sequence lengths, by dynamically selecting the optimal computation mechanism (Attention or SSM) per token segment and layer. |
| Comprehensive Relighting: Generalizable and Consistent Monocular Human | ||
| Relighting and Harmonization (Read more on arXiv or HuggingFace) | Zhixin Shu, Krishna Kumar Singh, Xin Sun, Jingyuan Liu, Junying Wang | This paper presents Comprehensive Relighting, a novel diffusion-based framework for generalizable and temporally consistent monocular human relighting and background harmonization. The main objective is to develop a single model capable of controllably relighting humans in images/videos (using Spherical Harmonics or background scenes), ensuring harmonization and temporal coherence across arbitrary body parts and scenes without large-scale supervised video data. The methodology utilizes a pre-trained latent diffusion model in a coarse-to-fine framework conditioned via ControlNet on coarse shading and background inputs, combined with an unsupervisedly trained temporal module (using cycle consistency) integrated via spatio-temporal feature blending and followed by guided refinement. Results show superior performance over baselines, achieving, for example, the best temporal consistency score (tLPIPS of 0.026, lower is better) on a challenging synthetic video benchmark (Scenario 3), compared to the next best (0.028). For AI practitioners, this work demonstrates adapting diffusion priors with conditioning and unsupervised temporal learning offers a potent strategy for tackling complex, data-limited generative video tasks, enabling the development of more robust and controllable video editing/synthesis tools. |
| EvMic: Event-based Non-contact sound recovery from effective | ||
| spatial-temporal modeling (Read more on arXiv or HuggingFace) | Lu Zhang, Xudong XU, Xu Jia, Shi Guo, yyzqy | EvMic introduces a deep learning pipeline for non-contact sound recovery using event cameras, overcoming traditional camera limitations. The objective is to effectively recover sound signals from object vibrations captured by event cameras by modeling spatial-temporal event data. The methodology employs a laser matrix for enhanced gradient capture, a synthetic dataset (EvMic) for training, and a network combining sparse convolutions, Mamba for temporal modeling, and a spatial aggregation block (SAB) for fusing information from multiple locations. The proposed method achieves superior performance on synthetic data, yielding an average SNR of 1.214 dB, significantly outperforming the EvPhase baseline (-0.079 dB). For AI practitioners, this demonstrates the potential of event-based vision and tailored architectures (sparse ConvNets, SSMs like Mamba, attention) for recovering high-frequency signals from subtle physical phenomena, offering a new modality for sensor fusion and signal processing tasks. |
| MedSAM2: Segment Anything in 3D Medical Images and Videos (Read more on arXiv or HuggingFace) | Mohammed Baharoon, Bihui Chen, Sumin Kim, Zongxin Yang, Jun Ma | MedSAM2 is a promptable foundation model for general-purpose 3D medical image and video segmentation. The objective was to create a versatile model capable of segmenting diverse structures across modalities by overcoming the 2D limitations of prior work and enabling efficient large-scale annotation. The methodology involved fine-tuning the lightweight SAM2.1-Tiny architecture on a large curated dataset (>455k 3D pairs, 76k video frames) using bounding box prompts and a human-in-the-loop iterative refinement process. Primary results demonstrate superior segmentation performance over baseline SAM2.1 models across CT, MRI, PET, ultrasound, and endoscopy data, alongside a user study showing an over 85% reduction in manual annotation time for 3D CT lesions. For AI practitioners, MedSAM2 provides an efficient, deployable tool integrated into common platforms (3D Slicer, Gradio, etc.) to significantly accelerate the creation of large-scale annotated medical datasets and streamline segmentation workflows. |
| BEATS: Bias Evaluation and Assessment Test Suite for Large Language | ||
| Models (Read more on arXiv or HuggingFace) | Lisa Erickson, tbandopa, alokabhishek | This research introduces BEATS, a framework and benchmark using 29 metrics to evaluate Bias, Ethics, Fairness, and Factuality (BEFF) in Large Language Models. The main objective was to develop a systematic framework and establish a standard benchmark for measuring and detecting BEFF metrics within LLMs. Key methodology involved using a curated dataset of 901 evaluation questions, performing inference on five major LLMs, and employing a consortium of three LLMs-as-judges to score responses based on the BEFF metrics, followed by statistical analysis including ANOVA. The primary result showed that 37.65% of generated outputs from tested industry-leading models contained some form of bias, indicating substantial risk. For AI practitioners, this implies a critical need for rigorous bias assessment using tools like BEATS before deploying LLMs, especially in sensitive applications, to inform necessary mitigation strategies. |
Papers for 2025-04-04
| Title | Authors | Summary |
|---|---|---|
| Advances and Challenges in Foundation Agents: From Brain-Inspired | ||
| Intelligence to Evolutionary, Collaborative, and Safe Systems (Read more on arXiv or HuggingFace) | KaitaoSong, JinlinW, Peiyan, xinfeng1i, Bang-UdeM-Mila | This survey presents a comprehensive overview of LLM-powered Foundation Agents, proposing a modular, brain-inspired architecture integrating cognitive science and neuroscience principles. The main objective is to structure the understanding of advanced intelligent agents by exploring their modular foundations, self-enhancement mechanisms, collaborative/evolutionary dynamics, and safety aspects. The methodology involves a structured literature review and synthesis, mapping agent components (memory, world modeling, reward, emotion) to brain functions and analyzing self-optimization (AutoML, LLM-driven), multi-agent systems, and safety/ethical threats. As a survey, the paper synthesizes existing research across these four areas rather than presenting novel quantitative findings, identifying key research gaps, challenges, and opportunities. For AI practitioners, this work provides a unified framework for designing, evaluating, and ensuring the safety of complex Foundation Agents, emphasizing the need to harmonize modular design, adaptive capabilities, and collaborative potential with robust safety and ethical considerations. |
| Envisioning Beyond the Pixels: Benchmarking Reasoning-Informed Visual | ||
| Editing (Read more on arXiv or HuggingFace) | Rethinker, GTZhai, KexianTang, zpy777, PhoenixZ | This paper introduces RISEBench, the first benchmark designed to evaluate Reasoning-Informed Visual Editing (RISE) capabilities in Large Multi-modality Models (LMMs). The main objective is to systematically assess LMM performance on visual editing tasks requiring Temporal, Causal, Spatial, and Logical reasoning beyond simple pixel manipulation. The methodology involves curating image-instruction test cases for each reasoning type and evaluating model outputs (from models like GPT-4o-Native, Gemini-2.0-Flash, EMU2) using both human judges and an LMM-as-a-judge (GPT-4o) framework across dimensions of Instruction Reasoning, Appearance Consistency, and Visual Plausibility. Primary results indicate that while GPT-4o-Native significantly outperforms other models with a 35.9% overall accuracy, even this state-of-the-art model struggles notably with logical reasoning tasks (37.5% accuracy), and open-source models achieve near-zero accuracy on RISEBench. The principal implication for AI practitioners is that current SOTA LMMs exhibit significant deficiencies in integrating complex, especially logical, reasoning within visual editing, highlighting a critical area requiring further research and development before such capabilities can be reliably deployed. |
| GPT-ImgEval: A Comprehensive Benchmark for Diagnosing GPT4o in Image | ||
| Generation (Read more on arXiv or HuggingFace) | shawnxyh, BestWishYsh, SereinH, liweijia, Yejy53 | This paper introduces GPT-ImgEval, a benchmark for quantitatively and qualitatively evaluating OpenAI’s GPT-4o model in image generation and editing tasks. The main objective was to assess GPT-4o’s performance across generation quality, editing proficiency, and world knowledge-informed synthesis, while also investigating its potential underlying architecture. Methodology involved evaluating GPT-4o using the GenEval, Reason-Edit, and WISE datasets via custom automation scripts, and employing a classification model trained to distinguish between diffusion and auto-regressive outputs to infer GPT-4o’s generation mechanism. Primary results indicate GPT-4o significantly surpasses prior models, achieving an overall score of 0.84 on GenEval, and empirical analysis suggests it likely uses a hybrid auto-regressive architecture combined with a diffusion-based head, contrary to VAR-like structures. For AI practitioners, this work provides a standardized evaluation framework, highlights GPT-4o’s advanced capabilities and specific limitations (e.g., editing inconsistencies, non-English text issues), and notes its outputs are detectable by current forensic models, impacting considerations for deployment and safety. |
| Rethinking RL Scaling for Vision Language Models: A Transparent, | ||
| From-Scratch Framework and Comprehensive Evaluation Scheme (Read more on arXiv or HuggingFace) | Pengfei, IanZhong, Ryan1122, steffichern, ManTle | This paper introduces MAYE, a transparent, from-scratch Reinforcement Learning (RL) framework for Vision-Language Models (VLMs), alongside a comprehensive evaluation scheme. The main objective is to improve reproducibility and standardized assessment in RL for VLMs, addressing limitations of complex, opaque existing frameworks. Methodologically, it presents a minimal four-step RL pipeline (using Reinforce++ with KL penalty) built with standard libraries and introduces an evaluation scheme tracking dynamics like accuracy curves, response length, and reflection ratios. Key results show RL consistently surpasses Supervised Fine-Tuning (SFT) generalization, achieving a 1.35x average accuracy increase (peaking at 1.76x) on the mm_math5k validation set compared to the baseline, even when SFT uses high-quality data; findings also indicate response length sensitivity to random seeds and correlation between reflection and output length. For AI practitioners, this provides a reproducible baseline framework (MAYE) for VLM RL experimentation and demonstrates RL’s potential for superior generalization over SFT on visual reasoning tasks, suggesting its utility even with access to good supervised data. |
| SkyReels-A2: Compose Anything in Video Diffusion Transformers (Read more on arXiv or HuggingFace) | raul678, ruiwang, diqiu7, Debang, onion | This paper introduces SkyReels-A2, an open-source framework for composing videos from text prompts and multiple reference images (characters, objects, scenes). The primary research objective is to generate high-fidelity videos that maintain strict identity consistency for each specified element while coherently composing the scene according to the text prompt, defining this as the elements-to-video (E2V) task. Key methodologies include a comprehensive data pipeline for constructing prompt-reference-video triplets, a novel joint image-text embedding model integrated into a diffusion transformer architecture with distinct spatial and semantic feature branches, and inference acceleration strategies. Evaluated on the proposed A2-Bench benchmark, SkyReels-A2 achieves comparable quantitative results to closed-source models, notably scoring 0.809 in object consistency, slightly outperforming competitors like Vidu (0.796) and Keling (0.790). For AI practitioners, SkyReels-A2 provides a publicly available model and benchmark for controllable multi-element video generation, facilitating development in areas requiring precise visual element control and composition, such as virtual e-commerce or creative content production. |
| Scaling Analysis of Interleaved Speech-Text Language Models (Read more on arXiv or HuggingFace) | adiyoss, MajoRoth, hassid, gallilmaimon | This paper analyzes the scaling behaviour of interleaved speech-text language models (SLMs), finding they scale more efficiently than textless SLMs. The main objective is to determine if SLMs trained with interleaved speech and text data scale more efficiently with compute compared to textless SLMs. The methodology involves training dozens of interleaved SLMs across various sizes (0.5B-7B), compute budgets (2e18-2e20 FLOPs), and TextLM initialisations (e.g., Qwen2.5, Llama3.2), evaluating performance on speech-only validation loss and semantic metrics (sSC, tSC) using an ISO-FLOP curve approach. Results show interleaved SLMs scale significantly better with compute, indicating compute budgets should favour larger model sizes over more training tokens; for a 2e20 FLOP budget, a 7B parameter model trained on 4.2B tokens outperformed smaller models on more tokens, contrasting with textless SLM scaling predictions. The principal implication for AI practitioners is that when training large interleaved SLMs (e.g., >4.5B tokens), allocating more compute towards larger, high-quality pre-trained TextLM-initialised models is more efficient than towards increasing training tokens alone for improving semantic speech abilities. |
| ShortV: Efficient Multimodal Large Language Models by Freezing Visual | ||
| Tokens in Ineffective Layers (Read more on arXiv or HuggingFace) | xphan, sanmusunrise, luyaojie, chenjiawei-icip, yuanqianhao | ShortV enhances Multimodal Large Language Model (MLLM) efficiency by identifying and freezing visual token computations in ineffective layers. The primary objective is to reduce the high computational overhead of MLLMs, specifically addressing redundancy in how different layers process visual tokens. A novel metric, Layer Contribution (LC), is introduced to quantify a layer’s impact by measuring the KL divergence in model output logits when that layer’s transformations on specific tokens (visual or text) are bypassed; ShortV uses LC to identify layers ineffective for visual tokens and replaces them with sparse layers where visual computations are frozen. Experiments demonstrate that ShortV can freeze visual token processing in approximately 60% of MLLM layers (e.g., achieving 50% FLOPs reduction on LLaVA-NeXT-13B with N=24 replaced layers) with negligible performance degradation. For AI practitioners, ShortV offers a training-free, parameter-free method to significantly decrease MLLM inference costs by exploiting layer-wise redundancy for visual tokens, and it is compatible with token pruning techniques. |
| Audio-visual Controlled Video Diffusion with Masked Selective State | ||
| Spaces Modeling for Natural Talking Head Generation (Read more on arXiv or HuggingFace) | Jun Zhou, Zixiang Zhou, danxuhk, xuzn, HarlanHong | This paper introduces ACTalker, an end-to-end video diffusion framework for natural talking head generation controlled simultaneously by audio and facial motion signals without conflict. The primary objective is to enable fine-grained control using multiple driving signals while preventing conflicts and ensuring spatio-temporal coherence. Key methodologies involve a parallel-control mamba (PCM) layer leveraging Masked Selective State Space Models (Mask-SSM) and a mask-drop strategy to direct each signal’s influence to specific facial regions within a stable video diffusion architecture. Experimental results demonstrate state-of-the-art performance, achieving a Sync-C score of 5.317 and an FVD-Inc score of 232.374 on the CelebV-HD dataset under audio-only control, surpassing previous methods. For AI practitioners, this work presents a novel application of Mamba (SSM) structures for efficient, conflict-free multi-modal conditioning in video generation, offering precise control over synthesized facial dynamics. |
| ZClip: Adaptive Spike Mitigation for LLM Pre-Training (Read more on arXiv or HuggingFace) | gueraf, nilabhra, louisowen6, akanyaani | ZClip introduces an adaptive gradient clipping method based on z-scores to enhance stability during large language model (LLM) pre-training. The primary objective is to mitigate gradient instability and malignant loss spikes that disrupt training, necessitating costly interventions like checkpoint restoration. ZClip dynamically adjusts the gradient clipping threshold by tracking the exponential moving average (EMA) of the gradient norm’s mean and standard deviation, applying z-score-based anomaly detection to identify and scale down spikes. Experiments on a 1B LLaMA model demonstrated that ZClip enabled stable training at a high learning rate (3.0x10⁻³), reaching baseline validation loss using 18.6B fewer tokens (over 35% faster) compared to fixed clipping at a lower, stable rate (5.0x10⁻⁴). For AI practitioners, ZClip offers a method to improve LLM pre-training stability and efficiency, potentially reducing training time and compute costs by allowing for more aggressive learning rates without succumbing to catastrophic divergence. |
| Inference-Time Scaling for Generalist Reward Modeling (Read more on arXiv or HuggingFace) | Chong Ruan, Shirong Ma, Runxin Xu, Peiyi Wang, Zijun Liu | This paper introduces Self-Principled Critique Tuning (SPCT) to enhance the inference-time scalability and performance of generalist generative reward models (GRMs). The main objective is to investigate if a specific learning method can enable effective inference-time scaling for GRMs, improving reward quality beyond standard model or compute scaling. The key methodology involves SPCT, which uses rejective fine-tuning and rule-based online RL to train GRMs to generate adaptive principles and critiques, combined with inference-time scaling via parallel sampling and voting, optionally guided by a meta RM. Primary results show DeepSeek-GRM-27B trained with SPCT achieves 69.9% overall accuracy on RM benchmarks, improving to 71.0% with voting@32, and further to 72.8% with meta RM guidance, demonstrating effective inference-time scaling compared to just increasing model size. For AI practitioners, this implies that using SPCT and inference-time sampling with GRMs can yield superior reward signals for aligning LLMs, potentially offering a more compute-efficient path to performance gains than solely relying on larger models. |
| Efficient Model Selection for Time Series Forecasting via LLMs (Read more on arXiv or HuggingFace) | Hongjie Chen, Franck-Dernoncourt, ryanrossi, tiankaiy, wwdd7718 | This paper investigates leveraging Large Language Models (LLMs) for efficient, zero-shot model selection in time series forecasting, eliminating the need for costly pre-computed performance matrices. The primary objective is to determine if LLMs can select optimal forecasting models and hyperparameters for unseen time series datasets solely through prompting. The methodology involves querying LLMs (Llama 3.2, GPT-4o, Gemini 2.0 flash) with prompts containing time series data and optionally meta-features or Chain-of-Thought (CoT) instructions to recommend a model configuration. Results demonstrate that the LLM approach, particularly Llama 3.2 using prompts with meta-features, outperforms traditional meta-learning (e.g., achieving 7.27% hit@10 accuracy vs. 4.51% for MLP) and heuristic baselines while reducing median inference time by up to 89x compared to naïve exhaustive evaluation. For AI practitioners, this suggests LLMs offer a computationally cheaper and faster alternative for selecting appropriate time series forecasting models without extensive prior model evaluations or meta-feature engineering, streamlining the model selection workflow. |
| Instruction-Guided Autoregressive Neural Network Parameter Generation (Read more on arXiv or HuggingFace) | Sung Ju Hwang, Song Chong, Bruno Andreis, bedio | This paper introduces IGPG, an instruction-guided autoregressive framework for generating neural network parameters conditioned on task and architecture specifications. The primary objective is to enable scalable and coherent parameter synthesis across diverse models and tasks, addressing limitations of prior methods like diffusion models. IGPG utilizes a VQ-VAE to tokenize parameters and an autoregressive transformer, conditioned on task/dataset embeddings and architecture descriptions, to generate weight tokens sequentially. Key results demonstrate competitive performance, including generating LoRA parameters that improve accuracy by up to 10% over baseline methods on vision benchmarks. For AI practitioners, IGPG offers a unified tool for rapid model initialization, efficient adaptation to new tasks, and potentially reduces the need for extensive fine-tuning by generating specialized weights on demand. |
| Interpreting Emergent Planning in Model-Free Reinforcement Learning (Read more on arXiv or HuggingFace) | David Krueger, Usman Anwar, Stephen Chung, agaralon, tuphs | This paper provides the first mechanistic evidence that model-free reinforcement learning agents (DRC) learn internal planning mechanisms in Sokoban using concept-based interpretability. The primary research objective was to determine if a DRC agent internally formulates, evaluates, and utilizes plans based on predicted future consequences without an explicit world model. The methodology involved probing ConvLSTM cell states for planning-relevant concepts (Agent Approach Direction CA, Box Push Direction CB), analyzing iterative plan formation across internal ticks, and performing causal interventions on activations to verify behavioral dependence. Results show the agent linearly represents CA and CB (e.g., final layer 1x1 probe Macro F1 for CB ~0.8 vs <0.3 baseline), forms plans iteratively resembling parallelized bidirectional search which refine with extra compute (Fig 6), and interventions causally steer behavior (e.g., 98.8% success rate for Layer 3 Agent-Shortcut interventions). The principal implication for AI practitioners is that complex planning capabilities can emerge implicitly in model-free architectures, suggesting that internal state representations and iterative computation may be key mechanisms for such behaviors, influencing agent design and analysis beyond purely behavioral metrics. |
| GenPRM: Scaling Test-Time Compute of Process Reward Models via | ||
| Generative Reasoning (Read more on arXiv or HuggingFace) | Saputello, dmux, ChetKao, iseesaw, RyanLiu112 | GenPRM introduces a generative process reward model utilizing explicit reasoning and code verification to scale test-time compute for LLM verification. The objective is to overcome limitations of current Process Reward Models (PRMs) by enhancing their process supervision capabilities and enabling test-time scaling (TTS) through generative modeling. GenPRM achieves this by performing multi-step Chain-of-Thought (CoT) reasoning integrated with code generation and execution for verification, using Relative Progress Estimation (RPE) and rationale synthesis for training data generation. Experiments demonstrate that a 7B GenPRM significantly outperforms prior models, surpassing the much larger Qwen2.5-Math-PRM-72B on ProcessBench (achieving 80.5 F1 score with Maj@8 scaling). For AI practitioners, this work shows that smaller generative PRMs, when combined with test-time scaling, can serve as highly effective and potentially more compute-efficient verifiers or critics compared to larger models or traditional scalar-based PRMs, improving the evaluation and refinement of complex reasoning processes. |
| Scaling Laws in Scientific Discovery with AI and Robot Scientists (Read more on arXiv or HuggingFace) | Zhenting Wang, Renjun Xu, Huazhe Xu, Heng Zhang, universea | This paper proposes the Autonomous Generalist Scientist (AGS) concept, integrating agentic AI and embodied robotics to automate the end-to-end scientific research lifecycle. The main objective is to outline a framework for AGS systems capable of independent, multi-domain scientific discovery by synergizing AI’s cognitive abilities with robotics’ physical interaction capabilities. The methodology involves proposing a conceptual framework featuring a five-module architecture (literature review, proposal generation, experimentation, manuscript writing, reflection/feedback) and defining five distinct levels of automation, ranging from Level 1 (Tool-Assisted) to Level 5 (Pioneer/ASIR). The paper hypothesizes new scaling laws for scientific discovery driven by AGS capabilities and number, rather than presenting empirical results; it details requirements for virtual (OS agents) and physical (embodied AI robots) task execution. For AI practitioners, the primary implication is the conceptual roadmap for developing integrated AI-robotic systems capable of complex, multi-stage, cross-domain automation, moving beyond specialized AI tools to handle tasks requiring both virtual reasoning and physical manipulation. |
| Sparse Autoencoders Learn Monosemantic Features in Vision-Language | ||
| Models (Read more on arXiv or HuggingFace) | Zeynep Akata, Serge Belongie, Quentin Bouniot, Shyamgopal Karthik, Mateusz Pach | This work extends Sparse Autoencoders (SAEs) to Vision-Language Models (VLMs) like CLIP, demonstrating their ability to learn more interpretable, monosemantic features from vision representations. The primary objective is to quantitatively evaluate whether SAEs applied post-hoc to VLM activations enhance neuron monosemanticity and enable model control. Methodology involves training various SAE types on CLIP layer activations and introducing a Monosemanticity Score (MS) metric, calculating activation-weighted pairwise image embedding similarity for neurons. Results demonstrate SAE neurons achieve significantly higher monosemanticity (e.g., MS increased from 0.48 in the base VLM to 0.81 with an SAE for specific neurons shown) and reveal hierarchical concept structures, especially with Matryoshka SAEs. For AI practitioners, this research validates SAEs as an unsupervised method to interpret VLM representations and directly steer the output concepts of multimodal LLMs like LLaVA by intervening on SAE activations, without modifying the base model. |
| Whisper-LM: Improving ASR Models with Language Models for Low-Resource | ||
| Languages (Read more on arXiv or HuggingFace) | Ibon Saratxaga, Eva Navas, inmahernaez, zuazo | This research improves Whisper ASR models for low-resource languages by integrating external n-gram and large language models (LLMs) with fine-tuned models at inference time. The main objective was to enhance transcription accuracy and robustness, particularly in low-resource and out-of-distribution scenarios, by combining acoustic model probabilities with language model scores. Key methodology involved fine-tuning Whisper models per language, followed by integrating KenLM 5-gram models or language-specific LLMs by modifying beam search scores using optimized weighting parameters. Primary results demonstrate substantial Word Error Rate (WER) reductions, achieving up to 51% improvement for in-distribution Basque data with 5-gram models, while LLMs offered consistently robust, albeit more moderate, gains across languages. For AI practitioners, this indicates that integrating external LMs significantly boosts Whisper’s performance for under-resourced languages, but optimal performance requires careful language model parameter tuning and attention to evaluation settings. |
Papers for 2025-04-03
| Title | Authors | Summary |
|---|---|---|
| MergeVQ: A Unified Framework for Visual Generation and Representation | ||
| with Disentangled Token Merging and Quantization (Read more on arXiv or HuggingFace) | Cheng Tan, Juanxi, ZedongWangAI, LuyuanZhang01, Lupin1998 | MergeVQ presents a unified framework integrating token merging into VQ-based models to balance visual representation learning and autoregressive generation. The primary objective is to overcome the trade-off between generation quality, representation learning, and efficiency inherent in existing VQ-MIM approaches. Key methodologies include disentangling semantics via token merging (ToMe) while preserving spatial details in a recoverable source matrix, employing Look-up Free Quantization (LFQ), using cross-attention for detail recovery, global alignment via self-distillation (DINO), and introducing MergeAR with KV Cache compression for efficient generation. Experiments on ImageNet-1K show the representation-focused variant achieves 79.8% linear probe accuracy using only 36 merged tokens, while the generative variant achieves a competitive class-conditional generation gFID of 3.05 using MergeAR. For AI practitioners, MergeVQ offers a pathway to build more computationally efficient unified vision models, as demonstrated by its ability to achieve strong representation learning performance with significantly reduced token counts (36 tokens), potentially lowering pre-training and inference costs. |
| Improved Visual-Spatial Reasoning via R1-Zero-Like Training (Read more on arXiv or HuggingFace) | Zijian Kong, Yanhao Zhang, Qingsong Xie, Zhenyi Liao, zhijie3 | This work enhances visual-spatial reasoning in Multimodal Large Language Models (MLLMs) using R1-Zero-like GRPO training. The primary objective was to improve visual-spatial intelligence (VSI) capabilities, particularly in small- to medium-sized Qwen2-VL models where Chain of Thought (CoT) prompting proved ineffective. The key methodology involved constructing the VSI-100k dataset from ScanNet and applying Group Relative Policy Optimization (GRPO) while identifying the necessity of retaining the KL penalty. The resulting vsGRPO-2B model outperformed its Qwen2-VL-2B base by 12.1% on the VSI-bench benchmark and surpassed GPT-4o performance. For AI practitioners, this demonstrates that GRPO training with curated datasets is a potent technique to specifically boost MLLM reasoning faculties like VSI, offering substantial gains over base models and even surpassing larger or closed-source alternatives for targeted tasks. |
| AnimeGamer: Infinite Anime Life Simulation with Next Game State | ||
| Prediction (Read more on arXiv or HuggingFace) | Ying Shan, Jing Liao, Yixiao Ge, Yuying Ge, Howe666 | AnimeGamer introduces an MLLM-based framework for generating infinite, interactive anime life simulation games featuring dynamic video outputs and character state updates from language instructions. The primary objective is to create contextually consistent and dynamic multi-turn game states, addressing limitations of prior static image or text-only methods. The key methodology involves using an MLLM to predict novel action-aware multimodal representations from historical context and instructions, which are then decoded into video clips using a fine-tuned video diffusion model alongside character state prediction. AnimeGamer significantly outperforms baselines in quantitative evaluations, achieving higher character consistency (CLIP-I 0.8132 vs. 0.7960) and superior motion quality (ACC-F 0.6744 vs. 0.4249). For AI practitioners, this work demonstrates an effective approach using MLLMs to generate coherent, dynamic video-based interactive experiences by bridging language and video synthesis via specialized multimodal representations, enhancing immersion in generative games. |
| VideoScene: Distilling Video Diffusion Model to Generate 3D Scenes in | ||
| One Step (Read more on arXiv or HuggingFace) | Yueqi Duan, Jiawei Chi, Fangfu Liu, hanyang-21 | VideoScene introduces a framework to distill video diffusion models for efficient, one-step 3D scene generation from only two input images. The main objective is to bridge the gap between slow, multi-step video diffusion methods and the need for fast, 3D-consistent scene generation from sparse views. The key methodology involves a 3D-aware leap flow distillation strategy, initialized using a coarse scene from a feed-forward 3DGS model (MVSplat), and a dynamic denoising policy network (DDPNet) trained via contextual bandits to optimize leap timesteps. Primarily, VideoScene achieves significantly faster inference (~3s) while maintaining high quality; its 1-step generation on RealEstate10K yields an FVD of 103.42, vastly outperforming 1-step baselines and remaining competitive with their 50-step versions (e.g., CogVideoX-5B 50-step FVD 521.04). For AI practitioners, this offers an efficient tool for generating temporally coherent and geometrically consistent 3D video sequences from minimal input, drastically reducing computational cost for sparse-view 3D reconstruction tasks. |
| Understanding R1-Zero-Like Training: A Critical Perspective (Read more on arXiv or HuggingFace) | Tianyu Pang, Wenjun Li, QPHutu, Cameron-Chen, lkevinzc | This paper critically analyzes R1-Zero-like LLM training, examining base model properties and RL optimization biases, particularly in GRPO. The primary objective is to understand how base model pretraining affects RL outcomes and to identify and mitigate biases in the GRPO algorithm. Methodology includes evaluating various base models (e.g., Qwen2.5, DeepSeek-V3-Base) on math benchmarks with different templates and comparing GRPO against a proposed unbiased variant, Dr. GRPO, in RL experiments. Key findings demonstrate that some base models exhibit strong initial reasoning (Qwen2.5 improves ~60% without templates), GRPO introduces length and standard deviation normalization biases impacting token efficiency, and the proposed Dr. GRPO optimizer corrects these, enabling a 7B model to achieve 43.3% accuracy on AIME 2024. The principal implication for practitioners is that understanding base model capabilities and utilizing unbiased RL optimizers like Dr. GRPO are essential for efficient reasoning enhancement, avoiding artifactual response length increases from biased optimization. |
| DreamActor-M1: Holistic, Expressive and Robust Human Image Animation | ||
| with Hybrid Guidance (Read more on arXiv or HuggingFace) | Tianshu Hu, Longhao Zhang, Lizhen Wang, Zhengkun Rong, Yuxuan Luo | This paper introduces DreamActor-M1, a Diffusion Transformer (DiT) based framework for robust human image animation. The primary objective is to overcome limitations in existing methods regarding fine-grained holistic control, multi-scale adaptability (portraits to full-body), and long-term temporal coherence, particularly for unseen regions. Key methodologies include using hybrid motion guidance signals (implicit facial latent representations, 3D head spheres, 3D body skeletons with bone length adjustment), complementary appearance guidance for unseen areas, and a progressive multi-scale training strategy. The proposed method achieved superior quantitative results, for instance, an FVD score of 122.0 on their collected body animation dataset, outperforming prior works like Animate Anyone (158.3) and MimicMotion (149.9). For AI practitioners, this work demonstrates a robust DiT-based approach with hybrid explicit/implicit controls and appearance guidance, enabling the generation of higher-fidelity, more expressive, and temporally consistent human animations across diverse scales and viewpoints. |
| PaperBench: Evaluating AI’s Ability to Replicate AI Research (Read more on arXiv or HuggingFace) | Jun Shern Chan, James Aung, Dane Sherburn, Oliver Jaffe, Giulio Starace | PaperBench introduces a benchmark to evaluate AI agents’ ability to replicate state-of-the-art AI research papers from scratch. The objective is to assess how well AI agents can understand paper contributions, develop codebases, and execute experiments to reproduce empirical results. The methodology involves providing agents with 20 ICML 2024 papers and using detailed, author-approved hierarchical rubrics alongside an LLM-based judge to evaluate the agent-generated code repository and its execution outputs. Results show the best agent, Claude 3.5 Sonnet with scaffolding, achieved an average replication score of 21.0%, significantly lower than a human expert baseline (41.4% on a subset), indicating current models have limited autonomous AI R&D replication capabilities. For AI practitioners, this highlights that while agents show nascent ability, they are not yet proficient at the complex, long-horizon task of independently replicating and validating frontier AI research, requiring substantial human oversight for such tasks. |
| ScholarCopilot: Training Large Language Models for Academic Writing with | ||
| Accurate Citations (Read more on arXiv or HuggingFace) | Zhiheng Lyu, Huaye Zeng, Ping Nie, Xueguang Ma, Yubo Wang | ScholarCopilot introduces a unified framework for training LLMs to generate academic text with accurate, context-aware citations. The main objective is to overcome limitations of traditional RAG systems by integrating dynamic retrieval directly into the generation process for improved citation relevance and quality in academic writing. The methodology involves dynamically generating special retrieval tokens ([RET]) during text generation, using their representations for similarity search against a database, and feeding retrieved references back into the model, optimizing generation and retrieval jointly. ScholarCopilot achieved 40.1% top-1 retrieval accuracy, significantly outperforming E5-Mistral-7B-Instruct (15.0%), and obtained a generation quality score of 16.2/25, surpassing larger models like Qwen-2.5-72B-Instruct (15.8/25). For AI practitioners, this work demonstrates a unified, dynamic RAG approach that can enhance LLM factual accuracy and contextual relevance for specialized generation tasks requiring precise citations, offering a potentially more efficient alternative to separate retrieval/generation pipelines. |
| Towards Physically Plausible Video Generation via VLM Planning (Read more on arXiv or HuggingFace) | Lei Bai, Zhenfei Yin, Yiming Zhang, Baolu Li, Xindi Yang | This paper proposes a two-stage framework using a Vision Language Model (VLM) planner and a Video Diffusion Model (VDM) synthesizer to generate physically plausible videos. The objective is to enhance physical plausibility in video generation by explicitly incorporating physics priors, addressing the limitations of standard VDMs in understanding physical laws. The methodology involves a VLM performing coarse-grained, physics-aware motion planning via chain-of-thought (CoT) reasoning to predict rough object trajectories, which then guide a VDM through injected structured noise derived from optical flow for fine-level motion synthesis. Quantitative results on the PhyGenBench benchmark show the proposed method achieved an average score of 0.60, outperforming the best compared image-to-video method (SG-I2V at 0.54) by 11.1% in physical plausibility assessment. For AI practitioners, this demonstrates a method to integrate explicit physical reasoning from VLMs into VDMs to improve the realism and physical consistency of generated video content, particularly for scenarios involving object interactions governed by physics. |
| ILLUME+: Illuminating Unified MLLM with Dual Visual Tokenization and | ||
| Diffusion Refinement (Read more on arXiv or HuggingFace) | Yunlong Yuan, Guansong Lu, Junwei Yang, Chunwei Wang, Runhui Huang | ILLUME+ enhances unified Multimodal Large Language Models (MLLMs) by integrating dual visual tokenization and diffusion refinement for improved understanding, generation, and editing. The objective is to create a single MLLM that overcomes limitations of prior models, such as poor texture preservation in editing or weaker semantic understanding, by effectively unifying these three core capabilities. Key methodologies include the DualViTok tokenizer preserving both semantic and texture details, a diffusion model decoder for high-fidelity image reconstruction and super-resolution, and a coarse-to-fine image representation strategy within the MLLM. Primary results show the 3B parameter ILLUME+ achieves competitive performance across understanding, generation, and editing benchmarks, including an improved Fréchet Inception Distance (FID) of 6.00 on the MJHQ-30k generation benchmark compared to its predecessor. For AI practitioners, this work presents a unified model architecture that supports flexible resolution inputs/outputs and demonstrates strong performance in fine-grained editing tasks, potentially offering a more versatile foundation for complex, interactive multimodal applications. |
| Articulated Kinematics Distillation from Video Diffusion Models (Read more on arXiv or HuggingFace) | Chenfanfu Jiang, Yongxin Chen, Tsung-Yi Lin, Qianli Ma, Xuan Li | Articulated Kinematics Distillation (AKD) synthesizes articulated motions for rigged 3D assets by leveraging video diffusion models. The objective is to generate high-fidelity, structurally consistent character animations from text prompts, addressing limitations of prior text-to-4D methods based on neural deformation fields. AKD utilizes a low-DoF skeleton-based representation optimized via Score Distillation Sampling (SDS) with a pre-trained video diffusion model, incorporating explicit ground rendering and optional physics-based motion tracking. Experiments show AKD outperforms TC4D, achieving higher automated scores (e.g., Semantic Adherence 0.81±0.26 vs 0.40±0.34) and preference in user studies for motion quality and physical plausibility. For AI practitioners, AKD offers a method to generate controllable, physically grounded 3D character animations from text by effectively combining generative video priors with explicit articulated structure, improving consistency over deformation field approaches. |
| Safeguarding Vision-Language Models: Mitigating Vulnerabilities to | ||
| Gaussian Noise in Perturbation-based Attacks (Read more on arXiv or HuggingFace) | Zhendong Liu, Yushen Zuo, sofyc, AllenChai, Jarvis1111 | This paper investigates Vision-Language Model (VLM) vulnerability to Gaussian noise perturbations and proposes noise-augmented fine-tuning and a diffusion-based defense (DiffPure-VLM) to mitigate these risks. The primary objective is to systematically analyze VLM robustness against visual Gaussian noise and develop effective defense strategies against both simple noise and optimization-based adversarial attacks while preserving model helpfulness. Key methodologies include creating the Robust-VLGuard dataset with aligned/misaligned safety pairs, employing Gaussian noise augmentation during safety fine-tuning, and proposing the DiffPure-VLM pipeline which uses diffusion models to transform adversarial perturbations into Gaussian-like noise manageable by the fine-tuned VLMs. Primary results demonstrate that while baseline VLMs degrade significantly under Gaussian noise, the proposed noise-augmented fine-tuning enhances robustness, and DiffPure-VLM substantially reduces optimization-based attack success rates; for example, with InternVL2-8B-RobustVLGuard under a €=32/255 attack, DiffPure-VLM (t*=50) lowered the attack success rate from 70.6% to 33.4%. For AI practitioners, this implies that incorporating noise-augmented safety fine-tuning and employing diffusion-based preprocessing defenses like DiffPure-VLM are practical strategies to significantly bolster VLM security against visual perturbation attacks without excessive computational overhead or loss of core functionality. |
| Boost Your Own Human Image Generation Model via Direct Preference | ||
| Optimization with AI Feedback (Read more on arXiv or HuggingFace) | Hyunjoon Lee, Yonggyu Kim, sanghyeonna | This paper introduces HG-DPO, a method enhancing human image generation realism by applying Direct Preference Optimization (DPO) with real images and curriculum learning. The main objective is to improve diffusion models for human image synthesis by overcoming the limitations of standard DPO, which typically relies only on generated images. HG-DPO utilizes a novel preference structure where real images serve as preferred (winning) examples and generated images as non-preferred (losing), combined with a three-stage curriculum learning pipeline (easy, normal, hard) and AI feedback for dataset construction. Results demonstrate HG-DPO significantly outperforms baseline models and prior DPO methods, achieving a lower FID of 29.41 compared to the base model’s 37.34 and higher CI-S of 0.9858 versus 0.9573. For AI practitioners, this provides a framework to boost the quality and realism of text-to-human image generation models by effectively integrating real-world image data into the preference learning process without costly human annotation, and enhances personalized generation tasks. |
| DASH: Detection and Assessment of Systematic Hallucinations of VLMs (Read more on arXiv or HuggingFace) | Matthias Hein, Maximilian Augustin, YanNeu | This paper introduces DASH, an automated pipeline for detecting systematic false-positive object hallucinations in Vision-Language Models (VLMs) using large-scale, real-world image data. The main objective is to systematically identify clusters of semantically similar real-world images that cause a VLM to incorrectly affirm the presence of an object not actually depicted. Key methodologies include DASH-LLM, which uses LLM-generated text queries for image retrieval, and DASH-OPT, which optimizes latent diffusion model inputs to generate misleading images, both followed by k-NN retrieval on ReLAION-5B and clustering. Applying DASH to PaliGemma and two LLaVA-NeXT models across 380 objects yielded over 19k hallucination clusters containing over 950k images; fine-tuning PaliGemma on DASH-identified images improved accuracy on the derived DASH-B benchmark by 11.6%. For AI practitioners, this work highlights that significant object hallucination issues persist beyond standard benchmarks, necessitating open-world testing methods like DASH for reliable VLM assessment and providing datasets (DASH-B) for more rigorous evaluation and potential mitigation fine-tuning. |
| LSNet: See Large, Focus Small (Read more on arXiv or HuggingFace) | Guiguang Ding, Jungong Han, Zijia Lin, Hui Chen, jameslahm | LSNet introduces a lightweight vision network family leveraging a novel LS convolution inspired by the human vision system’s “See Large, Focus Small” strategy. The primary objective is to enhance the performance and efficiency balance in lightweight models by improving the token mixing process, specifically perception and aggregation under limited computational budgets. The key methodology involves the proposed LS (Large-Small) convolution, which uses large-kernel static depth-wise convolution for broad perception and small-kernel grouped dynamic convolution for adaptive, focused aggregation. Results demonstrate state-of-the-art performance; for instance, LSNet-B achieves 80.3% top-1 accuracy on ImageNet-1K with 1.3G FLOPs, outperforming comparable models like AFFNet and RepViT-M1.1 in both accuracy and efficiency. For AI practitioners, LSNet provides a new efficient architectural block (LS convolution) and model series offering improved accuracy-efficiency trade-offs for vision tasks deployed on resource-constrained platforms. |
| VerifiAgent: a Unified Verification Agent in Language Model Reasoning (Read more on arXiv or HuggingFace) | Ehsan Shareghi, Wray Buntine, Jiuzhou Han | This paper introduces VerifiAgent, a unified agent employing two verification levels (meta and tool-based adaptive) to enhance large language model (LLM) reasoning reliability. The main research objective is to develop a generalisable and efficient verification framework for diverse LLM reasoning tasks, overcoming the limitations of current methods. VerifiAgent utilizes a two-layer methodology involving meta-verification for completeness and consistency, followed by tool-based adaptive verification which autonomously selects external tools (e.g., Python interpreter, search engine, symbolic solver) based on the reasoning type. Experimental results show VerifiAgent outperforms baseline verification methods across mathematical, logical, commonsense, and hybrid reasoning tasks, achieving 0.96 accuracy on GSM8K compared to baselines like deductive verifier (0.95). For AI practitioners, VerifiAgent offers a plug-and-play framework to improve the reliability and accuracy of LLM reasoning outputs, particularly in inference scaling scenarios, achieving better results with fewer samples and lower cost than methods like PRMs. |
| Enhanced OoD Detection through Cross-Modal Alignment of Multi-Modal | ||
| Representations (Read more on arXiv or HuggingFace) | Sangheum Hwang, mawjdgus | This paper introduces Cross-Modal Alignment (CMA), a multi-modal fine-tuning method to enhance Out-of-Distribution (OoD) detection in vision-language models. The primary objective is to improve OoD performance by mitigating the modality gap observed between image and text embeddings during standard fine-tuning. CMA employs a regularization loss during fine-tuning to explicitly align in-distribution image-text embedding pairs in the hyperspherical representation space, shown theoretically to correspond to maximizing the log-likelihood of a joint energy-based model. The proposed CMA method, when combined with the NegLabel scoring function, achieved state-of-the-art OoD performance on the MOS benchmark, attaining an average FPR95 of 19.93% and 95.13% AUROC, significantly outperforming existing zero-shot and fine-tuning approaches while maintaining high ID accuracy (82.64% on ImageNet-1k). For AI practitioners, this work demonstrates that explicitly regularizing for cross-modal alignment during fine-tuning can substantially improve model robustness by enhancing both OoD detection and in-distribution classification, thereby increasing the reliability of VLMs deployed in open environments. |
Papers for 2025-04-02
| Title | Authors | Summary |
|---|---|---|
| Any2Caption:Interpreting Any Condition to Caption for Controllable Video | ||
| Generation (Read more on arXiv or HuggingFace) | shuicheng, dizhang, Xintao, WeicaiYe, ChocoWu | Any2Caption introduces an MLLM-based framework to interpret diverse multimodal conditions into structured captions for controllable video generation. The main objective is to accurately interpret complex user intent from various inputs (text, images, specialized cues like pose/camera) to improve video synthesis control and quality. The methodology involves decoupling interpretation from generation, using a Qwen2-LLM with dedicated encoders to generate detailed, structured captions, trained on the new Any2CapIns dataset (337K instances). Results show high caption fidelity (e.g., 91.95 BERTSCORE) and improved video quality and controllability when integrated with existing video generators across various conditions. For AI practitioners, the key implication is the ability to enhance control over existing video generation models using complex multimodal inputs by integrating this interpretation module, which outputs structured text captions, without needing to retrain the core video generator. |
| Exploring the Effect of Reinforcement Learning on Video Understanding: | ||
| Insights from SEED-Bench-R1 (Read more on arXiv or HuggingFace) | yshan2u, yxgeee, ruiwang, tttoaster, ChenYi99 | SEED-Bench-R1 is introduced to systematically evaluate reinforcement learning (RL) post-training for multimodal large language model (MLLM) video understanding. The primary objective is to compare the effectiveness and generalization of RL (specifically GRPO) against supervised fine-tuning (SFT) for video tasks requiring both perception and logical reasoning. Using Qwen2-VL-Instruct-7B, the study compared GRPO trained with outcome-based rewards against SFT on the hierarchical SEED-Bench-R1 benchmark (L1: In-distribution, L2/L3: OOD). Results show GRPO significantly outperforms SFT in data efficiency and generalization, particularly in OOD scenarios (e.g., 44.89% vs 38.15% accuracy on Level-3), and extends generalization benefits to benchmarks like LongVideoBench (43.40% vs 40.00%). For AI practitioners, this implies RL, even with simple outcome rewards, is highly effective at enhancing MLLM visual perception and OOD generalization for video tasks compared to SFT, though analysis notes RL may compromise logical coherence in the reasoning chain. |
| CodeARC: Benchmarking Reasoning Capabilities of LLM Agents for Inductive | ||
| Program Synthesis (Read more on arXiv or HuggingFace) | Naveen Kannan, Jiannan Cao, kaiyan289, tarsur909, anjiangwei | This paper introduces CodeARC, an interactive benchmark evaluating LLM agents on inductive program synthesis. The main objective is to assess LLMs’ ability to infer hidden functions solely from input-output examples through interaction, departing from static evaluation protocols. Key methodology involves agents querying a hidden target function, synthesizing candidates, and using a differential testing oracle for feedback and iterative refinement under budget constraints on 1114 Python functions. Primary results indicate the task is challenging: the best-performing model, o3-mini, achieved a 52.7% success rate on the anonymized dataset, and fine-tuning LLaMA-3.1-8B-Instruct improved performance by up to 31% relatively. For AI practitioners, this work provides a more realistic benchmark revealing significant limitations in current LLMs’ inductive reasoning for code synthesis and suggests interactive refinement and targeted fine-tuning as avenues for improvement. |
| JudgeLRM: Large Reasoning Models as a Judge (Read more on arXiv or HuggingFace) | Jiaying Wu, Nuo Chen, bhooi, qingyunzou, zhiyuanhucs | This paper introduces JudgeLRM, a family of LLMs trained via reinforcement learning (RL) to serve as evaluators, specifically targeting complex reasoning tasks where SFT judges falter. The research investigates whether enhancing reasoning capabilities improves LLM judge performance and proposes an RL-based training approach using judge-wise, outcome-driven rewards. Key methodology involves training base LLMs (Qwen2.5) using Group Relative Policy Optimization (GRPO) with a custom reward function combining structural correctness and content alignment (relation, absolute, confidence metrics) against ground-truth judgments. Primary results show JudgeLRM models outperform SFT-tuned and state-of-the-art reasoning models; notably, JudgeLRM-7B surpasses DeepSeek-R1 by 2.79% in F1 score on the JudgeLM benchmark, excelling particularly on tasks requiring deep reasoning. For AI practitioners, this implies that RL with carefully designed, reasoning-focused rewards is a more effective method than SFT for developing robust LLM evaluators capable of handling nuanced, complex judgment tasks, suggesting RL should be considered for building reliable automated evaluation systems. |
| GeometryCrafter: Consistent Geometry Estimation for Open-world Videos | ||
| with Diffusion Priors (Read more on arXiv or HuggingFace) | Xiaoyu Li, yshan2u, wbhu-tc, xiangjun0211, slothfulxtx | GeometryCrafter generates temporally consistent, metrically accurate point map sequences from open-world videos using diffusion priors. The main objective is to estimate high-fidelity, temporally coherent point maps with correct metric scale from videos, overcoming the affine ambiguity and temporal inconsistency limitations of prior diffusion-based depth and geometry estimation methods. The key methodology employs a novel point map Variational Autoencoder (VAE) with a dual-encoder design (using an inherited SVD encoder and a residual encoder) to encode unbounded point maps while maintaining latent compatibility, integrated with a video diffusion model finetuned using these latents and per-frame geometry priors. Primary results demonstrate state-of-the-art performance, achieving an average rank of 1.9 on point map estimation across seven diverse benchmark datasets, indicating superior 3D accuracy and temporal consistency compared to previous methods. For AI practitioners, this provides a framework to extract metrically accurate, temporally consistent geometry from videos, directly usable for applications like 3D/4D reconstruction or depth-conditioned video editing/generation without post-hoc scale recovery. |
| Agent S2: A Compositional Generalist-Specialist Framework for Computer | ||
| Use Agents (Read more on arXiv or HuggingFace) | Vincent Tu, Kyle Wong, xw-eric, jc-y42, saa1605 | Agent S2 introduces a compositional generalist-specialist framework enhancing computer use agent capabilities via specialized modules. The primary objective is to address limitations in GUI grounding precision, long-horizon task planning, and reliance on single generalist models for diverse cognitive tasks. Methodologically, Agent S2 employs a Mixture-of-Grounding technique routing actions to specialized grounding experts and Proactive Hierarchical Planning for dynamic plan refinement based on evolving observations. Agent S2 achieved new state-of-the-art results, notably a 34.5% success rate on the OSWorld 50-step evaluation, a 32.7% relative improvement over the leading Claude Computer Use baseline. For AI practitioners, this demonstrates the effectiveness of composing generalist planning with specialized grounding modules to overcome bottlenecks in monolithic models for complex GUI automation tasks. |
| Z1: Efficient Test-time Scaling with Code (Read more on arXiv or HuggingFace) | Xiao-Ping Zhang, armanc, yilunzhao, yh1567, zjy2001 | Z1 proposes an efficient test-time compute scaling method for LLMs using code-related reasoning trajectories and a novel shifted thinking window. The research aims to reduce the excessive thinking token cost associated with test-time scaling in Large Reasoning Models (LRMs) while preserving performance. Key methodology involves training an LLM (Qwen2.5-Coder-7B-Instruct) on a curated dataset (Z1-Code-Reasoning-107K) containing both short and long code solution trajectories and employing a “Shifted Thinking Window” during inference that avoids fixed delimiters and caps reasoning tokens. The resulting model, Z1-7B, matches the performance of R1-Distill-Qwen-7B on three reasoning benchmarks while using only about 30% of its average thinking tokens, and notably generalizes to non-code tasks like GPQA Diamond (47.5%). For AI practitioners, this demonstrates a method to significantly improve the computational efficiency and reduce inference costs of LRMs for complex reasoning tasks by fine-tuning with varied-length code trajectories and adopting a flexible, adaptive thinking process during inference. |
| MixerMDM: Learnable Composition of Human Motion Diffusion Models (Read more on arXiv or HuggingFace) | José García-Rodríguez, Sergio Escalera, Cristina Palmero, Germs96, pabloruizponce | MixerMDM introduces a learnable technique for composing pre-trained text-conditioned human motion diffusion models. The main research objective is to dynamically combine motions from specialized single-person and interaction models to achieve fine-grained control over individual movements within complex interactions. The key methodology involves a lightweight Mixer module, trained adversarially against multiple discriminators (one per pre-trained model), to predict dynamic, context-dependent mixing weights at each denoising step, using the pre-trained models’ outputs as pseudo-ground truth. Primary results demonstrate superior performance over fixed-weight or scheduled methods, with MixerMDM achieving significantly better alignment and consistency, ranking first in 85.14% of user study comparisons based on motion alignment to textual descriptions. For AI practitioners, MixerMDM provides a modular framework to combine specialized, pre-trained diffusion models for generating nuanced, controllable human motion sequences without requiring retraining of the base models or explicit ground truth for the combined outputs. |
| Open-Qwen2VL: Compute-Efficient Pre-Training of Fully-Open Multimodal | ||
| LLMs on Academic Resources (Read more on arXiv or HuggingFace) | Heng Wang, Yu Tian, windwest, yanglj55, weizhiwang | Open-Qwen2VL details compute-efficient pre-training of a fully open-source 2B parameter Multimodal Large Language Model (MLLM) on academic-scale resources. The objective is to develop and openly release an efficient MLLM pre-training pipeline reproducible with limited compute, specifically using 8xA100-40G GPUs. Key methodologies include low-to-high dynamic image resolution (144 visual tokens in pre-training, 729 in SFT), multimodal sequence packing, and data filtering using both CLIP-based methods and MLLM-based techniques (MLM-Filter) on a 29M image-text pair dataset. The resulting instruction-tuned Open-Qwen2VL, pre-trained on 5B packed multimodal tokens (using 442 A100-40G GPU hours), outperforms the partially-open Qwen2-VL-2B on benchmarks such as MMBench (achieving 80.9), SEEDBench, MMStar, and MathVista, despite using only 0.36% of Qwen2-VL’s reported pre-training tokens. For AI practitioners, this work provides a fully open-sourced blueprint—including codebase, data filtering/packing scripts, curated pre-training data, and model checkpoints—demonstrating that efficient, high-performance MLLM pre-training is attainable without extensive industrial-scale resources, enabled by optimized data curation and training techniques. |
| Harnessing the Reasoning Economy: A Survey of Efficient Reasoning for | ||
| Large Language Models (Read more on arXiv or HuggingFace) | Sudanl, pangjh3, BeyondHsueh, Merlin-Hongru, Ray121381 | This survey systematically reviews strategies for achieving “Reasoning Economy” in Large Language Models (LLMs), balancing performance benefits against computational budgets. The primary objective is to analyze the causes of reasoning inefficiency (e.g., length bias, deceptive behaviors), understand different reasoning patterns, and survey potential solutions across post-training and test-time inference stages. It employs a comprehensive literature review, categorizing challenges stemming from post-training methods (like Superficial Alignment leading to length bias) and test-time usage (like unreasonable computation allocation) and corresponding optimization solutions (e.g., behavior regulation, usage improvement). Key findings identify specific inefficiencies like length bias (where RMs may prefer longer responses, e.g., 63.1% in RLCD) and overly cautious reasoning, while highlighting solutions such as long2short RL methods (e.g., SimPO reducing lengths by 30-40%) and adaptive computation allocation based on task complexity. For AI practitioners, the principal implication is the need to shift from static, one-size-fits-all inference approaches towards dynamic, adaptive strategies (e.g., adaptive budget allocation, algorithm selection) to optimize resource utilization and unlock LLMs’ full potential efficiently. |
| OmniMMI: A Comprehensive Multi-modal Interaction Benchmark in Streaming | ||
| Video Contexts (Read more on arXiv or HuggingFace) | Tong Wu, Bo Chen, Yueqian Wang, zlzheng, ColorfulAI | This paper introduces OmniMMI, a benchmark for evaluating MLLMs in streaming video interaction, and M4, a framework enhancing these capabilities. The primary objective is to evaluate and improve the real-world interactive performance of OmniLLMs in streaming video contexts, focusing on streaming understanding and proactive reasoning challenges underexplored by existing benchmarks. Methodology involved curating the OmniMMI dataset (1,121 videos, 2,290 questions across six subtasks including dynamic state grounding and proactive alerting) and developing the Multi-modal Multiplexing Modeling (M4) framework using multiplexing techniques and an attention-based inference method for efficient, proactive processing. Experimental results show existing MLLMs perform poorly on OmniMMI, particularly struggling with proactive tasks and multi-turn dependencies, while the proposed lightweight M4 model demonstrates significant improvement, achieving 68.5% accuracy on the Proactive Turn-taking task after audio adaptation (M4-a). For AI practitioners, this research underscores the inadequacy of current models for real-time interaction, provides OmniMMI as a necessary tool for evaluating streaming/proactive capabilities, and suggests the M4 architecture as a resource-efficient approach to develop models that can simultaneously perceive and generate responses in dynamic environments. |
| Command A: An Enterprise-Ready Large Language Model (Read more on arXiv or HuggingFace) | salthammer, yazeed7, jayalammar, ArashAhmadian, aakanksha | This report details Command A, a 111B parameter multilingual large language model optimized for enterprise RAG and agentic tasks, alongside the smaller Command R7B. The primary objective was to develop and evaluate Command A and R7B as efficient, high-performing LLMs tailored for real-world enterprise settings, focusing on multilingualism, Retrieval Augmented Generation (RAG), and tool use. Key methodologies include a decentralised post-training strategy combining supervised fine-tuning (SFT) and reinforcement learning (RL) across specialized expert models, followed by parameter merging (linear soup), and a polishing phase using algorithms like Self-improving Robust Preference Optimisation (SRPO). Command A achieves competitive results, scoring 80.0 on the MATH benchmark and 51.7 on Taubench, while maintaining efficiency by requiring only two A100/H100 GPUs for inference and delivering up to 156 tokens/sec. For AI practitioners, Command A offers an efficient foundation for enterprise applications needing strong RAG and agentic capabilities, while the reported decentralised training and merging approach presents a method for integrating diverse expert functionalities into a single model. |
| Recitation over Reasoning: How Cutting-Edge Language Models Can Fail on | ||
| Elementary School-Level Reasoning Problems? (Read more on arXiv or HuggingFace) | Xuesong Yao, xmerge123, ALEXoDu, yfxu, kaiyan289 | This paper demonstrates that cutting-edge LLMs often recite solutions rather than genuinely reason, even on elementary problems. The research objective was to determine if LLMs possess true reasoning ability or merely replicate patterns seen during training, particularly when faced with subtly altered conditions. A novel multi-modal benchmark, RoR-Bench, was created featuring pairs of original problems and variants with minor but crucial condition shifts. Empirical analysis revealed severe recitation behavior, with top models like OpenAI-o1 and DeepSeek-R1 experiencing performance drops exceeding 60% on modified elementary arithmetic and reasoning problems compared to their original counterparts. For AI practitioners, this highlights a critical need to re-evaluate LLM intelligence claims and emphasizes that current models may lack robustness, potentially failing unexpectedly when encountering slight deviations from learned patterns. |
| AdaMMS: Model Merging for Heterogeneous Multimodal Large Language Models | ||
| with Unsupervised Coefficient Optimization (Read more on arXiv or HuggingFace) | Yiru Wang, Jiabo Ye, Xiaochen Wang, Yiyang Du, carboncoo | AdaMMS introduces an unsupervised method for merging heterogeneous Multimodal Large Language Models (MLLMs) with differing architectures. The primary objective is to effectively combine capabilities from distinct MLLMs without requiring labeled data for optimizing the merging hyperparameters. The methodology involves parameter mapping to align weights, linear interpolation for merging, and an unsupervised search step that selects the optimal interpolation coefficient based on minimizing generation consistency differences across candidate merged models using a small unlabeled dataset. Experiments show AdaMMS outperforms supervised baselines; for example, merging LLaVA-OneVision-7B into Qwen2-VL-7B yielded a SUM score of 563.56, a +26.84 gain over the original models’ average. AI practitioners can leverage AdaMMS to fuse heterogeneous MLLMs efficiently, creating enhanced models without supervised data by using generation consistency as a proxy for task performance during optimization. |
| When To Solve, When To Verify: Compute-Optimal Problem Solving and | ||
| Generative Verification for LLM Reasoning (Read more on arXiv or HuggingFace) | anna-rohrbach, kaiweichang, adityagrover, arianhosseini, hbXNov | This research compares the compute-efficiency of Self-Consistency (SC) and Generative Reward Models (GenRM) for LLM reasoning, revealing SC’s superiority at lower budgets. The study investigates whether allocating a fixed inference budget towards generating more solutions (SC) or generating fewer solutions with multiple verifications (GenRM) yields better LLM reasoning performance, and how to optimally balance solutions and verifications for GenRM. A compute-matched analysis compared SC and GenRM across various models, tasks, and budgets, calculating FLOPs based on solution (S) and verification (V) generation; inference scaling laws were derived by fitting optimal solution (S_opt) and verification (V_opt) counts to compute budget C. Primary results show SC outperforms GenRM until high compute budgets are reached; for Llama-3.1-8B on MATH, GenRM required 8x the compute of SC to match its performance and 128x to achieve a 3.8% gain, while compute-optimal GenRM requires scaling solutions faster (S_opt ∝ C^0.57) than verifications (V_opt ∝ C^0.39). AI practitioners should prioritize SC for LLM reasoning under typical compute constraints; if using GenRM at high budgets, allocate compute preferentially towards increasing solution count over verification count per solution for optimal efficiency. |
| Scaling Language-Free Visual Representation Learning (Read more on arXiv or HuggingFace) | liuzhuang13, koustuvs, JiachenZhu, tsbpp, davidfan97 | This paper investigates scaling language-free visual self-supervised learning (SSL) on web-scale data, comparing its performance against Contrastive Language-Image Pretraining (CLIP) primarily on Visual Question Answering (VQA). The research aims to determine if visual SSL lags behind CLIP due to the absence of language supervision or disparities in training data. Key methodology involves training DINOv2 (SSL) and CLIP models (1B to 7B parameters) on the identical 2 billion sample MetaCLIP dataset and evaluating using the Cambrian-1 VQA suite and traditional vision benchmarks. Primary results indicate visual SSL scales better with model and data size than CLIP on VQA; specifically, a 7B parameter Web-DINO model trained on 8 billion examples outperforms a comparable CLIP model on average VQA performance across 16 tasks. The principal implication for AI practitioners is that appropriately scaled visual SSL can yield vision encoders competitive with language-supervised models for multimodal tasks like VQA, providing a strong vision-centric alternative without needing paired text data during pretraining. |
| Multi-Token Attention (Read more on arXiv or HuggingFace) | sainbar, spermwhale, Tianlu, Golovneva | This paper introduces Multi-Token Attention (MTA), enhancing LLM attention by conditioning weights on multiple query and key vectors simultaneously via convolution operations. The primary objective is to overcome the “single token attention” bottleneck, allowing models to locate relevant context using richer, multi-token criteria rather than single vector similarity. MTA modifies standard attention by applying convolutions across query, key, and head dimensions (termed key-query convolution and head mixing convolution), often coupled with group normalization. Experiments demonstrate MTA achieves lower perplexity on language modeling (11.09 avg PPL vs 11.25 for an 880M Transformer baseline) and notably improves performance on long-context tasks like Needle-in-a-Haystack and BabiLong compared to baselines. For AI practitioners, MTA offers a method to improve model performance in scenarios requiring identification of context based on multiple simultaneous conditions, particularly beneficial for long-context reasoning, by incorporating these convolutional modifications into the attention mechanism. |
| Efficient LLaMA-3.2-Vision by Trimming Cross-attended Visual Features (Read more on arXiv or HuggingFace) | Jaeyeon Kim, Donguk Lim, Seungmin Yang, Ki-Ung Song, Jewon Lee | This paper presents Trimmed Llama, a method for improving inference efficiency in cross-attention-based Large Vision-Language Models (LVLMs) by pruning visual features. The main objective is to mitigate the computational bottleneck caused by the large Key-Value (KV) cache size associated with image tokens in cross-attention layers. The key methodology involves exploiting the sparsity and inter-layer resemblance of cross-attention patterns, using head-wise attention scores from the first cross-attention layer to selectively prune redundant visual features for subsequent layers. Primary results show that Trimmed Llama can reduce visual feature usage by up to 50% (e.g., Kratio=0.15 retaining ~41.6% features for the 11B model) while maintaining performance parity with baseline Llama-3.2-Vision models on benchmarks like MME and LLaVA-Bench, alongside reduced inference latency (e.g., 14.2% reduction for batch size 16). For AI practitioners, this provides a training-free technique to decrease inference latency and memory consumption for cross-attention LVLMs with minimal impact on task performance. |
| Inference-Time Scaling for Complex Tasks: Where We Stand and What Lies | ||
| Ahead (Read more on arXiv or HuggingFace) | Neel Joshi, Shivam Garg, Lingjiao Chen, Jingya Chen, Vidhisha Balachandran | This extensive empirical study evaluates the benefits and limitations of inference-time scaling methods across diverse complex tasks for large language models (LLMs). The main objective was to investigate how scaling performance, including accuracy and token usage tradeoffs, varies across nine state-of-the-art conventional and reasoning-tuned models on eight challenging benchmarks (e.g., math, NP-hard problems, planning, spatial reasoning). Key methodologies included evaluating models using standard Chain-of-Thought (CoT), parallel scaling (sampling N generations with aggregators like best-of-N), and sequential scaling (iterative refinement with self-critique), approximating performance bounds. Primary results show inference-time scaling benefits vary significantly by task and diminish with complexity; notably, increased token consumption does not reliably yield higher accuracy across models (e.g., on AIME 25, DeepSeek R1 used >5x more tokens than Claude 3.7 Sonnet for <3% accuracy difference). The principal implication for AI practitioners is that leveraging inference-time compute requires careful task-specific consideration and highlights the critical need for developing robust, efficient verifiers and adaptive scaling strategies, as current approaches show inconsistent gains and cost nondeterminism. |
| Discovering Knowledge Deficiencies of Language Models on Massive | ||
| Knowledge Base (Read more on arXiv or HuggingFace) | Ryotaro Shimizu, Jieyu Zhang, Xuwei Ding, MaksimSTW, linxinso | This paper introduces Stochastic Error Ascent (SEA), a scalable framework for efficiently discovering factual knowledge deficiencies in closed-weight LLMs against massive knowledge bases under budget constraints. The primary objective is to develop a scalable and budget-constrained method for automatically uncovering knowledge deficiencies (errors) in closed-weight LLMs by evaluating them against large knowledge bases. The core methodology, SEA, uses stochastic optimization to iteratively retrieve knowledge base paragraphs semantically similar to prior LLM failures, employing hierarchical retrieval and a relation DAG to guide the search efficiently. Empirically, SEA uncovered 40.7× more knowledge errors than the Automated Capability Discovery baseline and 26.7% more than AutoBencher, while significantly reducing the cost-per-error. For AI practitioners, SEA provides a cost-effective method to pinpoint specific factual weaknesses in LLMs, enabling targeted improvements through data curation or fine-tuning to enhance model reliability. |
| m1: Unleash the Potential of Test-Time Scaling for Medical Reasoning | ||
| with Large Language Models (Read more on arXiv or HuggingFace) | Yuyin Zhou, Xianfeng Tang, Hui Liu, Juncheng Wu, Xiaoke Huang | This paper introduces m1, a method applying test-time scaling to enhance the medical reasoning capabilities of Large Language Models (LLMs). The primary objective was to investigate the effectiveness of test-time scaling for medical QA, contrasting it with mathematical reasoning tasks. The methodology involved curating medical QA datasets (m1K, m23K), fine-tuning Qwen2.5 models (7B, 32B) on these datasets using Supervised Fine-Tuning (SFT), and controlling the “thinking” token budget during inference. Results show that increasing the thinking budget improves accuracy (e.g., m1-7B-23K achieved 60.32% average accuracy), but plateaus around 4K tokens; budget forcing offered limited benefits, and performance gains were ultimately constrained by the model’s inherent medical knowledge. For AI practitioners, this implies that while test-time scaling enhances medical reasoning, it is insufficient alone; complementing it with improved knowledge grounding via high-quality data curation and larger model capacity is essential for further performance gains, especially on complex medical tasks. |
| Chapter-Llama: Efficient Chaptering in Hour-Long Videos with LLMs (Read more on arXiv or HuggingFace) | Gül Varol, Cordelia Schmid, Antoine Yang, Lucas Ventura | Chapter-Llama introduces an efficient LLM-based framework for automatic video chaptering in hour-long videos. The primary objective is to partition long videos into semantic chapters and generate corresponding titles automatically. The methodology involves finetuning a large language model (Llama-3.1-8B) using text inputs derived from ASR transcripts and descriptive captions of sparsely sampled keyframes, selected via a novel speech-guided strategy. Results show substantial improvement over the state-of-the-art on VidChapters-7M, achieving a 45.3 F1 score compared to the previous best of 26.7. For AI practitioners, this work presents a scalable, text-only approach leveraging LLMs and efficient frame sampling for indexing and structuring long-form video content without direct video feature processing. |
| Towards Trustworthy GUI Agents: A Survey (Read more on arXiv or HuggingFace) | Ninghao Liu, Wenhu Chen, Wenlin Yao, Wenhao Yu, Yucheng Shi | This survey reviews the critical dimensions of trustworthiness for GUI agents interacting with digital interfaces via foundation models. The paper’s objective is to systematically examine security vulnerabilities, reliability, explainability, ethical considerations, and evaluation methodologies pertinent to GUI agent trustworthiness. It employs a literature survey methodology, categorizing research into five key trustworthiness areas and summarizing existing attacks (e.g., WebPI, AEIA-MN), defenses (e.g., GuardAgent, CLEAR), and evaluation frameworks (e.g., ST-WebAgentBench, Agent-SafetyBench). Key findings identify significant security vulnerabilities, such as environmental injection attacks achieving up to 93% success rates (AEIA-MN), alongside challenges in reliability (hallucination) and privacy, while noting that current research often overlooks these aspects for functional performance. For AI practitioners, this necessitates a shift from solely optimizing task completion towards implementing holistic, multi-layered defenses, robust evaluation benchmarks incorporating safety metrics, and user-centric transparency mechanisms to ensure secure and responsible GUI agent deployment. |
| DiET-GS: Diffusion Prior and Event Stream-Assisted Motion Deblurring 3D | ||
| Gaussian Splatting (Read more on arXiv or HuggingFace) | Gim Hee Lee, onandon | DiET-GS introduces a novel framework for motion deblurring in 3D Gaussian Splatting using event streams and diffusion priors. The research addresses the problem of reconstructing sharp 3D representations from blurry multi-view images. It leverages an event double integral prior and a pretrained diffusion model within a two-stage training strategy. DiET-GS outperforms existing methods, achieving a MUSIQ score of 51.71 on real-world datasets, but the DiET-GS++ has a longer training time compared to E2NeRF and Ev-DeblurNeRF. This provides AI practitioners with an approach for improving novel view synthesis from motion-blurred images. |
| ManipTrans: Efficient Dexterous Bimanual Manipulation Transfer via | ||
| Residual Learning (Read more on arXiv or HuggingFace) | Siyuan Huang, Yuyang Li, Tengyu Liu, Puhao Li, Kailin Li | i) The paper introduces MANIPTRANS, a two-stage method for efficient transfer of human bimanual skills to dexterous robotic hands in simulation. ii) The primary research objective is to facilitate the transfer of human hand manipulation skills, especially bimanual actions, to dexterous robotic hands in simulation enabling accurate tracking of reference motions. iii) The method uses a pre-trained generalist trajectory imitator for hand motion mimicking followed by a fine-tuned residual module under interaction constraints. iv) MANIPTRANS achieves superior success rates (58.1/39.5 for single/bimanual respectively) compared to SOTA methods and constructs DEXMANIPNET, a dataset of 3.3K episodes of robotic manipulation, improving motion fidelity. v) The development of MANIPTRANS offers AI practitioners an efficient and generalizable framework for creating large-scale, high-quality datasets of dexterous manipulation enabling more effective training of robot control policies. |
| MB-ORES: A Multi-Branch Object Reasoner for Visual Grounding in Remote | ||
| Sensing (Read more on arXiv or HuggingFace) | Mustapha lebbah, Hanane Azzag, rdkarim | MB-ORES introduces a unified framework for object detection (OD) and visual grounding (VG) in remote sensing (RS) imagery. The paper aims to improve visual grounding in RS images by fine-tuning an open-set object detector with referring expression data and then processing outputs via a graph-based representation and a multi-branch, task-aware architecture. The methodology incorporates a multi-branch network for feature integration, an object reasoning network for proposal ranking, and a soft selection mechanism for object localization. Experiments on DIOR-RSVG show MB-ORES outperforms existing methods, increasing performance by +3.38% to +14.89% across threshold levels, while on the OPT-RSVG dataset, meanIoU increased by +6.98%. This implies a unified OD/VG approach, applicable by AI practitioners in the remote sensing domain, can achieve state-of-the-art performance while retaining OD capabilities, offering a more versatile tool. |
Papers for 2025-04-01
| Title | Authors | Summary |
|---|---|---|
| TextCrafter: Accurately Rendering Multiple Texts in Complex Visual | ||
| Scenes (Read more on arXiv or HuggingFace) | Nikai Du, yingtai, jzzzzk, Chenzzzzzz, zhen-nan | TextCrafter is a training-free framework designed to accurately render multiple texts across different regions in complex visual scenes generated by diffusion models. The primary objective is to address limitations like text distortion, omission, and blurriness encountered in Complex Visual Text Generation (CVTG). The methodology involves a progressive three-stage approach: Instance Fusion to align text content with its visual carrier, Region Insulation to separate text prompts and initialize layout using pre-trained model priors, and Text Focus to enhance text token attention for improved fidelity. Experiments on the newly proposed CVTG-2K benchmark show TextCrafter achieves a 0.7370 average Word Accuracy, significantly improving OCR accuracy by over 45% compared to the baseline FLUX model it builds upon. For AI practitioners, this provides an effective method to enhance multi-text rendering capabilities in text-to-image systems without requiring additional model training or fine-tuning, improving performance on complex scene generation with detailed textual elements. |
| MoCha: Towards Movie-Grade Talking Character Synthesis (Read more on arXiv or HuggingFace) | Luczzz, daixl1992, FelixXu, haoyum1997, lim142857 | MoCha introduces an end-to-end Diffusion Transformer model for generating movie-grade talking characters directly from speech and text inputs without auxiliary conditions. The primary objective is to create realistic characters with synchronized lip movements, natural facial expressions, coherent full-body actions, and support for multi-character, turn-based conversations, addressing limitations in prior work focused on talking heads or general video synthesis lacking speech control. Key methodologies include a speech-video window attention mechanism for improved lip-sync, a joint training strategy leveraging both speech-labeled and text-only video data for better generalization, and structured character-tagged prompts for multi-character dialogue. MoCha significantly outperforms baselines on the MoCha-Bench benchmark, achieving superior human evaluation scores across all five axes (e.g., +1.40 in Lip-Sync Quality over the next best) and better quantitative lip-sync metrics (Sync-C: 6.037 vs 4.866). For AI practitioners, MoCha offers a direct speech+text-to-video synthesis approach for controllable character animation, enabling richer narrative generation for applications like automated filmmaking and virtual avatars without reliance on intermediate representations like keypoints or explicit pose control. |
| What, How, Where, and How Well? A Survey on Test-Time Scaling in Large | ||
| Language Models (Read more on arXiv or HuggingFace) | nancy-zwx, demolei, RubinSun, silentspring2, DonJoey | This survey introduces a unified four-dimensional framework (what, how, where, how well) to systematically organize and analyze research on Test-Time Scaling (TTS) in Large Language Models. Its objective is to address the lack of a comprehensive overview by categorizing TTS methods, applications, and evaluation metrics, identifying trends, and outlining future directions. The paper proposes a multi-axis taxonomy and conducts an extensive literature review, decomposing techniques like parallel, sequential, hybrid, and internal scaling, alongside tuning-based (SFT, RL) and inference-based (stimulation, verification, search, aggregation) implementation strategies. The review confirms TTS significantly enhances LLM performance across various tasks, observing scaling-law-like improvements with increased compute, and highlights specific techniques like internal scaling via RL (e.g., DeepSeek-R1) or search methods yielding efficiency gains (e.g., ETS achieving 1.8x KV cache reduction). AI practitioners can utilize the taxonomy and guidelines (Section 7) to select, combine, and evaluate complementary TTS strategies (e.g., Self-Consistency, MCTS, STaR, internal scaling) for balancing performance, cost, and task-specific requirements in LLM deployment. |
| Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement | ||
| Learning on the Base Model (Read more on arXiv or HuggingFace) | Xiangyu Zhang, Qi Han, djiang, YinminZhang, reign12 | Open-Reasoner-Zero (ORZ) introduces an open-source, minimalist approach for large-scale reinforcement learning (RL) focused on enhancing reasoning in base language models. The primary objective was to determine if vanilla PPO with simple rule-based rewards and no KL regularization could scale LLM reasoning performance and response length effectively. The methodology involved applying PPO with GAE (λ=1, γ=1) and a binary correctness reward directly to Qwen2.5 base models (0.5B to 32B) using a curated reasoning dataset. Results showed that ORZ-32B surpassed the DeepSeek-R1-Zero-Qwen-32B model on benchmarks like MATH500 (92.2 vs 91.6) and GPQA Diamond (55.5 vs 55.0) using only 1/10th the training steps, demonstrating stable scaling without KL constraints. The principal implication for AI practitioners is that complex RLHF setups with KL regularization may not be necessary for scaling reasoning; a simpler, resource-efficient PPO configuration can yield strong results directly on base models. |
| RIG: Synergizing Reasoning and Imagination in End-to-End Generalist | ||
| Policy (Read more on arXiv or HuggingFace) | Haian Huang, Zhonghan Zhao, GaoangWang, pppppM, ZwwWayne | RIG introduces an end-to-end generalist policy that synergizes reasoning and imagination for embodied agents. The research aims to improve sample efficiency and generalization by integrating reasoning and imagination into a single Transformer model. The methodology involves a progressive data collection strategy to generate reasoning-enriched and dream-review trajectories coupled with language model-based training. Experimental results in Minecraft demonstrate that RIG achieves state-of-the-art performance, showing more than 17× sample efficiency improvements compared to prior works, requiring only 111 hours of video data; also shown is the improvement of robustness and interoperability of generalist policy. RIG provides AI practitioners with an architecture that enhances the performance and scalability of embodied agents by combining reasoning and imagination, offering a pathway towards more efficient and robust policy learning in complex environments. |
| Effectively Controlling Reasoning Models through Thinking Intervention (Read more on arXiv or HuggingFace) | Prateek Mittal, Jiachen T. Wang, cxiang, tongwu2020 | Reasoning models can be controlled through Thinking Intervention, a paradigm for guiding internal reasoning processes via strategic token insertion or revision. The research question explores fine-grained control over model behavior by guiding internal reasoning processes of LLMs. The methodology involves comprehensive evaluations across instruction following, instruction hierarchy, and safety alignment tasks. Results show that Thinking Intervention achieves up to a 6.7% accuracy gain in instruction-following, a 15.4% improvement in reasoning about instruction hierarchies, and a 40.0% increase in refusal rates for unsafe prompts using open-source DeepSeek R1 models. Thinking Intervention enables fine-grained control over reasoning trajectories, aligning model behavior with specific task objectives, allowing for more reliable and aligned AI systems. |
| Query and Conquer: Execution-Guided SQL Generation (Read more on arXiv or HuggingFace) | sfc-mwydmuch, Borchmann | i) The paper introduces an execution-guided self-consistency approach for text-to-SQL generation. ii) The research aims to improve accuracy in complex text-to-SQL tasks by leveraging execution results for candidate query selection. iii) The methodology utilizes exact and approximate execution-based similarity metrics within the Minimum Bayes Risk (MBR) decoding framework. iv) The Qwen 2.5 Coder 7B model employing this method achieves nearly a 10% accuracy improvement, matching the performance of O1 while reducing inference cost by 30 times. v) AI practitioners can leverage execution-guided self-consistency to improve the performance of smaller, cost-effective models in text-to-SQL tasks. |
| SketchVideo: Sketch-based Video Generation and Editing (Read more on arXiv or HuggingFace) | dizhang, WeicaiYe, Xintao, fuhongbo, Okrin | SketchVideo presents a unified framework for generating and editing videos conditioned on sparse keyframe sketches and text prompts. The research objective is to achieve precise spatial layout and motion control in video synthesis and editing using temporally sparse user-drawn sketches. It utilizes a skip-residual sketch control structure for DiT models, an inter-frame attention mechanism for propagating sparse conditions, and a video insertion module with latent fusion for editing. Experiments show superior performance, with SketchVideo achieving the lowest LPIPS (27.56) and highest CLIP score (98.31) in generation benchmarks compared to methods like SparseCtrl. AI practitioners can implement this technique to provide users with fine-grained geometric and motion control in video creation/editing tools, enhancing controllability beyond text-only approaches. |
| TeleAntiFraud-28k: A Audio-Text Slow-Thinking Dataset for Telecom Fraud | ||
| Detection (Read more on arXiv or HuggingFace) | Kai Wu, Jingpeng Wang, HuangMinhua, WDong, JimmyMa99 | This paper introduces TeleAntiFraud-28k, an open-source audio-text dataset with slow-thinking annotations for telecom fraud detection. The research aims to overcome the lack of suitable multimodal training data by integrating audio signals with reasoning-oriented textual analysis for automated fraud identification. Methodology involves dataset construction via three strategies: processing real anonymized calls with ASR/TTS, semantic expansion using LLM self-instruction, and multi-agent adversarial simulation, followed by LLM-based annotation capturing reasoning steps. Key results include the creation of 28,511 audio-text pairs and the demonstration that fine-tuning Qwen2Audio on this dataset significantly boosted fraud detection F1 score to 84.78% (average F1 across tasks 83.00%) on the established TeleAntiFraud-Bench. For AI practitioners, this work provides a crucial dataset and benchmark for developing and evaluating multimodal, reasoning-capable audio language models specifically for the challenging task of telecom fraud detection. |
| Efficient Inference for Large Reasoning Models: A Survey (Read more on arXiv or HuggingFace) | jiaheng233, Bibaolong, HongyuChen, HongchengGao, yueliu1999 | This survey reviews and categorizes methods for improving the inference token efficiency of Large Reasoning Models (LRMs) while maintaining reasoning quality. The primary objective is to analyze techniques mitigating high token consumption, memory overhead, and inference time inherent in LRM’s deliberative reasoning processes. It introduces a taxonomy classifying approaches into explicit compact Chain-of-Thought (CoT), which reduces tokens while keeping explicit structure, and implicit latent CoT, which encodes reasoning in hidden representations, alongside empirical analysis. Key findings categorize methods based on whether they maintain explicit reasoning steps or encode them latently; for instance, on GSM8K, explicit methods like TokenSkip (ratio=0.5) achieve 86.70% accuracy using 113.05 tokens with LLaMA-3.1-8B-Instruct, while implicit methods like SoftCoT reach 85.81% accuracy with Qwen2.5-7B-Instruct, though its specific token cost comparison is not fully detailed in the provided table excerpt. AI practitioners gain insights into the performance/efficiency trade-offs of LRM optimization techniques, informing the selection of methods (e.g., explicit CoT for interpretability, implicit CoT for token reduction) for developing cost-effective reasoning applications. |
| Classical Planning with LLM-Generated Heuristics: Challenging the State | ||
| of the Art with Python Code (Read more on arXiv or HuggingFace) | jendrikseipp, andregrahl, abcorrea | This paper demonstrates using Large Language Models (LLMs) to automatically generate domain-dependent heuristic functions as Python code for classical planning tasks. The objective was to determine if LLM-generated heuristics could outperform traditional domain-independent heuristics and compete with state-of-the-art learned heuristics. The methodology involved prompting an LLM (e.g., DeepSeek R1) multiple times for a given planning domain, evaluating the resulting pool of Python heuristic functions on training tasks using Greedy Best-First Search (GBFS), and selecting the best-performing one. Results show the selected LLM-generated heuristics significantly outperformed the widely used hFF heuristic (solving 373 vs. 243 test tasks in Pyperplan) and were competitive with state-of-the-art learned heuristics implemented in optimized C++, even when run in an unoptimized Python planner. For AI practitioners, this implies LLMs can automate the creation of highly effective, domain-specific heuristics for planning, potentially accelerating development and improving performance without requiring deep heuristic engineering expertise or specialized learning pipelines. |
| Expanding RL with Verifiable Rewards Across Diverse Domains (Read more on arXiv or HuggingFace) | zptu, haitaominlp, douvleplus, freesunshine0316, yudian | This paper extends Reinforcement Learning with Verifiable Rewards (RLVR) to diverse domains like medicine and economics, using a distilled generative reward model. The main objective is to investigate RLVR’s applicability beyond well-structured tasks and evaluate if a single, trained reward model can effectively provide cross-domain reward signals for free-form answers without domain-specific annotations. The methodology involves training a 7B parameter reward model using judgments distilled from a larger teacher LLM (Qwen2.5-72B-Instruct) and incorporating model-based soft scoring for RL fine-tuning (using REINFORCE, RLOO, etc.) of a base 7B policy model. Using RLOO with the distilled 7B reward model (RM-7B) and soft scoring yielded a 30.0% average accuracy on multi-subject tasks, outperforming the baseline rule-based reward (16.6%) and matching the performance using the much larger Qwen2.5-72B model directly for rewards (30.6%). For AI practitioners, this suggests that a smaller, distilled generative reward model can effectively guide RL fine-tuning across diverse domains with unstructured answers, offering a computationally efficient alternative to large teacher models or domain-specific reward engineering, enhancing RLVR’s scalability and robustness. |
| Progressive Rendering Distillation: Adapting Stable Diffusion for | ||
| Instant Text-to-Mesh Generation without 3D Data (Read more on arXiv or HuggingFace) | Zhen Lei, Xiangyu Zhu, Rongyuan Wu, DarklordLeto, ZhiyuanthePony | This paper presents Progressive Rendering Distillation (PRD) to adapt Stable Diffusion (SD) for instant text-to-mesh generation without 3D ground-truth data. The objective is to overcome the 3D data scarcity problem by distilling knowledge from multi-view 2D diffusion models into an SD-based native 3D generator. PRD progressively denoises latent noise over a few steps, decoding intermediate results into Triplanes and using score distillation with SD, MVDream, and RichDreamer as teachers; Parameter-Efficient Triplane Adaptation (PETA) adds only 2.5% trainable parameters via LoRA. The resulting model, TriplaneTurbo, generates high-quality textured meshes in 1.2 seconds, achieving a CLIP Score of 68.2, outperforming prior methods in speed and quality without 3D training data. For AI practitioners, this work demonstrates an effective, data-efficient method to repurpose large 2D diffusion models for rapid 3D content creation, significantly reducing reliance on 3D datasets and accelerating generation pipelines. |
| TokenHSI: Unified Synthesis of Physical Human-Scene Interactions through | ||
| Task Tokenization (Read more on arXiv or HuggingFace) | BoDai, WenjiaWang, frankzydou, Zeshi209, lianganimation | TokenHSI introduces a unified transformer-based policy using task tokenization to synthesize diverse, physically plausible human-scene interactions. The primary objective is to develop a single, versatile physics-based controller capable of learning multiple foundational HSI skills and efficiently adapting them to novel, complex scenarios like skill composition or environment variations. Key methodology involves separate tokenizers for shared humanoid proprioception and distinct task states, combined within a transformer encoder via a masking mechanism, enabling multi-task learning and flexible adaptation by adding new tokenizers and lightweight adapter layers. Primary results demonstrate successful unification of diverse skills (following, sitting, climbing, carrying) and superior adaptation compared to baselines, achieving a 99.2% success rate on the challenging Climb + Carry skill composition task. For AI practitioners, this provides an efficient and extensible framework for building versatile physics-based agents capable of complex interactions, reducing the need for separate controllers per skill and enabling rapid adaptation to new tasks with minimal parameter fine-tuning. |
| KOFFVQA: An Objectively Evaluated Free-form VQA Benchmark for Large | ||
| Vision-Language Models in the Korean Language (Read more on arXiv or HuggingFace) | lastdefiance20, yoonshik1205 | This paper introduces KOFFVQA, a novel Korean free-form Visual Question Answering benchmark designed for objective evaluation of Large Vision-Language Models (VLMs). The main objective is to overcome the limitations of existing VLM evaluation methods, namely the subjectivity of judge models and the lack of Korean-specific benchmarks, by providing a reliable framework for assessing open-ended VLM responses. The methodology involves a benchmark dataset of 275 curated image-question pairs, each accompanied by detailed, objective grading criteria, which guide an LLM judge (specifically Gemma 2 9B in testing) to score VLM responses on a scale of 0-10 across 10 performance categories. Results from evaluating 47 VLMs show this criteria-guided LLM-judge approach achieves significantly higher evaluation consistency (e.g., mean score standard deviation of 0.398 for Gemma 2 9B vs. 0.584 for ground-truth comparison) and accuracy (89.3% correct grading for Gemma 2 9B) compared to methods using ground-truth comparisons or VLM-as-a-judge, which was found prone to visual hallucinations. For AI practitioners, this work provides a robust benchmark and methodology for objectively evaluating the free-form reasoning and Korean language capabilities of VLMs, highlighting that explicit, objective criteria significantly improve judge model reliability over subjective or ground-truth-comparative approaches. |
| UPME: An Unsupervised Peer Review Framework for Multimodal Large | ||
| Language Model Evaluation (Read more on arXiv or HuggingFace) | Zheyuan Liu, Yibing, yuehuang, MunanNing, 77Hui | This paper introduces UPME, an unsupervised peer review framework for evaluating Multimodal Large Language Models (MLLMs) using only image data, eliminating the need for human QA annotations. The research objective is to develop an objective MLLM evaluation method that avoids the high cost of human annotation and mitigates biases found in MLLM-as-a-judge systems. UPME utilizes a peer review process where MLLMs generate questions for images and evaluate peer answers using a vision-language scoring system (assessing correctness, visual understanding/reasoning, image-text correlation) refined by dynamic weight optimization based on evaluation consistency. Experimental results show UPME achieves high alignment with human judgments, attaining a Pearson correlation of 0.944 on the MMstar dataset, while significantly reducing verbosity and self-preference biases compared to baseline peer review methods. For AI practitioners, UPME offers a scalable, automated, and less biased approach to evaluate MLLM performance, particularly for visual capabilities, without requiring extensive human-annotated datasets. |
| Easi3R: Estimating Disentangled Motion from DUSt3R Without Training (Read more on arXiv or HuggingFace) | Anpei Chen, Andreas Geiger, Yuliang Xiu, faneggg, rover-xingyu | Easi3R introduces a training-free method to adapt the static 3D reconstruction model DUSt3R for dynamic 4D reconstruction by disentangling motion from its attention maps. The main objective is to extract and separate camera and object motion information implicitly encoded within DUSt3R’s attention layers without requiring retraining or fine-tuning on dynamic datasets. The key methodology involves aggregating spatial and temporal cross-attention maps to derive dynamic object segmentations, which are then used for attention re-weighting during a second inference pass and optional segmentation-aware global alignment. Easi3R significantly outperforms previous methods trained or fine-tuned on dynamic data across camera pose estimation, dynamic object segmentation (e.g., achieving 53.0 JM on DAVIS-all without SAM2 using the MonST3R backbone), and 4D point map reconstruction. For AI practitioners, this implies that task adaptation of large pre-trained models can sometimes be achieved through careful analysis and manipulation of internal representations like attention maps during inference, reducing the need for costly retraining on specialized dynamic datasets. |
| MeshCraft: Exploring Efficient and Controllable Mesh Generation with | ||
| Flow-based DiTs (Read more on arXiv or HuggingFace) | Xiaoshui Huang, Zexiang Liu, Di Huang, Junyi Chen, Xianglong He | MeshCraft introduces a novel framework for efficient and controllable 3D mesh generation using flow-based diffusion transformers. The paper addresses the challenge of slow generation speeds and uncontrollable face numbers in existing mesh generation techniques. MeshCraft employs a transformer-based VAE to encode and decode meshes in a continuous latent space and a flow-based diffusion transformer conditioned on the number of faces. Experiments demonstrate MeshCraft achieves a 35x speed increase compared to MeshGPT while maintaining state-of-the-art mesh quality. The framework’s efficient and controllable mesh generation capability enables AI practitioners to rapidly generate high-quality 3D assets with user-defined specifications. |
| Bridging Evolutionary Multiobjective Optimization and GPU Acceleration | ||
| via Tensorization (Read more on arXiv or HuggingFace) | Ran Cheng, Kebin Sun, Naiwei Yu, Hao Li, ZhenyuLiang | i) This paper introduces a tensorization methodology to accelerate evolutionary multiobjective optimization (EMO) algorithms on GPUs. ii) The research aims to bridge the gap between EMO algorithms and GPU computing by transforming EMO data structures and operations into tensor representations. iii) The methodology involves tensorizing data structures and operations within EMO algorithms and applying this tensorization to NSGA-III, MOEA/D, and HypE. iv) Experiments show that tensorized EMO algorithms achieve speedups of up to 1113× compared to CPU-based counterparts on a multi-objective robot control benchmark. v) Tensorization enables AI practitioners to effectively utilize GPUs to significantly improve the computational efficiency and scalability of EMO algorithms for complex optimization problems. |
| Decoupling Angles and Strength in Low-rank Adaptation (Read more on arXiv or HuggingFace) | Zeynep Akata, Leander Girrbach, Massimo Bini | i) The paper introduces Decoupled Low-rank Adaptation (DeLoRA), a novel parameter-efficient finetuning method. ii) The research aims to enhance the robustness of low-rank adaptation methods like LoRA by decoupling angular learning from adaptation strength. iii) DeLoRA normalizes and scales learnable low-rank matrices, bounding the transformation distance through normalization. iv) Experiments on subject-driven image generation demonstrate that DeLoRA achieves a DINO score of 0.693 and CLIP-I score of 0.820 matching or surpassing LoRA’s performance. v) AI practitioners can leverage DeLoRA to achieve more robust performance in adapting large-scale models to downstream tasks, particularly where hyperparameter tuning is challenging or extended training is required. |
| Entropy-Based Adaptive Weighting for Self-Training (Read more on arXiv or HuggingFace) | Wei Wang, Mingyu Derek Ma, Yihe Deng, Xiaoxuan Wang | i) This paper introduces Entropy-Based Adaptive Weighting for Self-Training (EAST), a novel method to improve mathematical reasoning in large language models (LLMs). ii) The research aims to address the challenge of effectively using self-generated data in self-training by prioritizing uncertain data points. iii) EAST assigns adaptive weights based on the entropy of the model’s sample distribution, using a mapping function with a tunable sharpness parameter integrated with SFT, DPO, and KTO loss functions. iv) On the MATH benchmark, EAST achieves approximately a 1% gain over the backbone model, and on GSM8K, it attains a further 1-2% performance boost compared to the vanilla method using the Llama-3.2-1B and Llama-3.1-8B architectures. v) EAST provides AI practitioners with an improved self-training strategy by reweighting training data to leverage uncertainty information, potentially increasing reasoning capabilities and reducing overfitting on overconfident data. |
Papers for 2025-03-31
| Title | Authors | Summary |
|---|---|---|
| AdaptiVocab: Enhancing LLM Efficiency in Focused Domains through | ||
| Lightweight Vocabulary Adaptation (Read more on arXiv or HuggingFace) | Roi Reichart, ehoffer, eyalbd, nitay, itaynakash | AdaptiVocab enhances Large Language Model (LLM) efficiency in focused domains through lightweight vocabulary adaptation. Its objective is to reduce latency and computational costs in domain-specific, low-resource settings by optimizing the LLM’s vocabulary. The methodology involves replacing low-frequency general tokens with high-frequency domain-specific n-gram tokens based on a token-saving score, initializing new embeddings using exponential weighting, and performing lightweight fine-tuning on embedding and adjacent layers. Results across two 7B LLMs and three niche domains show over a 25% reduction in token usage for both input processing and output generation, without compromising generation quality or end-task performance. For AI practitioners, this offers a resource-efficient technique to improve the inference speed and reduce the operational cost of LLMs deployed for specialized applications, particularly in settings with limited data or computational budgets. |
| Exploring Data Scaling Trends and Effects in Reinforcement Learning from | ||
| Human Feedback (Read more on arXiv or HuggingFace) | amusingchao, qingping95, zhengwu07, glnbyte, Swtheking | This paper investigates data scaling challenges in RLHF, proposing data construction and training strategies to mitigate reward hacking and improve response diversity. The primary objective is to identify and overcome data-driven bottlenecks hindering RLHF performance scaling. Methodology involves a hybrid reward system combining Reasoning Task Verifiers (RTV) and Generative Reward Models (GenRM) with ground truth, alongside a Pre-PPO prompt selection method prioritizing challenging prompts and early-stage math/coding task training. Results demonstrate the proposed ‘Data Scale’ approach significantly outperforms baseline PPO, achieving a +1.4 overall score improvement on the challenging TestSet V2.0 for the large model, and RTV exhibited the strongest resistance to reward hacking. For AI practitioners, this work highlights that strategic data curation and robust reward mechanisms (like RTV/GenRM-GT) are critical for enhancing RLHF performance and scalability, offering practical methods to address reward hacking and diversity issues. |
| Think Before Recommend: Unleashing the Latent Reasoning Power for | ||
| Sequential Recommendation (Read more on arXiv or HuggingFace) | Xu Chen, Jun Xu, TengShi, KID-22, TangJiakai5704 | This paper introduces ReaRec, an inference-time framework that enhances sequential recommendation (SeqRec) models by incorporating multi-step implicit reasoning. The objective is to overcome the limitations of traditional direct forward inference in capturing complex user preference dynamics, especially for long-tail items. ReaRec achieves this by autoregressively feeding the last hidden state back into the SeqRec model, using specialized reasoning position embeddings, and employs two learning strategies: Ensemble Reasoning Learning (ERL) and Progressive Reasoning Learning (PRL). Empirical results show ReaRec improves performance by an average of 7.49% across metrics on five datasets, and notably, post-hoc analysis reveals it can raise the performance ceiling of backbone SeqRec models by approximately 30-50%. For AI practitioners, ReaRec presents a model-agnostic method to potentially improve existing SeqRec systems by strategically increasing computation during inference rather than solely relying on model parameter scaling. |
| A Survey of Efficient Reasoning for Large Reasoning Models: Language, | ||
| Multimodality, and Beyond (Read more on arXiv or HuggingFace) | Elliott, weigao266, Warrieryes, yaful, Xiaoye08 | This survey reviews methods to enhance the computational efficiency of reasoning processes in Large Reasoning Models (LRMs) throughout their development lifecycle. The paper’s objective is to categorize patterns of reasoning inefficiency, such as excessive token generation and overthinking simple problems, and provide a comprehensive overview of techniques aiming to improve reasoning efficiency. Methodologically, it defines reasoning efficiency η(M) = E[Q(M,D) / C(M,D)] and systematically surveys literature, classifying techniques across pretraining, SFT, RL, and inference stages, including length budgeting, model switching, reasoning chain compression, and architectural modifications. Primary results highlight significant inefficiencies, exemplified by an LRM (QwQ-32B) using nearly 40 times more tokens than an instruction-tuned model for a simple math problem, and detail various strategies to reduce computational cost, often involving a trade-off with performance accuracy. The principal implication for AI practitioners is the catalog of techniques (e.g., length budgeting, SFT compression, latent-space reasoning) that can be applied to mitigate excessive token usage and latency, enabling more cost-effective and resource-aware deployment of LRMs, especially in applications like agent-based systems. |
| ORIGEN: Zero-Shot 3D Orientation Grounding in Text-to-Image Generation (Read more on arXiv or HuggingFace) | Jihyun Lee, Minhyuk, 32V, daehyeonchoi, myhong | ORIGEN introduces the first zero-shot method for grounding 3D orientation for multiple objects in text-to-image generation. The main research objective is to enable controllable 3D orientation in generated images without requiring specific training data or being limited to single objects or synthetic data. The key methodology involves a reward-guided sampling approach using a pretrained orientation estimation model (OrientAnything) and a one-step generative flow model, optimized via Langevin dynamics with adaptive time rescaling. Quantitative results on the MS-COCO-Single benchmark show ORIGEN achieves significantly better orientation alignment (e.g., 87.1% Acc.@22.5° azimuth accuracy) compared to prior orientation-conditioned models and training-free guidance methods. For AI practitioners, this provides a training-free mechanism to impose precise 3D orientation constraints on generated objects, improving spatial controllability in text-to-image synthesis for complex scenes. |
| Free4D: Tuning-free 4D Scene Generation with Spatial-Temporal | ||
| Consistency (Read more on arXiv or HuggingFace) | skhu101, GuangcongWang, FrozenBurning, Inso, tqliu | Free4D introduces a tuning-free framework for generating spatially-temporally consistent 4D scenes from a single image or text input. The primary objective is to produce high-quality, controllable 4D scene representations from limited observations without expensive training or finetuning, ensuring spatial-temporal consistency. Key methodologies involve initializing 4D geometry using image-to-video diffusion and dynamic reconstruction, generating consistent multi-view videos via adaptive guidance and latent replacement strategies, and optimizing a final 4D Gaussian Splatting representation using a coarse-to-fine strategy with modulation-based refinement. Compared to the text-to-4D baseline 4Real on VBench, Free4D demonstrates improved performance in Dynamics (47.4% vs 32.3%) and Aesthetics (64.7% vs 50.9%). For AI practitioners, this work offers an efficient pipeline for generating dynamic 4D scenes directly from single images, reducing reliance on large-scale 4D datasets or model tuning for applications in immersive media and virtual environments. |
| PHYSICS: Benchmarking Foundation Models on University-Level Physics | ||
| Problem Solving (Read more on arXiv or HuggingFace) | armanc, jsous, henryL7, yilunzhao, Carrie777 | This paper introduces PHYSICS, a benchmark with 1,297 university-level physics problems to evaluate foundation models’ advanced problem-solving skills. The primary objective is to assess foundation models’ abilities in multi-step reasoning, mathematical derivation, and domain-specific knowledge application in physics. The methodology involves expert annotation of PhD-qualifying exam problems and a robust automated evaluation system combining SymPy-based verification with GPT-4o assessment. Results show significant limitations even for top models, with the best proprietary model (o3-mini) achieving only 59.9% accuracy, revealing persistent challenges in calculation, assumption validity, and knowledge integration. For AI practitioners, this highlights the substantial gap remaining for models to reach expert-level scientific reasoning, necessitating further research into robust mathematical handling and effective knowledge grounding. |
| Perceptually Accurate 3D Talking Head Generation: New Definitions, | ||
| Speech-Mesh Representation, and Evaluation Metrics (Read more on arXiv or HuggingFace) | taehyunoh, akasha9890, backryun, Han-EunGi, Chae-Yeon | This paper defines criteria and introduces a speech-mesh representation and metrics for perceptually accurate 3D talking head generation. The research aims to define and improve the perceptual accuracy of lip movements in speech-driven 3D talking heads, focusing on Temporal Synchronization, Lip Readability, and Expressiveness. A speech-mesh synchronized representation is developed using a two-stage training process, leveraging large-scale 2D audio-visual data before aligning with 3D mesh data, and is applied as a perceptual loss and metric (PLRS), alongside two new physical metrics (MTM for synchronization, SLCC for expressiveness). Experiments show that incorporating the proposed perceptual loss significantly improves existing models across all three criteria; for instance, applying it to FaceFormer on the VOCASET dataset improved the Perceptual Lip Readability Score (PLRS) from 0.368 to 0.463. AI practitioners can utilize the proposed perceptual loss to enhance the realism of 3D talking heads and employ the introduced metrics (MTM, PLRS, SLCC) for a more comprehensive, perceptually-grounded evaluation beyond traditional geometric error metrics like LVE. |
| Segment Any Motion in Videos (Read more on arXiv or HuggingFace) | Nan Huang, qianqian68, akanazawa, kurtkeutzer, chenfengx | This paper introduces a novel method for Moving Object Segmentation (MOS) by integrating long-range trajectories, semantic features, and foundation model prompting. The objective is to accurately segment objects based solely on their observable motion within a video, even in challenging scenarios like occlusions or complex deformations. The methodology combines long-range point tracks with DINO semantic features using specialized Spatio-Temporal Trajectory Attention and Motion-Semantic Decoupled Embedding, followed by an iterative prompting strategy with SAM2 to generate dense masks from sparse tracks. The proposed approach achieves state-of-the-art results on multiple benchmarks, including a 91.0 F-score on the DAVIS2016 MOS task, outperforming previous methods. For AI practitioners, this work demonstrates a powerful technique for video understanding tasks, showcasing how combining long-term motion cues, semantic context, and large segmentation models like SAM2 can yield robust and precise segmentation of moving objects where traditional optical flow or VOS methods might fail. |
| Hi3DGen: High-fidelity 3D Geometry Generation from Images via Normal | ||
| Bridging (Read more on arXiv or HuggingFace) | Xiaoyang Guo, Jiahao Chang, Yushuang Wu, Chongjie Ye, LUZITENG | Hi3DGen introduces a novel framework for high-fidelity 3D geometry generation from single images by leveraging normal maps as an intermediate bridge. The primary objective is to accurately reproduce fine-grained geometric details from 2D images, addressing limitations like domain gaps and inherent RGB ambiguities in existing methods. Key methodology involves a noise-injected, dual-stream image-to-normal estimator (NiRNE) for sharp normal prediction, and a normal-to-geometry latent diffusion learner (NoRLD) with explicit normal map regularization, supported by a high-quality synthetic 3D dataset (DetailVerse). The framework demonstrates superior performance, with NiRNE achieving a Normal Error (NE) of 21.837 on the LUCES-MV dataset, significantly outperforming prior state-of-the-art methods, and user studies confirm higher perceived fidelity. For AI practitioners, this work presents a technique using normal maps as an explicit intermediate representation with regularization in latent diffusion to significantly enhance the geometric detail and fidelity of single-image 3D model generation pipelines. |
| ReFeed: Multi-dimensional Summarization Refinement with Reflective | ||
| Reasoning on Feedback (Read more on arXiv or HuggingFace) | jasoncai, hwany-j, Myyhlee, hyang0503, hamzzi | This paper introduces ReFeed, a pipeline employing reflective reasoning on feedback to refine text summaries across multiple quality dimensions simultaneously. The primary objective is to enhance summarization refinement beyond single dimensions like faithfulness, addressing inter-dimensional trade-offs, feedback ordering bias, and sensitivity to noisy LLM-generated feedback. ReFeed utilizes a novel dataset, SumFeed-CoT, containing Long-CoT reflective reasoning distilled from a large reasoning model, to fine-tune a lightweight model (LLaMA-3.1-8B) capable of backtracking and validating feedback during refinement. Experiments show ReFeed significantly outperforms baselines, improving average summary quality by 8.4 points over initial summaries and specifically boosting completeness by 13.6 points, while demonstrating robustness to feedback noise and order. For AI practitioners, ReFeed offers a method and dataset to build lightweight yet effective multi-dimensional refinement models that mitigate quality trade-offs by incorporating distilled reflective reasoning, crucial for robust real-world deployment. |
| OThink-MR1: Stimulating multimodal generalized reasoning capabilities | ||
| via dynamic reinforcement learning (Read more on arXiv or HuggingFace) | Changwang Zhang, Feng Liu, Yuting Zhang, Zhiyuan Liu, jwanglux | OThink-MR1 introduces GRPO-D, a dynamic reinforcement learning strategy, to enhance the generalized multimodal reasoning capabilities of MLLMs beyond standard fine-tuning. The primary objective is to overcome the limitations of SFT and static RL by developing a dynamic RL approach (GRPO-D) that fosters better same-task performance and cross-task generalization for multimodal reasoning. The key methodology is GRPO-D, which employs a dynamically adjusted Kullback-Leibler (KL) divergence weight during reinforcement learning fine-tuning to optimally balance policy exploration and exploitation based on verifiable multimodal task rewards. GRPO-D demonstrated superior same-task and cross-task performance, achieving over a 61.63% relative improvement versus SFT in cross-task generalization evaluations where SFT showed poor transferability. For AI practitioners, GRPO-D provides a superior fine-tuning technique for MLLMs, enabling the development of models with stronger, transferable reasoning abilities across diverse multimodal tasks without requiring retraining for each specific task. |
| Your ViT is Secretly an Image Segmentation Model (Read more on arXiv or HuggingFace) | Giuseppe Averta, Narges Norouzi, Alexander Hermans, Niccolò Cavagnero, Tommie Kerssies | This paper introduces the Encoder-only Mask Transformer (EoMT), demonstrating that a plain Vision Transformer (ViT) can perform image segmentation without task-specific components like adapters or decoders. The study investigates if these components are essential for state-of-the-art ViT-based segmentation, hypothesizing their relevance diminishes with larger models and extensive pre-training. By systematically removing components from a ViT-Adapter + Mask2Former baseline and repurposing the ViT encoder blocks to process learnable queries alongside patch tokens, supplemented by a mask annealing strategy for efficient inference, EoMT is developed. Results show that EoMT with ViT-L achieves comparable Panoptic Quality (56.0 PQ) to the baseline (57.1 PQ) on COCO while being 4.4x faster (128 FPS vs 29 FPS). For AI practitioners, this implies that investing compute in scaling ViT models and pre-training, rather than adding architectural complexity, can yield simpler, faster, and highly accurate segmentation models that readily benefit from foundation model advancements. |
| 4D-Bench: Benchmarking Multi-modal Large Language Models for 4D Object | ||
| Understanding (Read more on arXiv or HuggingFace) | mhelhoseiny, ajhamdi, TonNew, bing-li-ai, vxuanz | This paper introduces 4D-Bench, the first benchmark designed to evaluate the capabilities of Multimodal Large Language Models (MLLMs) in understanding dynamic 4D objects through question answering and captioning tasks. The objective is to assess current MLLM performance in multi-view spatial-temporal reasoning for 4D assets, addressing the lack of standardized evaluation in this domain. The methodology involved creating a dataset from rendered dynamic 3D objects (Objaverse-XL) into multi-view videos, curating data via motion and quality filters, and generating challenging QA pairs and human-annotated captions, followed by evaluating multiple MLLMs using accuracy and diverse captioning metrics, including GPT-4o assessment. Key results show MLLMs significantly underperform humans, with the state-of-the-art GPT-4o achieving only 62.98% overall accuracy on the 4D object QA task compared to a 91.08% human baseline, demonstrating particular weakness in object counting (37.29% average accuracy) and temporal reasoning. For AI practitioners, this highlights substantial MLLM limitations in integrating complex spatial-temporal information for 4D objects and handling counterfactual data, indicating a need for developing more robust models for applications involving dynamic 3D assets. |
| A Refined Analysis of Massive Activations in LLMs (Read more on arXiv or HuggingFace) | Fabian Güra, akanyaani, nilabhra, louisowen6 | This paper analyzes massive activations across diverse LLMs, challenging prior assumptions and evaluating mitigation strategies. The research objective is to systematically assess the characteristics, impact, and mitigation of massive activations across a broader range of GLU and non-GLU based LLM architectures than previously studied. Methodology involves intervention analysis (setting activations to zero/mean) on pre-trained models and retraining LLaMA-1B/GPT-2 with mitigation techniques (Attention KV Bias, TVR, DyT, hybrids), evaluating perplexity and downstream task performance. Primary results contradict prior claims, showing not all massive activations are detrimental, Attention KV bias mitigation is ineffective for architectures like LLaMA-1B, and hybrid strategies such as TVR + KV Bias successfully mitigate activations in LLaMA-1B (mean downstream task accuracy 52.0 vs 50.3 baseline) while preserving performance. The principal implication for AI practitioners is that mitigating massive activations, crucial for quantization and numerical stability, requires architecture-specific analysis and potentially hybrid approaches like TVR+KV Bias or TVR+DyT, as universal solutions are ineffective. |
| SparseFlex: High-Resolution and Arbitrary-Topology 3D Shape Modeling (Read more on arXiv or HuggingFace) | Lp256, pookiefoof, bennyguo, zouzx, XianglongHe | SparseFlex introduces a sparse-structured isosurface representation for high-resolution, arbitrary-topology 3D shape modeling. The primary objective is to create high-fidelity 3D meshes (up to 1024³) with complex geometries, open surfaces, and interiors directly from rendering supervision, overcoming limitations of existing methods. Key methodologies involve adapting Flexicubes within a sparse voxel structure and employing a novel frustum-aware sectional voxel training strategy that activates only relevant voxels during rendering to drastically reduce memory consumption. Experiments demonstrate state-of-the-art reconstruction accuracy, evidenced by an ~82% reduction in Chamfer Distance and an ~88% increase in F-score compared to previous methods on tested benchmarks. For AI practitioners, this work provides a memory-efficient pathway to train high-resolution, differentiable mesh reconstruction and generation models using only rendering losses, facilitating the creation of detailed 3D assets with arbitrary topology without costly watertight preprocessing. |
| MedAgent-Pro: Towards Multi-modal Evidence-based Medical Diagnosis via | ||
| Reasoning Agentic Workflow (Read more on arXiv or HuggingFace) | Yueming Jin, Chang Han Low, morson, ZiyueWang | MedAgent-Pro introduces a reasoning agentic workflow for evidence-based, multi-modal medical diagnosis. The primary objective is to enhance diagnostic reliability and explainability compared to standard MLLMs by strictly adhering to retrieved clinical criteria and enabling quantitative analysis. The methodology utilizes a hierarchical agentic workflow: a task-level planner uses RAG to generate diagnostic plans based on medical knowledge, while case-level tool agents (specialized vision/VQA models, coding agent) execute steps on patient data, followed by a decider agent integrating findings. MedAgent-Pro significantly outperformed baselines, achieving 90.4% mACC on Glaucoma diagnosis using its MOE decider, a 32.3% absolute improvement over the best single foundation model tested (BioMedClip). For AI practitioners, this work implies that augmenting MLLMs with structured agentic workflows, external specialized tools, and explicit knowledge retrieval is crucial for building reliable and interpretable systems in domains requiring rigorous, evidence-based quantitative reasoning like medical diagnosis. |
| X^{2}-Gaussian: 4D Radiative Gaussian Splatting for Continuous-time | ||
| Tomographic Reconstruction (Read more on arXiv or HuggingFace) | yixuanyuan, XGGNet, Fanzhiwen, CaiYuanhao, vortex778 | X²-Gaussian presents a novel framework for continuous-time 4D computed tomography (CT) reconstruction using dynamic radiative Gaussian splatting. The objective is to reconstruct 4D CT volumes at arbitrary time points directly from projections, eliminating discrete phase binning and the need for external respiratory gating devices. The methodology integrates dynamic radiative Gaussian splatting, modeled via a spatiotemporal encoder-decoder for continuous deformation prediction, with a self-supervised, physiology-driven periodic consistency loss to learn respiratory cycles directly from projection data. Results demonstrate state-of-the-art performance, achieving a 9.93 dB PSNR improvement over traditional methods and a 2.25 dB gain over prior Gaussian splatting approaches on the DIR dataset. For AI practitioners, this provides a hardware-free method for high-fidelity, continuous dynamic medical image reconstruction, potentially enhancing motion analysis in clinical applications like image-guided radiotherapy. |
| On Large Multimodal Models as Open-World Image Classifiers (Read more on arXiv or HuggingFace) | Yiming Wang, Enrico Fini, paolorota, massimilianom, altndrr | This paper evaluates Large Multimodal Models (LMMs) for open-world image classification beyond predefined categories. The objective was to assess LMM performance in an unconstrained classification setting and analyze prediction errors using novel metrics. The methodology involved evaluating 13 LMMs on 10 benchmarks using four metrics (Text Inclusion, Llama Inclusion, Semantic Similarity, Concept Similarity) to measure alignment between generated text and ground truth labels. Results indicate LMMs outperform open-world contrastive baselines (e.g., CaSED) on inclusion metrics but significantly underperform closed-world models (e.g., CLIP), with notable errors in granularity (e.g., predicting “dog” instead of “pug”) and fine-grained discrimination; for instance, even the best models struggled significantly on very fine-grained datasets, often achieving near 0% Text Inclusion. AI practitioners should recognize current LMMs’ limitations in specific open-world classification, noting that while promising, tailored prompting and reasoning only partially alleviate errors related to granularity and fine-grained distinctions compared to traditional closed-world approaches. |
| Reconstructing Humans with a Biomechanically Accurate Skeleton (Read more on arXiv or HuggingFace) | Qixing Huang, Etienne Vouga, Xiaowei Zhou, geopavlakos, IsshikiHugh | This paper presents HSMR, a method for single-image 3D human reconstruction using the biomechanically accurate SKEL model. The main objective is to estimate SKEL parameters directly from an image, overcoming the lack of paired image-SKEL training data. HSMR utilizes a transformer network trained with iteratively refined pseudo-ground truth SKEL parameters generated by converting existing SMPL datasets and optimizing against 2D keypoints (“SKELify”). HSMR achieves competitive performance on standard benchmarks compared to SMPL-based methods like HMR2.0, while significantly outperforming them (by >10mm PA-MPJPE) on datasets with extreme poses like MOYO and reducing unnatural joint rotations. For AI practitioners, this offers a way to generate more physically plausible 3D human models directly from images, which is crucial for biomechanics, robotics, and simulation applications where joint limits and skeletal accuracy are paramount. |
Papers for 2025-03-28
| Title | Authors | Summary |
|---|---|---|
| Video-R1: Reinforcing Video Reasoning in MLLMs (Read more on arXiv or HuggingFace) | Potentialts, guozonghao96, BreakLee, kxgong, KaituoFeng | Video-R1 introduces a rule-based reinforcement learning framework to enhance video reasoning capabilities within Multimodal Large Language Models (MLLMs). The primary objective is to adapt the R1 reasoning paradigm for video by addressing the lack of explicit temporal modeling in standard RL algorithms and the scarcity of high-quality video reasoning data. The methodology involves proposing the Temporal Group Relative Policy Optimization (T-GRPO) algorithm, which contrasts performance on ordered versus shuffled video frames, and utilizing curated hybrid datasets (Video-R1-COT-165k, Video-R1-260k) combining image and video reasoning samples. Key results show significant improvements across video benchmarks, notably achieving 35.8% accuracy on VSI-Bench with the 7B model, surpassing the proprietary GPT-4o model. For AI practitioners, this research demonstrates that temporal-aware RL algorithms like T-GRPO, coupled with hybrid image-video data, offer an effective approach to improve complex temporal reasoning in MLLMs for video understanding applications. |
| UI-R1: Enhancing Action Prediction of GUI Agents by Reinforcement | ||
| Learning (Read more on arXiv or HuggingFace) | Xi Yin, hsli-cuhk, guoyaxuan0106, Yuxiang007, LZXzju | This paper introduces UI-R1, leveraging rule-based reinforcement learning (RL) to enhance graphical user interface (GUI) action prediction for multimodal large language models (MLLMs). The main objective was to investigate if rule-based RL could improve MLLM reasoning capabilities for GUI action prediction using significantly less data than supervised fine-tuning (SFT). Key methodology involved curating a 136-sample mobile GUI task dataset, designing a unified rule-based reward function for action type and coordinate accuracy, and applying Group Relative Policy Optimization (GRPO) for reinforcement fine-tuning (RFT) on a Qwen2.5-VL-3B model. The primary result showed UI-R1-3B improved action type accuracy by 15% and grounding accuracy by 10.3% on the in-domain ANDROIDCONTROL benchmark compared to its base model, while using only 136 training samples, and achieved competitive out-of-domain performance against larger SFT models trained on 76K data. The principal implication for AI practitioners is that rule-based RFT presents a highly data-efficient method for improving GUI agent performance and generalization, offering a viable alternative to large-scale SFT, particularly in resource-constrained or OOD scenarios. |
| Challenging the Boundaries of Reasoning: An Olympiad-Level Math | ||
| Benchmark for Large Language Models (Read more on arXiv or HuggingFace) | Wayne Xin Zhao, jrwen, TimothyCzp, EliverQ, CoderBak | This paper introduces OlymMATH, a new bilingual Olympiad-level mathematical benchmark designed to rigorously evaluate the complex reasoning capabilities of large language models (LLMs). The primary objective is to address the saturation of existing math reasoning benchmarks by providing a more challenging test set derived from manually verified, non-digital sources. The methodology involved curating 200 problems (split into AIME-level easy and harder Olympiad-level tiers) across four mathematical fields, providing parallel English and Chinese versions with verifiable numerical answers. Empirical results show state-of-the-art models like DeepSeek-R1 achieve low accuracy (21.2% Pass@1) on the OlymMATH-EN-HARD subset, indicating significant limitations in current LLM reasoning. For AI practitioners, OlymMATH serves as a demanding benchmark to better differentiate advanced reasoning models and identify weaknesses, such as reliance on heuristics over rigorous derivation, guiding the development of more robust mathematical problem-solving capabilities. |
| VBench-2.0: Advancing Video Generation Benchmark Suite for Intrinsic | ||
| Faithfulness (Read more on arXiv or HuggingFace) | mimihe, yinanhe, jackyhate, HongboLiu, Ziqi | VBench-2.0 introduces an automated benchmark suite designed to evaluate the intrinsic faithfulness of video generation models, moving beyond superficial quality assessments. Its primary objective is to systematically measure adherence to principles like physics, commonsense reasoning, human fidelity, controllability, and creativity across 18 fine-grained dimensions. The methodology integrates Vision-Language Models (VLMs) and Large Language Models (LLMs) through text description alignment and video-based multi-question answering, alongside specialist detectors and heuristics, validated via human preference annotations. Evaluations reveal current state-of-the-art models struggle significantly with complex plot generation (~10-12% scores) and dynamic attribute control (~8-24% scores), although VBench-2.0’s automated metrics show strong alignment with human judgment (Spearman’s ρ > 0.8 across most dimensions). For AI practitioners, VBench-2.0 provides a standardized framework to assess and guide the development of video generation models towards greater realism and adherence to world principles, crucial for applications requiring simulation and complex scene understanding. |
| LeX-Art: Rethinking Text Generation via Scalable High-Quality Data | ||
| Synthesis (Read more on arXiv or HuggingFace) | Dakerqi, afdsafas, Xxxy13, QJerry, stzhao | LeX-Art introduces a data-centric framework using scalable high-quality synthesis to improve visual text rendering in text-to-image (T2I) generation. The main objective is to bridge the gap between prompt expressiveness and text rendering fidelity by enhancing data quality and fine-tuning models, rather than relying solely on control-based architectural changes. The methodology involves using DeepSeek-R1 for prompt enrichment, generating the LeX-10K dataset (10K 1024x1024 images) via multi-stage filtering, developing the LeX-Enhancer prompt model, fine-tuning LeX-FLUX and LeX-Lumina T2I models, and introducing the LeX-Bench benchmark and PNED metric for evaluation. Primary results demonstrate significant improvements, with LeX-Lumina achieving a 79.81% PNED gain (indicating better text accuracy) on CreateBench compared to its baseline. For AI practitioners, the principal implication is that this scalable, data-centric approach, leveraging high-quality synthetic data and prompt enhancement, offers an effective method to substantially improve text rendering fidelity and aesthetics in T2I models without requiring complex model architecture modifications. |
| Large Language Model Agent: A Survey on Methodology, Applications and | ||
| Challenges (Read more on arXiv or HuggingFace) | qqlong, joeyleo, evan-gyy, yszhao, luojunyu | This survey systematically reviews Large Language Model (LLM) agents, covering their methodologies, applications, and challenges. The primary objective is to deconstruct LLM agent systems through a methodology-centered taxonomy, linking architectural foundations (construction), interaction mechanisms (collaboration), and improvement pathways (evolution). It employs a tripartite framework analyzing agent construction (profile, memory, planning, action execution), collaboration paradigms (centralized, decentralized, hybrid), and evolution mechanisms (autonomous learning, co-evolution, external resources), complemented by analysis of evaluation, tools, real-world issues, and applications. The survey provides a unified architectural perspective, identifies significant challenges including scalability, memory constraints, reliability, and evaluation complexity, and offers a structured understanding distinct from prior works focusing on isolated aspects. For AI practitioners, this work delivers a comprehensive taxonomy and framework for understanding the design principles, lifecycle, and practical considerations crucial for developing and deploying robust LLM agent systems. |
| Lumina-Image 2.0: A Unified and Efficient Image Generative Framework (Read more on arXiv or HuggingFace) | luyiting, Paper99, RuoyiDu, JackyZhuo, Dakerqi | Lumina-Image 2.0 introduces a unified and efficient text-to-image generation framework improving upon Lumina-Next. The main objective is to enhance image fidelity, prompt adherence, and generation efficiency through architectural unification and improved training data. Key methodologies include the Unified Next-DiT architecture for joint text-image token processing, the Unified Captioner (UniCap) for generating high-quality, multi-granularity captions, multi-stage progressive training, and inference optimizations like CFG-Renormalization and CFG-Truncation. Lumina-Image 2.0 achieves strong performance, scoring 87.20 on the DPG benchmark with only 2.6B parameters, demonstrating superior efficiency and scalability compared to prior models. For AI practitioners, this work presents an efficient (2.6B parameters) and unified transformer architecture applicable beyond T2I, alongside a specialized captioning system (UniCap) that significantly improves training data quality and model convergence, offering a practical approach to building performant generative models. |
| ReaRAG: Knowledge-guided Reasoning Enhances Factuality of Large | ||
| Reasoning Models with Iterative Retrieval Augmented Generation (Read more on arXiv or HuggingFace) | chenyn66, liuweichuan, NeoZ123, caoshulin, ZhiCheng0326 | ReaRAG enhances Large Reasoning Model (LRM) factuality for multi-hop QA using iterative, knowledge-guided Retrieval-Augmented Generation (RAG) with reflective reasoning. The objective is to improve LRM factual accuracy on complex QA tasks by mitigating reliance on parametric knowledge and issues like overthinking and error propagation found in prior iterative RAG and RL-based approaches. The methodology involves constructing a dataset with bounded reasoning chains, fine-tuning ReaRAG-9B (based on GLM-4-9B) using a Thought-Action-Observation paradigm, iteratively querying a RAG engine, and employing reflection to refine the reasoning trajectory. ReaRAG-9B significantly outperforms baselines on multi-hop QA benchmarks, achieving a 14.5% ACCL improvement over SearChain on MuSiQue (66.00 vs 51.50 ACCL). For AI practitioners, ReaRAG provides a fine-tuning framework and inference strategy to build more factually reliable QA systems by effectively integrating iterative external knowledge retrieval and explicit reasoning steps, reducing errors compared to solely prompt-based or single-retrieval RAG methods. |
| Embodied-Reasoner: Synergizing Visual Search, Reasoning, and Action for | ||
| Embodied Interactive Tasks (Read more on arXiv or HuggingFace) | Guiyang1001, tricktreat, yijiang, Gangao, zwq2018 | This paper presents Embodied-Reasoner, a model extending ol-style reasoning to interactive embodied search tasks by generating and learning from coherent Observation-Thought-Action trajectories. The primary objective is to enhance reasoning capabilities for embodied agents facing challenges like continuous multimodal interaction, spatial understanding, temporal reasoning, and self-reflection based on interaction history. Key methodology involves synthesizing 9.3k trajectories featuring diverse thinking processes (e.g., analysis, spatial reasoning, reflection) and employing a three-stage training pipeline comprising imitation learning, self-exploration via rejection sampling, and self-correction via reflection tuning. Results demonstrate significant improvements over advanced visual reasoning models, with Embodied-Reasoner exceeding OpenAI GPT-o1 by +9% and GPT-o3-mini by +24% in success rate, showing fewer repeated searches and better consistency on long-horizon tasks. For AI practitioners, this work provides a data synthesis and training framework to develop embodied agents with enhanced planning, reasoning, and interaction capabilities, particularly for complex tasks requiring adaptive behavior based on visual feedback and interaction history. |
| ResearchBench: Benchmarking LLMs in Scientific Discovery via | ||
| Inspiration-Based Task Decomposition (Read more on arXiv or HuggingFace) | yuqiangli, bgao22182, jinjieni, ZonglinY, yujieliu | ResearchBench introduces a benchmark for evaluating Large Language Models (LLMs) in scientific discovery by decomposing the process into inspiration retrieval, hypothesis composition, and ranking. The objective is to assess LLM performance on these fundamental sub-tasks using recent, contamination-resistant scientific literature across 12 disciplines. An automated LLM-based agentic framework extracts research components (questions, background, inspirations, hypotheses) from 1386 papers published in 2024, forming the basis for evaluation, including carefully selected negative examples for retrieval tasks. Results show LLMs excel at the out-of-distribution inspiration retrieval task (GPT-4o hit ratio: 45.65% for top 4% candidates), while hypothesis composition and ranking show moderate capabilities with potential for improvement; ranking is notably affected by position bias. For AI practitioners, this indicates LLMs can serve as “research hypothesis mines” capable of surfacing novel knowledge associations for automated discovery, though the bottleneck in retrieval suggests a reliance on pretraining depth over post-training refinement. |
| Optimal Stepsize for Diffusion Sampling (Read more on arXiv or HuggingFace) | Han Hu, Jianning Pei, cientgu | This paper introduces Optimal Stepsize Distillation (OSS), a dynamic programming framework to derive theoretically optimal stepsize schedules for accelerating diffusion model sampling. The objective is to overcome suboptimal discretization in diffusion sampling by focusing on principled stepsize schedule design, rather than solely optimizing update directions. OSS treats stepsize optimization as knowledge distillation, using dynamic programming to recursively minimize the global discretization error between a few-step student sampler and a many-step teacher reference trajectory. Experiments demonstrate that OSS enables significant acceleration, achieving 10x speedup for text-to-image generation while maintaining 99.4% of the teacher model’s performance on the GenEval benchmark. For AI practitioners, OSS provides a robust, architecture-agnostic method to drastically reduce diffusion model inference latency with minimal performance loss, enabling more efficient deployment. |
| Exploring the Evolution of Physics Cognition in Video Generation: A | ||
| Survey (Read more on arXiv or HuggingFace) | huangsiteng, wangcunxiang, huangsiteng, yishanwang, minnielin | This survey reviews the integration of physical cognition into video generation models, organizing advancements along an evolutionary path inspired by human cognitive development. The main objective is to systematically categorize methods for improving physical fidelity in generated videos, addressing the gap between visual realism and physical plausibility. The paper proposes a three-tier taxonomy (Basic Schematic Perception, Passive Cognition, Active Cognition) to classify techniques like motion-guided generation, physics-inspired regularization, simulation integration, and LLM-based reasoning. Despite progress, the survey highlights that even state-of-the-art models often violate fundamental physical laws, generating visually appealing but physically inconsistent results, as evidenced by evaluations on benchmarks like PhyGenBench [86] and Physics-IQ [84]. For AI practitioners, this implies that achieving physically plausible video generation, essential for applications like robotics and simulation, requires moving beyond visual mimicry towards integrating explicit physical knowledge and interaction mechanisms. |
| ChatAnyone: Stylized Real-time Portrait Video Generation with | ||
| Hierarchical Motion Diffusion Model (Read more on arXiv or HuggingFace) | Peng Zhang, Chaonan Ji, Jinwei Qi, Liefeng, shengxu97 | ChatAnyone introduces a novel framework for generating stylized, real-time upper-body portrait videos from audio using a hierarchical motion diffusion model and hybrid control fusion GAN. The primary objective is to create expressive digital humans with synchronized facial expressions, head poses, and upper-body movements including hands, enabling fine-grained style control. The methodology involves a two-stage process: first, hierarchical motion diffusion models predict explicit and implicit motion representations from audio and optional style references; second, a warping-based GAN synthesizes the video using these representations, injected hand controls, and a face refinement module. Key results demonstrate real-time performance (up to 30fps at 512x768 on a 4090 GPU) and improved quantitative metrics, such as achieving a PSNR of 24.88 in self-reenactment, significantly outperforming prior GAN-based methods. For AI practitioners, this provides an effective approach for developing highly expressive, controllable, and real-time digital avatars for interactive applications like video chat and virtual assistants, demonstrating the power of combining diffusion models for motion generation with GANs for efficient synthesis. |
| FinAudio: A Benchmark for Audio Large Language Models in Financial | ||
| Applications (Read more on arXiv or HuggingFace) | Yueru1, Shashidhar, ShirleyY, Acatsama, YupengCao | FinAudio introduces the first benchmark specifically designed to assess Audio Large Language Models (AudioLLMs) within the financial domain. The primary objective is to evaluate the capacity of current AudioLLMs on realistic financial audio tasks, revealing their strengths and limitations. The methodology involves defining three tasks (short-clip ASR, long-recording ASR, and summarization), curating five datasets (MDRM, SPGISpeech, Earnings-21, Earnings-22, FinAudioSum) totaling over 400 hours, and evaluating seven diverse AudioLLMs. Key results show significant performance variation, with Whisper-v3 achieving the lowest Word Error Rate (WER) on short-clip ASR (2-3%), but performance degrading across models for long audio ASR (Whisper-v3: 12-16% WER) and summarization being dependent on initial ASR quality. For AI practitioners, this benchmark reveals that while open-source models like Whisper-v3 provide a strong baseline, current AudioLLMs struggle with long financial recordings and specialized terminology/numerical data, highlighting the need for improved context handling and domain-specific adaptation. |
| Synthetic Video Enhances Physical Fidelity in Video Synthesis (Read more on arXiv or HuggingFace) | Ziyan Yang, Ziyu Wang, Qi Zhao, fengcheng1, Univstar | This research demonstrates that integrating synthetic videos from CGI pipelines improves the physical fidelity of generative video synthesis models. The objective was to investigate whether synthetic videos, generated with physical consistency using computer graphics, can enhance the physical realism (e.g., 3D consistency, human pose integrity) of diffusion-based video generation models. The methodology involved generating synthetic videos using Blender/Unreal Engine, curating this data based on factors like asset/rendering quality and camera setups, employing a specific captioning strategy, and introducing a training technique called SimDrop to integrate synthetic data while mitigating visual artifacts using a reference model and classifier-free guidance. Primary results show significant improvement in physical fidelity across tasks like large human motion, camera rotation, and layer decomposition; for instance, on the camera spin shot task, the synthetically-enhanced model achieved an 80% success rate in user studies compared to 20% for the baseline and reduced the 3D reconstruction re-projection error (ê_proj) from 0.437 to 0.135. The principal implication for AI practitioners is that leveraging carefully curated synthetic video data, combined with techniques like SimDrop, offers a data-centric approach to enhance the physical consistency and reduce artifacts in video generation models without requiring modifications to the core model architecture. |
| ZJUKLAB at SemEval-2025 Task 4: Unlearning via Model Merging (Read more on arXiv or HuggingFace) | Ziyan Jiang, Yi Zhong, Yanqiu Zhao, Saberlve, HaomingXu | ZJUKLAB employed TIES-Merging of two specialized models to address selective unlearning in Large Language Models for SemEval-2025 Task 4. The objective was to effectively erase sensitive content by balancing the trade-off between over-forgetting general knowledge and under-forgetting targeted data. Their methodology involved training two distinct LoRA models using Negative Preference Optimization (NPO), Gradient Descent on Retain set (GDR), and KL divergence minimization (KLR) to induce complementary biases, then merging them using TIES-Merging. The merged system ranked second online (Task Aggregate 0.944) and locally achieved an Aggregate Score of 0.806 and a near-optimal MIA AUC of 0.501, significantly outperforming the individual biased models. For AI practitioners, this demonstrates model merging as a practical technique to combine models with opposing unlearning biases for more effective and balanced sensitive data removal, though limitations in current evaluation metrics are noted. |
| Feature4X: Bridging Any Monocular Video to 4D Agentic AI with Versatile | ||
| Gaussian Feature Fields (Read more on arXiv or HuggingFace) | Hui Ren, Fanzhiwen, ir1d, ShuwangZhang00, shijiezhou | Feature4X provides a universal framework to lift arbitrary 2D vision foundation model functionalities into interactive 4D agentic AI systems using only monocular video input. Its main objective is to enable versatile 4D scene understanding and interaction (segmentation, editing, VQA) from readily available monocular videos, overcoming the limitations of 4D data scarcity. The key methodology involves distilling diverse 2D features into a compact, unified dynamic 4D Gaussian feature field represented using Gaussian Splatting and Motion Scaffolds, trained end-to-end and integrated with LLMs. Primary results include robust novel-view segmentation, language-guided 4D scene editing, and spatiotemporal VQA, with semantic segmentation achieving comparable accuracy to baselines while being approximately 6.2x more space-efficient (95.4MB vs 593.9MB). For AI practitioners, this offers a scalable method to extend existing 2D vision model capabilities to dynamic 4D environments, facilitating the development of interactive 4D agentic AI applications without requiring extensive annotated 4D datasets. |
| Unified Multimodal Discrete Diffusion (Read more on arXiv or HuggingFace) | Katerina Fragkiadaki, Deepak765, Sid1275, mihirpd, aswerdlow | This paper introduces UniDisc, a unified multimodal discrete diffusion model for joint text and image generation. The objective is to explore discrete diffusion models as an alternative unified generative formulation for joint text and image domains, comparing their advantages over autoregressive (AR) models. UniDisc employs a transformer architecture trained using a discrete diffusion process involving masking tokens (text and image) with an absorbing state and learning to denoise via a weighted cross-entropy objective. Results show UniDisc outperforms AR models in conditional generation using classifier-free guidance (CFG), enables zero-shot joint text-image inpainting, and demonstrates superior joint retrieval accuracy (e.g., 0.64 vs 0.17 on DataComp1B). For AI practitioners, UniDisc offers enhanced controllability, editability, and a flexible inference time vs. quality trade-off for multimodal generation tasks compared to traditional AR approaches, although scaling analysis indicates it requires approximately 13.2x more training compute for equivalent loss levels. |
| LOCATEdit: Graph Laplacian Optimized Cross Attention for Localized | ||
| Text-Guided Image Editing (Read more on arXiv or HuggingFace) | Sirisha Rambhatla, Meet Soni, Achint Soni | LOCATEdit introduces graph Laplacian optimization on cross- and self-attention maps (CASA graphs) for precise, localized text-guided image editing. The primary objective is to improve spatial consistency and confine edits to target regions, mitigating artifacts and distortions common in methods relying solely on cross-attention maps from diffusion models. Key methodology involves constructing CASA graphs from attention maps, applying graph Laplacian regularization to enforce smoothness and optimize attention values, integrating IP-Adapter guidance, and using selective pruning on text embedding differences. LOCATEdit significantly outperforms baselines on PIE-Bench, achieving, for example, a background preservation SSIM of 86.52 (x10^2) with DPM-Solver++(20), demonstrating superior localization and fidelity. For AI practitioners, this work provides a robust, training-free technique using graph-based optimization on attention mechanisms to achieve more controlled and spatially consistent results in text-guided generative image editing tasks. |
| LLPut: Investigating Large Language Models for Bug Report-Based Input | ||
| Generation (Read more on arXiv or HuggingFace) | Tarannum Shaila Zaman, imranraad, Subarna10, alifalhasan | This paper investigates the effectiveness of generative Large Language Models (LLMs) in extracting failure-inducing input commands from natural language bug reports. The primary research objective is to empirically evaluate how effectively three open-source generative LLMs (LLaMA, Qwen, Qwen-Coder) can extract these inputs compared to a fine-tuned BERT model. Using a dataset of 206 annotated Linux coreutils bug reports and a one-shot prompting strategy, the study evaluates extraction accuracy against human annotations using BLEU scores. The generative LLMs significantly outperformed the BERT baseline, with Qwen yielding the best results, achieving a BLEU-2 score of ≥ 0.5 for 62.62% of its extracted commands. For AI practitioners, this indicates that generative LLMs offer considerable potential for automating the extraction of executable commands from bug reports, aiding debugging workflows, though challenges in handling command variations and extraction failures persist. |
Papers for 2025-03-27
| Title | Authors | Summary |
|---|---|---|
| Dita: Scaling Diffusion Transformer for Generalist | ||
| Vision-Language-Action Policy (Read more on arXiv or HuggingFace) | TTTTTony, MIASANMIA, robot-haonan, TianyiZhang0213, zhihou | Dita introduces a scalable Diffusion Transformer architecture for generalist vision-language-action robot policies. The primary objective is to develop a versatile, open-source VLA model capable of zero-shot or few-shot generalization across diverse robotic embodiments, tasks, and environments, particularly addressing long-horizon tasks and environmental variations. The key methodology involves using a causal Transformer to directly denoise continuous action sequences via a diffusion process, conditioned in-context on raw visual tokens (from DINOv2 and Q-Former) and language instructions (from CLIP). Dita achieves state-of-the-art or competitive performance on simulation benchmarks, notably attaining an 82.4% average success rate on LIBERO (a ~6% improvement over prior methods), and demonstrates robust real-world adaptation with 10-shot finetuning on complex, long-horizon tasks under varying conditions. For AI practitioners, Dita provides a lightweight (334M parameters) and effective open-source framework that integrates Transformer scalability with inherent diffusion denoising via in-context conditioning, offering a strong baseline for developing adaptable robot policies requiring minimal task-specific data. |
| Qwen2.5-Omni Technical Report (Read more on arXiv or HuggingFace) | JialinWang, chenkq, bluelike, jinzheng-he, ZhifangGuo | Qwen2.5-Omni is an end-to-end multimodal model processing text, image, audio, and video to generate streaming text and speech responses. The primary objective is to develop a unified model capable of perceiving diverse streaming inputs, synchronizing temporal modalities like audio and video, and concurrently generating both text and low-latency speech outputs. Key methodologies include block-wise processing for input encoders, Time-aligned Multimodal RoPE (TMROPE) for audio-video synchronization, and a Thinker-Talker architecture separating text generation (Thinker LLM) from streaming speech token generation (Talker), using a sliding-window DiT for audio decoding. Primary results demonstrate state-of-the-art performance on benchmarks like OmniBench (56.13% average score), comparable end-to-end speech instruction following capabilities to text input on tasks like GSM8K (88.7% speech accuracy vs 91.6% text accuracy for Qwen2.5-7B), and robust streaming speech generation with 6.54% WER on the seed-tts-eval test-hard set after reinforcement learning. For AI practitioners, this work offers the Thinker-Talker architecture and TMROPE as a framework for building unified streaming multimodal systems that handle synchronized inputs and generate real-time text and speech, enabling more natural human-AI interaction. |
| LEGO-Puzzles: How Good Are MLLMs at Multi-Step Spatial Reasoning? (Read more on arXiv or HuggingFace) | Leoxing, KennyUTC, zengyh1900, favourisnotyou, KexianTang | This paper introduces LEGO-Puzzles, a benchmark designed to evaluate multi-step spatial reasoning in Multimodal Large Language Models (MLLMs). The objective is to assess MLLMs’ capabilities in both spatial understanding and sequential reasoning through diverse LEGO construction-based tasks. The methodology involves a curated dataset of over 1,100 visual question-answering (VQA) pairs across 11 tasks, alongside image generation evaluations, tested on 20 state-of-the-art MLLMs. Results reveal significant limitations; even the best MLLM (GPT-40) achieved only 57.7% overall accuracy, far below human performance (93.6%), with particular weaknesses in multi-step sequential reasoning and spatially grounded image generation. For AI practitioners, this highlights critical deficiencies in current MLLMs’ spatial intelligence, underscoring the need for advancements in models intended for complex real-world applications like robotics and automated assembly that demand robust sequential spatial reasoning. |
| Wan: Open and Advanced Large-Scale Video Generative Models (Read more on arXiv or HuggingFace) | HermanZ, chenweix7, chaojiemao, baoleai, ang-annng | This paper introduces Wan, an open-source suite of advanced large-scale video generative models based on the Diffusion Transformer paradigm. The objective is to push video generation boundaries by developing high-performance, efficient, and comprehensive open-source models (1.3B and 14B parameters) trained on billions of images/videos. Key methodologies include a novel spatio-temporal VAE, scalable pre-training with Flow Matching, large-scale data curation, and extensions to tasks like I2V, editing, personalization, and real-time generation. The 14B model achieved a leading Wan-Bench score of 0.724, outperforming competitors, while the 1.3B model demonstrated consumer-grade efficiency requiring only 8.19 GB VRAM for 480p inference. For AI practitioners, Wan provides open-source access to powerful (14B) and efficient (1.3B) foundation models, code, and training details, enabling the development of diverse video generation applications, including potential deployment on consumer GPUs with the 1.3B model. |
| Unconditional Priors Matter! Improving Conditional Generation of | ||
| Fine-Tuned Diffusion Models (Read more on arXiv or HuggingFace) | Jaihoon Kim, Minhyuk, phillipinseoul, prinphunya | This paper introduces a training-free method to enhance conditional generation from fine-tuned diffusion models by utilizing stronger unconditional priors from base models. The primary objective is to address the degradation in conditional generation quality caused by poor unconditional noise predictions learned during Classifier-Free Guidance (CFG) based fine-tuning. The key methodology involves replacing the unconditional noise prediction term in the CFG sampling process of the fine-tuned model with the corresponding prediction from its original base model or another pretrained model with robust unconditional generation capabilities. Results demonstrate significant improvements; for example, applying this method to Zero-1-to-3 novel view synthesis using SD2.1 as the unconditional prior improved LPIPS from 0.182 to 0.158 and PSNR from 16.647 to 17.801. For AI practitioners, this implies that during inference with CFG-based fine-tuned diffusion models, leveraging the unconditional prior from a separate, well-trained unconditional model can substantially boost conditional output quality without requiring model retraining or architectural changes. |
| Open Deep Search: Democratizing Search with Open-source Reasoning Agents (Read more on arXiv or HuggingFace) | speedyarda, ljirwin, pchiniya, cabxyz, salzubi401 | Open Deep Search (ODS) is introduced as an open-source framework augmenting LLMs with reasoning agents and web search tools to rival proprietary search AI. The primary objective is to bridge the performance gap between open-source and closed-source search AI solutions by enhancing LLM reasoning with real-time web information. ODS employs two main components: an Open Search Tool for improved web context retrieval and an Open Reasoning Agent (using ReAct or CodeAct) to orchestrate tool use, including the search tool, calculator, and code interpreter, based on user queries. Key results show ODS-v2 paired with DeepSeek-R1 achieves 75.3% accuracy on the FRAMES benchmark, outperforming GPT-4o Search Preview by 9.7%, and 88.3% on SimpleQA. For AI practitioners, ODS offers a modular, open-source system to integrate advanced search and reasoning into any base LLM, enabling state-of-the-art performance on fact-based question answering without dependence on closed systems. |
| GenHancer: Imperfect Generative Models are Secretly Strong | ||
| Vision-Centric Enhancers (Read more on arXiv or HuggingFace) | yshan2u, yxgeee, aether25, tttoaster, msj9817 | GenHancer enhances CLIP’s fine-grained visual representations using lightweight generative models without requiring perfect reconstruction or pre-trained denoisers. The objective is to explore how imperfect generative models can effectively transfer fine-grained visual knowledge to discriminative models like CLIP, investigating optimal conditioning, denoising configurations, and generation paradigms. The key methodology involves a two-stage post-training approach using lightweight, randomly initialized continuous or discrete denoisers conditioned solely on CLIP’s global ([CLS]) token for self-supervised reconstruction, employing techniques like LoRA and scaled Logit-Normal timestamp sampling. GenHancer consistently outperforms prior methods, achieving a 6.0% improvement over the baseline OpenAICLIP on the MMVP-VLM benchmark, demonstrating that perfect generation is not necessary for representation enhancement. For AI practitioners, this implies that fine-grained visual capabilities of CLIP-based systems (like MLLMs) can be significantly and efficiently improved post-hoc using lightweight generative models focused on specific conditioning (global token only) and training strategies, avoiding computationally expensive heavy denoisers. |
| BizGen: Advancing Article-level Visual Text Rendering for Infographics | ||
| Generation (Read more on arXiv or HuggingFace) | YuanYuhui, kevinlin311tw, bohanChen, Marseclipse, wukeming11 | BizGen introduces a framework for generating high-quality infographics and slides with accurate article-level visual text rendering and adherence to ultra-dense layouts. The primary objective is to overcome the challenges of significantly longer text contexts and the scarcity of high-quality business content data compared to standard text-to-image tasks. Key methodologies include the creation of a large-scale dataset (INFOGRAPHICS-650K) via retrieval-augmented generation and a novel layout-guided cross-attention mechanism with layout-conditional Classifier-Free Guidance (CFG) for region-wise control. BizGen significantly outperforms models like FLUX and SD3 on the BizEval benchmark, achieving over 25% absolute improvement in visual text spelling accuracy (OCR) on infographics with more than 20 layers compared to FLUX. For AI practitioners, BizGen offers a scalable data generation strategy and a controllable diffusion model architecture to produce complex, text-rich business graphics demanding high fidelity to dense layouts and long-form textual content. |
| Gemini Robotics: Bringing AI into the Physical World (Read more on arXiv or HuggingFace) | abalakrishna123, TravisAStrong, montse90, jalayrac, saminda | This paper introduces Gemini Robotics, a family of AI models based on Gemini 2.0 designed to bridge AI capabilities into the physical world via robotics. The main objective is to endow large multimodal models with robust embodied reasoning and dexterous physical interaction capabilities for general-purpose robot control. Key methodologies include enhancing Gemini 2.0’s embodied reasoning (Gemini Robotics-ER), evaluated on a new ERQA benchmark, and fine-tuning a Vision-Language-Action (VLA) model (Gemini Robotics) on extensive robot action data for direct, low-latency control. The generalist Gemini Robotics VLA achieved high proficiency out-of-the-box, succeeding on 50% of 20 diverse dexterous manipulation tasks with over 80% success rate, and demonstrated strong generalization and rapid adaptation to new tasks and embodiments. For AI practitioners, this work shows that large multimodal foundation models, when specifically trained for embodied reasoning and grounded with robot interaction data, provide a viable foundation for developing more general-purpose, dexterous, and adaptable robotic agents. |
| MCTS-RAG: Enhancing Retrieval-Augmented Generation with Monte Carlo Tree | ||
| Search (Read more on arXiv or HuggingFace) | armanc, chenzhao, yilunzhao, AlexCCtop | MCTS-RAG integrates Monte Carlo Tree Search (MCTS) with Retrieval-Augmented Generation (RAG) to improve reasoning capabilities of small language models (SLMs) on knowledge-intensive tasks. The research aims to overcome SLM limitations in accessing and utilizing external knowledge by dynamically combining structured reasoning search with adaptive retrieval. The methodology employs MCTS to explore reasoning paths, introducing specific RAG actions (Retrieval Reasoning, Retrieval Decompose) at decision points, guided by UCT, and evaluates paths using retrieved information. Key results show MCTS-RAG enabled Llama 3.1-8B to achieve over 20% absolute accuracy improvement on ComplexWebQA and roughly 15% on GPQA compared to baseline methods. For AI practitioners, this work presents an effective inference-time compute scaling method to significantly enhance the performance of smaller LMs on complex, knowledge-reliant tasks without model retraining, offering a pathway to achieve higher accuracy with more resource-efficient models. |
| AccVideo: Accelerating Video Diffusion Model with Synthetic Dataset (Read more on arXiv or HuggingFace) | Yunhong Wang, XihuiLiu, YaohuiW, AriaChen, aejion | AccVideo accelerates video diffusion models through distillation using a synthetic dataset of denoising trajectories. The research objective is to reduce the extensive inference steps required by video diffusion models while maintaining output quality by avoiding distillation on irrelevant data points. Key methodology involves generating a synthetic dataset (SynVid) with full denoising trajectories from a pretrained teacher model, training a student model using trajectory-based few-step guidance on keyframes from these trajectories, and employing an adversarial training strategy with timestep-aware discriminators. The primary result is an 8.5x reduction in inference time compared to the teacher model (HunyuanVideo), generating 720x1280 videos in 380s vs 3234s with comparable quality. For AI practitioners, this demonstrates an effective technique to significantly speed up high-resolution video generation from diffusion models, making them more feasible for real-world deployment by leveraging synthetic data distillation. |
| ViLBench: A Suite for Vision-Language Process Reward Modeling (Read more on arXiv or HuggingFace) | cihangxie, xianft, alihiker, Helicopt, PahaII | This paper introduces VILBENCH, a benchmark suite for vision-language process reward modeling, alongside a new dataset (ViLReward-73K) and a trained process reward model (ViLPRM). The main objective is to evaluate the effectiveness of vision-language large models (VLLMs) as process reward models (PRMs) and output reward models (ORMs), and to develop improved PRMs for tasks requiring step-wise reasoning. Key methodologies include benchmarking seven VLLMs on five VL datasets, filtering data to create VILBENCH emphasizing step-wise rewards, collecting preference data using an enhanced MCTS algorithm, and training a 3B parameter ViLPRM based on QwenVL-2.5. Primary results show neither ORM nor PRM consistently outperforms the other across tasks using general VLLMs, while the trained ViLPRM achieves an average improvement of 3.3% over standard Chain-of-Thought evaluation on VILBENCH. For AI practitioners, this indicates that specialized PRMs trained on process supervision data, like ViLPRM, can better evaluate complex vision-language reasoning steps than general VLLMs or ORMs, highlighting a pathway to improve model alignment and evaluation for multi-step multimodal tasks. |
| LogQuant: Log-Distributed 2-Bit Quantization of KV Cache with Superior | ||
| Accuracy Preservation (Read more on arXiv or HuggingFace) | Pingyi Luo, Bingsheng He, deciding, Zicong99, Concyclics | LogQuant introduces a log-distributed 2-bit quantization method for LLM KV Caches, improving accuracy preservation over existing techniques. The objective is to reduce KV Cache memory usage via 2-bit quantization while mitigating the associated accuracy loss by selectively preserving important tokens based on a log-distributed attention pattern. The methodology involves applying a base-2 logarithmic filtering strategy to retain tokens with decreasing density further from the current position, quantizing less critical tokens to 2-bits while keeping a dynamic window of recent tokens (2W to 3W) at full precision. LogQuant demonstrated superior performance, improving accuracy by 40%-200% on Math and Code tasks compared to KiVi at similar compression ratios, and boosting throughput by 25% over a BF16 baseline. For AI practitioners, LogQuant offers a way to deploy LLMs with long contexts more efficiently on memory-constrained hardware by significantly reducing KV Cache size with better accuracy retention than prior 2-bit quantization approaches. |
| ADS-Edit: A Multimodal Knowledge Editing Dataset for Autonomous Driving | ||
| Systems (Read more on arXiv or HuggingFace) | xzwnlp, bozhong, xiangchen-dvi, JizhanFang, Chenxiwang | This paper introduces ADS-Edit, a multimodal benchmark dataset for evaluating knowledge editing techniques applied to Large Multimodal Models (LMMs) in Autonomous Driving Systems (ADS). The research objective is to assess how effectively knowledge editing can update LMMs with domain-specific ADS knowledge (addressing traffic knowledge gaps, complex conditions, dynamic states) without requiring full retraining. The methodology involves constructing the ADS-Edit benchmark from existing ADS datasets (LingoQA, DriveLM, CODA-LM) with video, multi-view, and single-image data across perception, understanding, and decision-making scenarios, and evaluating four editing baselines (Prompt, AdaLora, GRACE, WISE) on reliability, generality, and locality. Primary results demonstrate that memory-based methods achieve high reliability (e.g., GRACE reached 100% reliability on single edits), but differ significantly in generality (GRACE <30%, WISE ~85-95%), with WISE showing strong locality (~100%). For AI practitioners, ADS-Edit provides a framework to evaluate and select knowledge editing methods for efficiently updating LMMs in ADS, indicating WISE offers a balanced trade-off for update reliability, generalization, and parameter preservation. |
| Beyond Words: Advancing Long-Text Image Generation via Multimodal | ||
| Autoregressive Models (Read more on arXiv or HuggingFace) | Min Li, Lijuan, zyang39, linjieli222, Awiny | This paper presents LongTextAR, a multimodal autoregressive model enabling high-fidelity long-text image generation. It addresses the challenge of accurately rendering extensive textual content in images, a limitation of current generative models. The methodology identifies Vector Quantization (VQ) tokenization bottlenecks and introduces TextBinarizer, a novel text-focused binary tokenizer, integrated into a Llama2-based autoregressive architecture trained on text-rich data. LongTextAR significantly outperforms models like SD3.5 Large, achieving 69.5% OCR accuracy on long texts (>10 words) versus 52.3% for SD3.5 Large, and offers controllable text rendering (font, size, color, alignment). For AI practitioners, this work demonstrates that specialized tokenization within an autoregressive framework provides a strong alternative to diffusion models for generating images requiring accurate, controllable long text, impacting applications like automated document and presentation creation. |
| Attention IoU: Examining Biases in CelebA using Attention Maps (Read more on arXiv or HuggingFace) | Vikram V. Ramaswamy, Olga Russakovsky, tyleryzhu, serianni | This paper introduces Attention-IoU, a metric using attention maps to quantify biases within computer vision classification models by analyzing internal representations. The objective is to identify spurious correlations and understand how specific image features contribute to biased predictions, moving beyond performance disparities. The core methodology uses a generalized Intersection-over-Union (Attention-IoU) to compare GradCAM attention maps against ground-truth feature masks (mask score) or other attribute attention maps (heatmap score). Validation on Waterbirds shows the mask score accurately tracks induced bias (decreasing from 0.72±0.02 to 0.42±0.03 as bias increases from 50% to 100%), and analysis on CelebA reveals Attention-IoU uncovers correlations like that between Blond_Hair and Male (heatmap score 0.72±0.02) potentially linked to unlabeled confounders, unlike Wavy_Hair (0.65±0.03). For AI practitioners, Attention-IoU provides a tool to pinpoint spatial sources of bias within models, indicating that biases can stem from internal representations not solely reflected in dataset label correlations, thus informing more targeted debiasing interventions. |
| Self-Supervised Learning of Motion Concepts by Optimizing | ||
| Counterfactuals (Read more on arXiv or HuggingFace) | Kevin Feigelis, Rahul Venkatesh, Seungwoo Kim, Stefan Stojanov, kmeisthax | Opt-CWM introduces a self-supervised technique for optical flow and occlusion estimation by optimizing counterfactual probes on a pre-trained video prediction model without labeled data. The primary objective is to develop a method that extracts motion concepts from unlabeled videos by learning optimal input perturbations for a base Counterfactual World Model (CWM), avoiding fixed heuristics. Key methodology involves parameterizing perturbations with a learnable network trained jointly with a sparse flow-conditioned predictor using an asymmetric masking principle and RGB reconstruction loss. Results demonstrate state-of-the-art performance on real-world benchmarks compared to other self-supervised methods, achieving an Average Jaccard (AJ) of 47.53 and Average Distance (AD) of 8.73 on TAP-Vid First (DAVIS). For AI practitioners, this work provides a scalable, self-supervised approach to extract robust motion primitives from vast unlabeled video data, beneficial for applications requiring motion understanding without reliance on synthetic datasets or manual heuristics. |
| Sparse Logit Sampling: Accelerating Knowledge Distillation in LLMs (Read more on arXiv or HuggingFace) | kw1jjang, Rock222, AndrewAhn, ya-mehdi, Anshumann | This paper introduces Random Sampling Knowledge Distillation (RS-KD), an importance-sampling method for accelerating LLM pre-training distillation using sparse teacher logits. The research aims to develop an efficient offline knowledge distillation strategy for LLM pre-training that requires storing only a sparse subset of teacher logits without compromising student model performance or calibration. The key methodology involves using importance sampling (specifically, sampling proportional to teacher probabilities) to create unbiased sparse target distributions, theoretically and empirically contrasting this with biased Top-K sampling approaches. Primary results show that RS-KD achieves performance comparable to full distillation using only 12 unique sampled tokens, maintains near-perfect calibration (ECE ~0.8%), preserves expected gradients (4° angular difference vs. FullKD), and offers significant training throughput gains (1.7x-2.6x faster than FullKD). For AI practitioners, RS-KD offers a computationally efficient method to pre-train smaller LLMs via offline distillation, drastically reducing the storage required for teacher logits (using ~0.01%) and accelerating training with marginal overhead compared to standard cross-entropy training. |
| DINeMo: Learning Neural Mesh Models with no 3D Annotations (Read more on arXiv or HuggingFace) | Alan Yuille, Weijie Guo, wufeim, guofeng1123 | DINeMo presents a neural mesh model for category-level 3D pose estimation trained without 3D annotations. The main objective is to overcome the limitation of requiring extensive 3D annotations for training neural mesh models, enabling broader applicability and scalability. The key methodology involves leveraging pseudo-correspondence derived from large visual foundation models (SD-DINO) via a novel bidirectional generation process that integrates local features and global context, combined with Grounded-SAM for enhanced inference. DINeMo significantly outperforms previous zero- and few-shot methods on PASCAL3D+ car pose estimation (e.g., narrowing the gap with fully-supervised methods by 67.3% on Acc@pi/18, LO) and demonstrates effective scaling with additional unlabeled training data. For AI practitioners, this work offers a viable pathway to develop robust 3D object understanding models without relying on difficult-to-obtain 3D ground truth, utilizing unlabeled image data for training. |
| Image as an IMU: Estimating Camera Motion from a Single Motion-Blurred | ||
| Image (Read more on arXiv or HuggingFace) | r0nn13, jerredchen | This paper introduces a method to estimate instantaneous camera rotational (ω) and translational (v) velocity directly from motion blur within a single image. The objective is to leverage motion blur, often considered an artifact, as the primary source of information for robust ego-motion estimation during fast camera movements, eliminating the need for IMUs or multi-frame analysis. The approach first predicts dense motion flow and monocular depth using a neural network, then recovers velocity by solving a differentiable linear least squares system derived from motion field equations, enabling end-to-end training on synthetic and real data. Evaluated on real-world data, the method yields state-of-the-art velocity estimates (e.g., average rotational RMSE 1.22/0.91/1.76 rad/s), significantly outperforming MASt3R and COLMAP, and achieves real-time performance (30 FPS). AI practitioners can apply this technique for real-time, drift-free, IMU-like velocity measurements in high-motion scenarios (e.g., robotics, AR/VR) using only a single blurred camera image, enhancing robustness where traditional VO/SLAM methods fail. |
| PathoHR: Breast Cancer Survival Prediction on High-Resolution | ||
| Pathological Images (Read more on arXiv or HuggingFace) | Rundong Xue, Jiaxuan Xiao, Jun Liu, Shiru Wang, Yang Luo | PathoHR is a novel pipeline for breast cancer survival prediction using enhanced high-resolution pathological image features and optimized similarity learning. The main objective is to improve survival prediction accuracy by effectively extracting representative features from high-resolution WSIs while managing computational costs and addressing tumor heterogeneity. The methodology involves patch-wise feature extraction using a pre-trained encoder, integrating a plug-and-play high-resolution Vision Transformer (ViTAR) for feature enhancement, and systematically evaluating various similarity metrics (e.g., Cosine, Euclidean, Attention Score) for adaptive token merging. Results demonstrate that using enhanced 16x16 patches with the PathoHR pipeline (specifically with cosine similarity) achieves superior performance (AUC 0.90741) compared to baseline methods using larger raw 24x24 patches (AUC 0.8), validating the approach’s effectiveness and efficiency. For AI practitioners, this implies that integrating resolution enhancement techniques (like high-res ViTs) with optimized similarity-based feature learning can enable more accurate analysis of large medical images using smaller patches, reducing computational overhead without sacrificing predictive power. |
Papers for 2025-03-26
| Title | Authors | Summary |
|---|---|---|
| Long-Context Autoregressive Video Modeling with Next-Frame Prediction (Read more on arXiv or HuggingFace) | Mike Zheng Shou, Weijia Mao, Yuchao Gu | This paper introduces Frame AutoRegressive (FAR), a baseline for long-context autoregressive video modeling using next-frame prediction. The research objective is to address challenges in long-context video modeling, namely visual redundancy impacting temporal extrapolation and computational costs associated with long sequences. Key methodologies include FAR trained with a frame-wise flow matching objective and causal attention, stochastic clean context to bridge the train-inference gap, FlexRoPE for improved test-time temporal extrapolation (up to 16x), and long short-term context modeling for efficient training on longer videos. Primary results show FAR achieves state-of-the-art performance, outperforming Token-AR and demonstrating better convergence than video diffusion transformers, achieving an FVD of 279 on UCF-101 (Table 2, FAR-XL). For AI practitioners, FAR provides an effective and simpler baseline framework for autoregressive video generation that naturally supports variable-length context and improves temporal consistency in long videos compared to existing methods. |
| CoMP: Continual Multimodal Pre-training for Vision Foundation Models (Read more on arXiv or HuggingFace) | Yu-Gang Jiang, Zuxuan Wu, Wujian Peng, Lingchen Meng, Row11n | This paper introduces COMP, a continual multimodal pre-training method enhancing Vision Foundation Models (VFMs) for native resolution processing and better language alignment. The objective is to adapt prevailing VFMs, regardless of their original training, to handle diverse image sizes and produce visual features more congruent with Large Language Model (LLM) representations. COMP utilizes Continual Rotary Position Embedding (C-ROPE) for variable resolution inputs and an Alignment Loss for explicit cross-modal feature alignment within a three-stage training framework. Results show COMP-SigLIP achieves significant gains, reaching 66.7 on ChartQA and 75.9 on DocVQA with a 0.5B LLM, while largely maintaining performance on unimodal tasks like ImageNet-1K classification (87.4%). For AI practitioners, COMP provides a mechanism to upgrade existing VFMs, enabling them to serve as more effective vision encoders for LLMs, particularly in tasks demanding fine-grained visual understanding from native resolution images. |
| Exploring Hallucination of Large Multimodal Models in Video | ||
| Understanding: Benchmark, Analysis and Mitigation (Read more on arXiv or HuggingFace) | Yue Liu, Baolong Bi, Jingyi Tang, Jiashu Qu, Hongcheng Gao | This paper introduces HAVEN, a benchmark to evaluate and mitigate hallucinations in Large Multimodal Models (LMMs) for video understanding. The main objective is to systematically analyze hallucination causes (prior conflict, in-context conflict, capability deficiency) and aspects (object, scene, event) in videos and develop mitigation strategies. Key methodology involves constructing the 6K-question HAVEN benchmark and proposing a thinking-based mitigation approach combining supervised reasoning fine-tuning (SRFT) and thinking-based direct preference optimization (TDPO). Primary results show significant variation in hallucination across 16 LMMs, with the proposed SRFT+TDPO method improving baseline accuracy by 7.65% on hallucination evaluation and reducing the consistency bias score by 4.5%. For AI practitioners, HAVEN offers a standardized tool to assess video LMM reliability regarding hallucinations, while the SRFT+TDPO training strategy presents a method to enhance model factuality and reasoning in video tasks. |
| Inference-Time Scaling for Flow Models via Stochastic Generation and | ||
| Rollover Budget Forcing (Read more on arXiv or HuggingFace) | Minhyuk Sung, Jisung Hwang, Taehoon Yoon, Jaihoon Kim | This paper introduces an inference-time scaling approach for pretrained flow models using stochastic generation and adaptive compute allocation to enhance alignment with user preferences. The main objective is to enable effective inference-time scaling, similar to diffusion models, for deterministic flow models without retraining. The key methodology involves converting the flow model’s ODE to an SDE, using a Variance Preserving (VP) interpolant instead of a linear one to increase diversity, and applying Rollover Budget Forcing (RBF) to adaptively allocate computation across timesteps. Results show the VP-SDE with RBF significantly improves compositional alignment, achieving a VQAScore of 0.925, outperforming the base model (0.726) and diffusion models even with fewer computations (NFEs). For AI practitioners, this method allows enhancing existing flow models to better follow complex prompts (e.g., counting, spatial relations) during inference, offering a computationally efficient way to improve output quality and alignment compared to standard generation or diffusion model scaling approaches. |
| Spot the Fake: Large Multimodal Model-Based Synthetic Image Detection | ||
| with Artifact Explanation (Read more on arXiv or HuggingFace) | Zichen Wen, Hengrui Kang, Peilin Feng, Junyan Ye, Siwei Wen | This paper introduces FakeVLM, a specialized large multimodal model for detecting synthetic images and providing artifact explanations, alongside the FakeClue dataset. The primary objective is to create an LMM-based system capable of accurately classifying images as real or synthetic (general and DeepFake) while offering interpretable, natural language explanations for detected artifacts. FakeVLM employs a LLaVA-v1.5 architecture, fine-tuning all parameters on the novel FakeClue dataset (>100k images, 7 categories) which features fine-grained artifact annotations generated via a multi-LMM strategy and category-specific prompts, framing detection as an explanatory visual question answering task. FakeVLM demonstrated superior performance over baseline LMMs, achieving 0.986 Accuracy and 0.981 F1 score on the FakeClue dataset for combined detection and explanation, nearing expert model performance in detection-only tasks without requiring auxiliary classifiers. For AI practitioners, FakeVLM offers a robust, single-model solution for synthetic image detection that inherently provides interpretability, enhancing trust and transparency in authenticity assessment pipelines compared to black-box classifiers or less specialized LMMs. |
| Scaling Vision Pre-Training to 4K Resolution (Read more on arXiv or HuggingFace) | Sifei Liu, Yao Lu, Han Cai, Boyi Li, Baifeng Shi | This paper introduces PS3, a method scaling CLIP-style vision pre-training to 4K resolution with near-constant computational cost by selectively processing local regions instead of entire high-resolution images. The objective is to overcome the prohibitive quadratic/quartic cost of training vision models on high-resolution inputs. PS3 employs a multi-stage architecture involving low-resolution global feature extraction, top-down/bottom-up patch selection based on saliency or text prompts, and multi-scale high-resolution feature extraction on selected patches using localized contrastive learning. Applied within a Multimodal Large Language Model (MLLM) named VILA-HD, PS3 significantly improves performance on high-resolution tasks; on the proposed 4KPro benchmark, VILA-HD achieves 74.2% accuracy, outperforming Qwen2-VL by 3.2% while being 2.96x faster. For AI practitioners, PS3 provides a computationally efficient pre-training framework enabling MLLMs to perceive fine-grained details in 4K images, significantly enhancing capabilities for tasks requiring high-resolution visual understanding with reduced inference latency compared to full-image processing or token pruning methods. |
| Think Twice: Enhancing LLM Reasoning by Scaling Multi-round Test-time | ||
| Thinking (Read more on arXiv or HuggingFace) | Yunjie Ji, Shuaiting Chen, Haotian Wang, Sitong Zhao, Xiaoyu Tian | This paper introduces “Multi-round Thinking,” a test-time scaling method enhancing large language model (LLM) reasoning by iteratively refining answers using previous outputs as prompts. The main objective is to improve LLM reasoning performance, especially on complex tasks, by overcoming limitations of single-step reasoning and cognitive inertia without requiring additional training. The key methodology involves repeatedly prompting the LLM with the original question concatenated with the model’s final answer from the previous round, using a specific prompt template. Primary results show consistent performance gains across models and benchmarks; for example, QwQ-32B improved pass@1 accuracy on AIME 2024 from 80.3% (Round 1) to 82.1% (Round 2), and DeepSeek-R1 improved from 79.7% to 82.0%. For AI practitioners, this simple, training-free technique offers a practical method to potentially enhance LLM accuracy at inference time simply by re-prompting, although it incurs additional computational cost and latency per round. |
| CoLLM: A Large Language Model for Composed Image Retrieval (Read more on arXiv or HuggingFace) | Son Tran, Mubarak Shah, Ashish Tawari, Jinyu Yang, Chuong Huynh | CoLLM introduces a Large Language Model (LLM) based framework for Composed Image Retrieval (CIR) that synthesizes training triplets dynamically from image-caption pairs. The objective is to overcome CIR data scarcity, enhance multimodal query understanding using LLMs, and improve evaluation benchmark reliability. Key methodology includes synthesizing reference image embeddings using Spherical Linear Interpolation (Slerp) and modification text using template-based interpolation between image-caption pairs, feeding these into an LLM for composed query embedding generation. CoLLM achieves state-of-the-art results on multiple CIR benchmarks, and the introduced MTCIR dataset yields up to 15% performance improvement for baseline models compared to other synthetic datasets. For AI practitioners, the principal implication is a method for supervised CIR model training without expensive manually annotated triplets, providing scalability alongside a large-scale synthetic dataset (MTCIR) and refined evaluation benchmarks. |
| MDocAgent: A Multi-Modal Multi-Agent Framework for Document | ||
| Understanding (Read more on arXiv or HuggingFace) | Yun Li, Tong Sun, Ruiyi Zhang, Peng Xia, Siwei Han | MDocAgent is a novel multi-modal, multi-agent framework integrating text and image retrieval-augmented generation (RAG) for improved document question answering (DocQA). The primary objective is to address the limitations of single-modal DocQA systems by effectively integrating and reasoning over both textual and visual information in complex documents. The methodology utilizes parallel text and image RAG pipelines feeding context to five specialized agents (General, Critical, Text, Image, Summarizing) that collaborate to extract, analyze, and synthesize information guided by extracted critical cues. Preliminary experiments show MDocAgent achieves an average performance improvement of 12.1% over current state-of-the-art methods on five benchmarks using top-1 retrieval. For AI practitioners, this demonstrates that a structured multi-agent, multi-modal RAG approach can enhance DocQA accuracy on complex documents by enabling detailed cross-modal understanding and synthesis beyond single-modal or basic LVLM capabilities. |
| Latent Space Super-Resolution for Higher-Resolution Image Generation | ||
| with Diffusion Models (Read more on arXiv or HuggingFace) | Seon Joo Kim, Jinwoo Kim, Sangmin Han, Jinho Jeong | This paper proposes LSRNA, a framework combining Latent space Super-Resolution (LSR) and Region-wise Noise Addition (RNA) to improve higher-resolution image generation with diffusion models. The objective is to overcome limitations like manifold deviation (latent upsampling) and smoothness (RGB upsampling) in reference-based high-resolution generation, enabling faster inference and better detail preservation beyond native model resolutions. The methodology involves training an LSR module to map low-resolution latents to the high-resolution manifold and using RNA to inject Canny edge-guided noise adaptively, enhancing high-frequency details without progressive upsampling. Integrating LSRNA into DemoFusion for 16x resolution (4096x4096) reduced generation time to 34% (1507s to 506s) and improved patch-FID from 32.89 to 29.12 compared to the baseline DemoFusion. AI practitioners can leverage LSRNA to accelerate and enhance detail in high-resolution image generation pipelines built on pretrained diffusion models, offering a superior alternative to progressive latent upscaling or RGB-space upsampling methods. |
| ReSearch: Learning to Reason with Search for LLMs via Reinforcement | ||
| Learning (Read more on arXiv or HuggingFace) | Chenzheng Zhu, Yijie Zhou, Haoze Sun, Tianpeng Li, Mingyang Chen | ReSearch trains Large Language Models (LLMs) to integrate reasoning with external search using reinforcement learning, without supervised data on reasoning steps. The primary objective is to enable LLMs to handle complex multi-hop questions requiring multiple retrieval steps by treating search operations as part of the reasoning chain. The key methodology involves using Group Relative Policy Optimization (GRPO), where the LLM generates text thoughts and search queries, receives retrieval results, and is optimized based solely on rewards derived from final answer correctness and format adherence. Experiments training Qwen2.5 models showed significant improvements over baselines on multi-hop QA benchmarks, with average absolute gains ranging from 8.9% to 22.4% across benchmarks, such as a 17.56% average LLM-as-a-judge improvement for the 7B model. For AI practitioners, this demonstrates a viable approach to train more capable reasoning and multi-step Retrieval-Augmented Generation (RAG) systems using reinforcement learning from final outcomes, reducing the need for costly supervised reasoning data and enhancing model generalizability. |
| LookAhead Tuning: Safer Language Models via Partial Answer Previews (Read more on arXiv or HuggingFace) | Mengshu Sun, Lin Yuan, Yujie Luo, Mengru Wang, Kangwei Liu | This paper introduces LookAhead Tuning, a data modification technique using partial answer previews to preserve large language model (LLM) safety during fine-tuning. The primary objective is to mitigate the degradation of safety alignment caused by fine-tuning, particularly on benign data, without sacrificing downstream task performance. The key methodology involves modifying training data instructions by appending either the initial tokens of the ground-truth answer (Real Answer) or a fixed prefix phrase (Virtual Answer), thereby minimizing perturbations to the model’s initial token distributions. Results show LookAhead Tuning (virtual) significantly improves safety metrics (e.g., +20.76% average Jailbreak Safe Rate) compared to vanilla fine-tuning, while maintaining comparable utility (-1.59% average decrease across tasks). For AI practitioners, this presents a simple, low-resource, data-centric method to fine-tune models more safely without requiring architectural changes or significant computational overhead. |
| Frequency Dynamic Convolution for Dense Image Prediction (Read more on arXiv or HuggingFace) | Ying Fu, Chenggang Yan, Liang Li, Lin Gu, CharlesChen2023 | Frequency Dynamic Convolution (FDConv) introduces a novel approach to enhance dynamic convolution by learning frequency-diverse weights within a fixed budget in the Fourier domain. The primary objective is to overcome the limited adaptability and high parameter cost associated with the frequency homogeneity observed in traditional dynamic convolution methods. FDConv employs Fourier Disjoint Weight (FDW) to create diverse parallel weights from frequency-grouped spectral coefficients, Kernel Spatial Modulation (KSM) for fine-grained spatial filter adjustment, and Frequency Band Modulation (FBM) for spatially varying frequency response adaptation. Applied to ResNet-50 for object detection, FDConv achieves an Apbox of 39.4 on COCO with only +3.6M parameters, outperforming prior methods requiring substantially larger parameter increases (e.g., ODConv +65.1M for 39.2 Apbox). For AI practitioners, FDConv provides a parameter-efficient module to improve the adaptability and performance of vision models on dense prediction tasks by explicitly managing weight frequency diversity, integrating readily into existing ConvNet and Transformer architectures. |
| LPOSS: Label Propagation Over Patches and Pixels for Open-vocabulary | ||
| Semantic Segmentation (Read more on arXiv or HuggingFace) | Giorgos Tolias, Jiří Matas, Yannis Kalantidis, Vladan Stojnić | This paper presents LPOSS/LPOSS+, a training-free label propagation method for improving open-vocabulary semantic segmentation using Vision-Language and Vision Models. The objective is to enhance coarse initial VLM patch-level predictions and overcome patch-resolution limitations by propagating labels across patches and then pixels. The methodology involves a two-stage label propagation (LP) process: first on a patch graph using Vision Model features for affinities (LPOSS), followed by pixel-level LP initialized with patch-level results (LPOSS+), enabling joint prediction across the entire image. LPOSS+ achieves state-of-the-art performance among training-free methods, attaining an average mIoU of 42.1% across eight datasets with ViT-B/16 backbones. For AI practitioners, LPOSS+ offers a plug-and-play, training-free technique to significantly refine segmentation outputs from existing VLMs, particularly improving accuracy near object boundaries without requiring model retraining. |
| Gumbel-Softmax Flow Matching with Straight-Through Guidance for | ||
| Controllable Biological Sequence Generation (Read more on arXiv or HuggingFace) | Alexander Tong, Yinuo Zhang, Sophia Tang, pranamanam | This paper introduces Gumbel-Softmax Flow Matching and Score Matching, generative frameworks operating on the continuous simplex for biological sequence design. The primary objective is to develop a scalable and controllable method for generating discrete sequences like DNA and proteins by learning smooth interpolations from noise to data using a novel Gumbel-Softmax interpolant with time-dependent temperature. Methodologically, it derives velocity fields for flow matching and score functions for score matching based on this interpolant and introduces Straight-Through Guided Flows (STGFlow), a training-free classifier guidance technique leveraging straight-through estimators. Results demonstrate state-of-the-art performance in conditional DNA promoter design (MSE 0.029), competitive de novo protein generation, and effective target-binding peptide design using STGFlow guidance, outperforming existing binders in docking scores. For AI practitioners, this provides a scalable flow-matching framework for discrete data generation on the simplex, offering a modular, training-free guidance mechanism (STGFlow) to control generation towards desired properties using pre-trained classifiers. |
| Strong Baseline: Multi-UAV Tracking via YOLOv12 with BoT-SORT-ReID (Read more on arXiv or HuggingFace) | wish44165 | This paper presents a strong baseline for multi-UAV tracking in thermal infrared video using YOLOv12 and BoT-SORT-ReID. The objective was to establish a straightforward yet effective tracking workflow leveraging recent advances in detection and tracking, evaluated against the Anti-UAV Challenge metrics. The methodology integrates the YOLOv12 detector with the BoT-SORT tracker (including ReID for multi-object tracking), utilizing staged training and tailored inference strategies for SOT and MOT tasks without contrast enhancement or temporal fusion. Results demonstrate competitive performance, significantly improving over official baselines, achieving a MOTA score of 0.7609 on Track 3, with increased image input resolution identified as the most significant factor contributing approximately 0.1 to score improvement. For AI practitioners, this work provides a validated high-performance baseline for thermal UAV tracking, emphasizing the effectiveness of combining state-of-the-art detection/tracking models and highlighting input resolution tuning as crucial for optimizing performance. |
| When Words Outperform Vision: VLMs Can Self-Improve Via Text-Only | ||
| Training For Human-Centered Decision Making (Read more on arXiv or HuggingFace) | Yu Yin, Jing Li, Zhe Hu | This study demonstrates that Visual Language Models (VLMs) can enhance human-centered decision-making capabilities through text-only training, even achieving self-improvement using data from smaller counterpart LLMs. The primary objective was to improve VLM performance on complex decision-making tasks where they initially underperform compared to text-only LLMs. The methodology involved evaluating baseline models on the VIVA benchmark and then fine-tuning VLMs using synthesized text-only situational data generated by either GPT-4o or Llama-3.1 8B. Results show significant accuracy improvements post-training (e.g., Qwen2-VL improved from 80.32% to 83.15% using GPT-4o data) and notably, that training data generated by the smaller Llama 8B yielded comparable gains, demonstrating VLM self-improvement. For AI practitioners, this indicates that VLM reasoning can be effectively and efficiently enhanced for human-centric tasks via text-only data, bypassing the need for costly image-text pairs and enabling improvement using accessible LLM counterparts. |
| Towards a Unified Copernicus Foundation Model for Earth Vision (Read more on arXiv or HuggingFace) | Thomas Dujardin, Adam J. Stewart, Chenying Liu, Zhitong Xiong, Yi Wang | This paper introduces a unified framework for Earth observation (EO) foundation models integrating data from all major Copernicus Sentinel missions. The objective is to develop a single model capable of processing diverse spectral/non-spectral sensor data and metadata, overcoming the limitations of sensor-specific approaches. The methodology involves creating Copernicus-Pretrain (18.7M aligned images), Copernicus-FM (a model using dynamic hypernetworks and Fourier-encoded metadata), and Copernicus-Bench (a 15-task benchmark). Copernicus-FM demonstrates superior performance, significantly improving results on Sentinel-3/5P tasks compared to prior models and supervised training, achieving an RMSE of 789.4 on AQ-O3-S5P compared to 1755.6 for DOFA [69], with metadata integration yielding substantial gains (e.g., +22.4% OA on EuroSAT-S1). For AI practitioners, this work offers a scalable architecture (Copernicus-FM) and resources (Copernicus-Pretrain, Copernicus-Bench) enabling the development of versatile foundation models for multimodal geospatial data, applicable across diverse EO tasks including atmospheric and climate studies. |
Papers for 2025-03-25
| Title | Authors | Summary |
|---|---|---|
| I Have Covered All the Bases Here: Interpreting Reasoning Features in | ||
| Large Language Models via Sparse Autoencoders (Read more on arXiv or HuggingFace) | Polina Druzhinina, Andrey Galichin, tlenusik, razzant, therem | This research identifies and validates reasoning-specific features in Large Language Models (LLMs) using Sparse Autoencoders (SAEs). The main research question is how reasoning capabilities are internally encoded within LLMs, specifically the DeepSeek-R1 series. The key methodology involves training SAEs on LLM activations, proposing a “ReasonScore” metric to identify reasoning features, and using feature steering to analyze their impact. Primary results show that steering identified features increases reasoning trace length, such as feature i=46379 increasing the completion length by 29% for the AIME 2024 task. The principal implication is that AI practitioners can use SAEs and feature steering to interpret, and potentially improve, the internal reasoning processes of LLMs. |
| Position: Interactive Generative Video as Next-Generation Game Engine (Read more on arXiv or HuggingFace) | XihuiLiu, dizhang, Xintao, chehx, VictorYuki | This position paper proposes Interactive Generative Video (IGV) as the foundation for Generative Game Engines (GGE), enabling AI-driven game development. The main research objective is to demonstrate how IGV can overcome current game engine limitations and serve as the core technology for next-generation game development. The key methodology involves extending video generation models with interactivity, user control, memory, physics-awareness, and causal reasoning to create a comprehensive GGE framework. A hierarchical maturity roadmap (L0-L4) is presented, outlining progressive steps from manual game development to self-evolving world ecosystems, including systems with level L2, where the engine continuously generates physics-compliant video based on users interactions. The principal implication for AI practitioners is that IGV offers a viable pathway to create games with unlimited content, realistic physics, and adaptive gameplay, reducing development barriers and expanding creative possibilities. |
| Video-T1: Test-Time Scaling for Video Generation (Read more on arXiv or HuggingFace) | Hanyang Wang, duanyueqi, xhangzhan, iseesaw, Liuff23 | The paper introduces Video-T1, a framework for improving video generation quality by scaling computation at test time. The main research question is how much video generation quality can be improved by allowing a model to use more inference-time compute, given a challenging text prompt. The key methodology involves reinterpreting test-time scaling as a search problem and using test-time verifiers and heuristic algorithms, including random linear search and Tree-of-Frames (ToF), to sample better trajectories from Gaussian noise. Experiments on text-conditioned video generation benchmarks show that increasing test-time compute consistently improves video quality; for example, the CogVideoX-5B model with Test-Time Scaling (TTS) achieved a total score of 84.42, a 3.44% increase. AI practitioners can use this framework to significantly enhance the quality of generated videos without retraining, by scaling inference-time computation. |
| Aether: Geometric-Aware Unified World Modeling (Read more on arXiv or HuggingFace) | Junyichen, lizizun, AmberHeart, ZhouTimeMachine, HaoyiZhu | AETHER is a unified world model that integrates 4D reconstruction, action-conditioned video prediction, and visual planning using synthetic data. The main research objective is to develop a framework that enables geometry-aware reasoning in world models by jointly optimizing reconstruction, prediction, and planning capabilities. The key methodology involves post-training a video diffusion model with synthetic 4D data, utilizing a robust camera pose annotation pipeline, and integrating cross-task and cross-modal conditioning signals. Primary results show AETHER achieved a zero-shot Absolute Relative error (Abs Rel) of 0.056 on the KITTI dataset for video depth estimation, surpassing prior methods. Principal implication for AI practitioners is that AETHER provides an effective framework for post-training world models with scalable synthetic data, achieving strong zero-shot transfer to real-world tasks and enabling actionable planning. |
| SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for | ||
| Open Base Models in the Wild (Read more on arXiv or HuggingFace) | jxhe, HelicHe, SivilTaram, yuzhen17, AndrewZeng | Zero reinforcement learning (zero RL) training can significantly improve the reasoning abilities of open base language models. The paper investigates how zero RL training impacts the reasoning capabilities of diverse open base language models. The methodology involves training 10 base models (e.g., Llama3-8B, Mistral-7B, Qwen2.5 series) using the GRPO algorithm, with rule-based rewards based solely on answer correctness, on the training sets of GSM8K and MATH datasets. Results show that zero RL training consistently improves accuracy and response length, with Qwen-2.5-32B’s Pass@1 on AIME 24 increasing from 10.0 to 36.7. The study provides AI practitioners with key design factors and empirical findings to enable successful zero RL training, emphasizing alignment of data difficulty with model capability and avoiding overly restrictive format rewards. |
| OmnimatteZero: Training-free Real-time Omnimatte with Pre-trained Video | ||
| Diffusion Models (Read more on arXiv or HuggingFace) | Nir Darshan, ramiben, galchechik, m98levy, Dvir | OmnimatteZero is a training-free approach for video object removal, extraction, and layer composition using pre-trained video diffusion models. The main research objective is to adapt zero-shot image inpainting techniques for efficient and high-quality video omnimatte without requiring model training or optimization. The key methodology leverages self-attention maps from video diffusion models to identify object footprints and effects, then uses latent arithmetic for object layer isolation and blending. OmnimatteZero achieves a PSNR of 39.09 and LPIPS of 0.012 on the Movie dataset for background reconstruction, outperforming all existing methods, and runs at 0.04 seconds per frame on an A100 GPU. AI practitioners can utilize this method for real-time video editing applications like object removal and layer composition without any fine-tuning, requiring only a pre-trained video diffusion model. |
| LEMMA: Learning from Errors for MatheMatical Advancement in LLMs (Read more on arXiv or HuggingFace) | mingchenlin2025, Word2Li, QizhiPei, LHL3341, panzs | LEMMA is a framework that enhances LLMs’ mathematical reasoning by learning from error-corrective trajectories. The main research objective is to improve LLMs’ reflective reasoning capabilities by constructing and learning from data consisting of incorrect solutions, erroneous steps, and reflection connections to correct solutions. The key methodology involves an error-type grounded mistake augmentation method to collect diverse errors, constructing paired reflection data via “Fix & Continue” and “Fresh & Restart” mechanisms, and connecting trajectories with model-aware reflection links. Primary results show that models fine-tuned with LEMMA achieved a 62.4% average accuracy on in-distribution and out-of-distribution math datasets using LLaMA3-8B, outperforming strong baselines. Principal implication is that AI practitioners can significantly improve LLMs’ mathematical reasoning abilities by systematically constructing and learning from structured error data, without reliance on complex external critique models. |
| Equivariant Image Modeling (Read more on arXiv or HuggingFace) | Li Li, Zigang Geng, hanhu2, Mendel192, dongruixiao | The paper introduces an equivariant image modeling framework that aligns optimization targets across subtasks in image generation. The core research question is: Can a task decomposition framework be established to inherently align optimization targets across subtasks in image generation? The method uses column-wise tokenization and windowed causal attention to enhance translational symmetry and enforce consistent contextual relationships. When evaluated on class-conditioned ImageNet generation at 256x256 resolution, the proposed approach achieves a generative FID (gFID) of 5.57, comparable to state-of-the-art AR models with fewer computational resources. The principal implication is that, AI practitioners can improve model efficiency and zero-shot generalization in generative modeling by leveraging inherent equivariance properties of visual data. |
| Training-free Diffusion Acceleration with Bottleneck Sampling (Read more on arXiv or HuggingFace) | lazybone128, Lingaaaaaaa, xiaoxuefeng, renyuxi, tyfeld | The paper introduces Bottleneck Sampling, a training-free framework to accelerate inference in diffusion models by leveraging low-resolution priors. The main research objective is to reduce the computational cost of high-resolution image and video generation in diffusion models without sacrificing output quality. The key methodology is a high-low-high denoising workflow that performs high-resolution denoising at initial and final stages and low-resolution denoising in intermediate steps, with adaptive resolution transition points and timestep shifting. Primary results show that Bottleneck Sampling accelerates inference by up to 3x for image generation and 2.5x for video generation, while maintaining comparable output quality to standard full-resolution sampling. For AI practitioners, Bottleneck Sampling provides a plug-and-play acceleration strategy for existing diffusion models that does not require retraining or architectural modifications, enhancing deployment in resource-constrained environments. |
| Judge Anything: MLLM as a Judge Across Any Modality (Read more on arXiv or HuggingFace) | shuang72, Frywind, NiuniuWang, yuhangchen, fjchendp | This paper introduces TASKANYTHING and JUDGEANYTHING benchmarks to evaluate Multimodal LLMs (MLLMs) as judges across various modalities for multimodal understanding and generation tasks. The main research objective is to evaluate whether MLLMs can serve as a unified judge for assessing the understanding and generation ability of any-to-any modality tasks. The key methodology involves constructing two benchmarks: TASKANYTHING, with 1,500 open-ended queries across 15 any-to-any modality categories, and JUDGEANYTHING, evaluating MLLMs’ judging abilities using Pair Comparison and Score Evaluation settings against human annotations. The primary results show that MLLMs align more closely with human preferences on Pair Comparison than Score Evaluation, with Gemini-1.5-Pro achieving an average of 70.6% accuracy on Pair Comparison for Multimodal Understanding tasks. Principal implication for AI practitioners: Current MLLM-as-a-Judge systems show promise but face limitations, especially in Multimodal Generation tasks, highlighting the need for refined evaluation protocols and improved alignment with human preferences in model development. |
| FFN Fusion: Rethinking Sequential Computation in Large Language Models (Read more on arXiv or HuggingFace) | geifmany, AmnonGeifman, omripuny, mdabbah-nvidia, abercovich | FFN Fusion is a novel architectural optimization that reduces sequential computation in large language models by parallelizing Feed-Forward Network (FFN) layers. The main research objective is to investigate whether sequences of FFN layers in transformers can be parallelized to reduce inference latency while preserving model accuracy. The key methodology involves identifying and fusing consecutive FFN layers into wider, parallel layers, supported by a block-wise dependency analysis and a distillation-based refinement. The primary result is that Ultra-253B-Base, created using FFN Fusion, achieves a 1.71x speedup in inference latency and 35x reduction of the per-token cost compared to its parent Llama-3.1-405B model, while maintaining or exceeding its performance. AI practitioners can apply FFN Fusion to significantly improve the inference efficiency of large language models, particularly in resource-constrained deployment scenarios. |
| CFG-Zero*: Improved Classifier-Free Guidance for Flow Matching Models (Read more on arXiv or HuggingFace) | Ziwei Liu, Raymond A. Yeh, Amber Yijia Zheng, weepiess2383 | CFG-Zero* enhances classifier-free guidance for flow matching models by addressing inaccuracies in early-stage velocity estimation. The main research objective is to improve the sample quality and controllability of flow matching models during generation when the learned velocity is underfitted. The key methodology involves introducing an optimized scale to correct for velocity inaccuracies and a “zero-init” technique that zeros out the first few steps of the ODE solver. Primary results show that CFG-Zero* achieves the best FID Score of 2.10 and sFID Score of 4.59 on ImageNet-256, outperforming existing methods. Principal implication for AI practitioners is that CFG-Zero* can be readily integrated into flow matching models to improve image fidelity and text alignment, particularly during the early stages of training or when models are underfitted. |
| Video SimpleQA: Towards Factuality Evaluation in Large Video Language | ||
| Models (Read more on arXiv or HuggingFace) | Pengfei Hu, zhangysk, Drexubery, grejioh, mengcao | Video SimpleQA, a new benchmark, evaluates the factual accuracy of large video language models (LVLMs). The main research objective is to develop and introduce a comprehensive benchmark for assessing the factuality of LVLMs in video contexts. The key methodology involves creating a dataset of 2030 question-answer pairs derived from 1293 videos, with questions requiring external knowledge, designed to be fact-seeking, and having definitive, short-form, and externally verified answers. Primary results indicate that the best-performing model, Gemini-1.5-Pro, achieves an F-score of only 54.4%, and open-source models perform notably worse. The principal implication for AI practitioners is the need to address significant deficiencies in factual adherence of current LVLMs, highlighting a critical area for improvement in developing models that can accurately and reliably process video information. |
| AgentRxiv: Towards Collaborative Autonomous Research (Read more on arXiv or HuggingFace) | Samuel Schmidgall, mdmoor | Here’s a summary of the paper “AgentRxiv: Towards Collaborative Autonomous Research” by Schmidgall and Moor, following the provided guidelines: 1. AgentRxiv is a framework enabling LLM agent laboratories to collaboratively conduct research by sharing and building upon findings via a centralized preprint server. 2. Main research question/objective: To determine if autonomous LLM agents can collaboratively improve research performance by sharing and building upon each other’s work. 3. Key methodology: Agent laboratories developed reasoning/prompting techniques, uploading/retrieving reports on a shared server, with performance evaluated on benchmarks like MATH-500. 4. Primary results: Agents with access to prior research achieved higher performance improvements (11.4% relative improvement on MATH-500) compared to isolated agents. Multiple labs using the System were able to reach a best performance of 79.8% 5. Principal implication for AI practitioners: AgentRxiv demonstrates a viable path for accelerating AI research through agent collaboration, potentially leading to faster discovery and improved generalization of techniques. |
| MagicComp: Training-free Dual-Phase Refinement for Compositional Video | ||
| Generation (Read more on arXiv or HuggingFace) | Hongyu Zhang, ClownRat, Pengjin, BestWishYsh, dyf | MagicComp is a training-free framework that improves compositional text-to-video generation through dual-phase refinement during conditioning and denoising. The main research objective is to address challenges in compositional video generation, such as attribute binding, spatial relationships, and interactions between multiple subjects, without additional training. The key methodology involves Semantic Anchor Disambiguation (SAD) to resolve inter-subject ambiguity during conditioning, and Dynamic Layout Fusion Attention (DLFA) for spatial-attribute binding during denoising. Results on T2V-CompBench show that MagicComp achieves a Consist-attr score of 0.7665, outperforming the baseline CogVideoX-2B’s score of 0.6775. The principal implication for AI practioners is that MagicComp can be integrated into existing text-to-video architectures to enhance compositional video generation quality without requiring additional training or significant increases in inference time. |
| Vision-R1: Evolving Human-Free Alignment in Large Vision-Language Models | ||
| via Vision-Guided Reinforcement Learning (Read more on arXiv or HuggingFace) | Fan Yang, Hongyin Zhao, Shurong Zheng, Yousong Zhu, Yufei Zhan | Vision-R1 is a vision-guided reinforcement learning algorithm that improves object localization in Large Vision-Language Models (LVLMs) using only curated instruction data. The main research objective is to enhance LVLM capabilities in object localization tasks without relying on human-annotated preference data or specialized reward models. The key methodology involves a criterion-driven reward function based on visual feedback and a progressive rule refinement strategy that dynamically adjusts reward criteria during training. Results show that fine-tuning a 7B LVLM with Vision-R1 achieved up to 50% improvement, and specifically, increased the Average Precision (mAP) on the ODINW-13 benchmark by 9.0 points compared to supervised fine tuning for Qwen2.5-VL-7B. AI practitioners can utilize Vision-R1 to improve object localization performance in LVLMs without the need for costly human-annotated preference data, leading to substantial gains in model accuracy. |
| Reasoning to Learn from Latent Thoughts (Read more on arXiv or HuggingFace) | Tatsunori Hashimoto, cmaddis, nband, ryoungj | This paper introduces “reasoning to learn,” an approach for improving language model (LM) pretraining data efficiency by explicitly modeling and inferring the latent human thoughts underlying text generation. The main research objective is to investigate whether augmenting observed text data with inferred latent thoughts can improve data efficiency in LM pretraining, particularly in a data-constrained regime. The key methodology involves training LMs to jointly model the distribution of observed text and synthesized latent thoughts, using an EM algorithm (BoLT) to iteratively improve latent thought quality and LM capability. Primary results show that a 1.1B LM pretrained with GPT-4o-mini synthesized latent thoughts achieves 25.4% accuracy on MATH, significantly outperforming the 5.74% accuracy achieved by training on raw data alone. For AI practitioners, this implies that incorporating synthesized latent thoughts during pretraining can lead to substantial data efficiency improvements, enabling the development of more capable models with limited data. |
| Defeating Prompt Injections by Design (Read more on arXiv or HuggingFace) | Tianqi Fan, ftramer, carlini, iliashum, dedeswim | CaMeL is a system designed to protect Large Language Model (LLM) agents from prompt injection attacks by enforcing explicit security policies. The main research question is how to design a robust defense that prevents prompt injection attacks in LLM agents interacting with untrusted data, without modifying the underlying model. The key methodology involves extracting control and data flows from user queries, representing them as pseudo-Python code, and enforcing security policies via a custom Python interpreter that tracks provenance and capabilities. The primary results demonstrate that CaMeL solves 67% of tasks with provable security in the AgentDojo benchmark, with some utility degradation on specific task suites, and eliminates almost all prompt injection attacks when combined with capabilities and policy. The principal implication for AI practitioners is that using capability-based security, explicit isolation, and a custom interpreter to manage data and control flows can significantly enhance the security of LLM agent systems against prompt injections, without relying solely on inherent model robustness. |
| Typed-RAG: Type-aware Multi-Aspect Decomposition for Non-Factoid | ||
| Question Answering (Read more on arXiv or HuggingFace) | Yunho Maeng, Hyeonseo Nam, Ahjeong Park, keirahrlee, oneonlee | Typed-RAG is a framework for non-factoid question answering that improves response quality by classifying questions and decomposing multi-aspect queries. The main research objective is to address the limitations of existing retrieval-augmented generation (RAG) systems in handling the complexity and diversity of non-factoid questions (NFQs). The key methodology is Typed-RAG, a type-aware, multi-aspect decomposition approach that integrates question type classification and aspect-based decomposition into the RAG pipeline. Experimental results on the Wiki-NFQA dataset show that Typed-RAG outperforms baselines, achieving a Mean Reciprocal Rank (MRR) of 0.8413 with a GPT-40 mini scorer and Mistral-7B base model configuration. Principal implication is that AI practitioners can create NFQA models by leveraging type-aware and multi-aspect decomposition strategies to create a more comprehensive RAG system. |
| AlphaSpace: Enabling Robotic Actions through Semantic Tokenization and | ||
| Symbolic Reasoning (Read more on arXiv or HuggingFace) | Bui Quang Huy, Dinh Bach Vu, alandao | AlphaSpace enhances spatial reasoning in language models for 3D robotic manipulation using semantic tokenization and symbolic reasoning. The main objective is to improve the ability of language models to perform precise object manipulation in 3D Cartesian space without relying on vision-based embeddings. The key methodology involves a hierarchical semantics-based tokenization strategy that encodes spatial information (including height) and object attributes, combined with synthetic reasoning data for training. AlphaSpace achieves a total accuracy of 66.67% on the EmbodiedBench Manipulation Subtask, significantly outperforming GPT-4o (37.5%) and Claude 3.5 Sonnet (29.17%). AI practitioners can leverage this approach to develop more efficient and accurate robotic control systems that rely less on computationally expensive visual processing and more on structured spatial representations. |
| AMD-Hummingbird: Towards an Efficient Text-to-Video Model (Read more on arXiv or HuggingFace) | Dong Zhou, He Cui, Takashi Isobe, ebarsoum, gemengmeng | AMD-Hummingbird is a lightweight text-to-video (T2V) generation framework that balances computational efficiency with high visual quality. The main research objective is to develop a T2V model suitable for resource-constrained devices by addressing the trade-off between model size and visual fidelity. The key methodology involves a two-stage diffusion model distillation pipeline: first pruning the U-Net architecture and then enhancing visual quality via visual feedback learning, combined with a data processing pipeline using LLMs and VQA models. The primary result is that Hummingbird achieves a 31x speedup compared to VideoCrafter2 and reduces U-Net parameters from 1.4 billion to 0.7 billion, while attaining the highest overall VBench score. For AI practitioners, this provides a practical and efficient solution for T2V generation, combining performance, scalability, and flexibility, especially beneficial for deployment on devices with limited computational resources. |
| Lost in Cultural Translation: Do LLMs Struggle with Math Across Cultural | ||
| Contexts? (Read more on arXiv or HuggingFace) | Bhoomika Lohana, jaswindersingh2, 55mv, Abdul084, abedk | Large Language Models (LLMs) demonstrate reduced mathematical reasoning performance when presented with culturally adapted math word problems, despite the underlying mathematical structure remaining constant. The research investigates whether LLMs’ mathematical reasoning abilities persist across different cultural contexts. Six culturally adapted datasets were synthesized from the GSM8K benchmark by modifying cultural elements (names, foods, places) while preserving mathematical logic. Fourteen LLMs were evaluated, revealing that models performed worse on culturally adapted problems compared to the original GSM8K, with Meta LLaMA 3.1-8B showing the largest accuracy drop (5.9%) on the Somalia dataset. AI practitioners should prioritize diverse and representative training data to improve LLMs’ robustness in real-world applications across various cultural contexts. |
| Variance Control via Weight Rescaling in LLM Pre-training (Read more on arXiv or HuggingFace) | gueraf, nilabhra, akanyaani, louisowen6 | This paper introduces weight initialization and variance control techniques to improve LLM pre-training. The main research objective is to investigate how controlling weight variance, both at initialization and during training, impacts LLM stability and downstream task performance. The key methodology involves proposing Layer Index Rescaling (LIR) for weight initialization and Target Variance Rescaling (TVR) for variance control during training, and evaluating these on a 1B parameter LLaMA model using various benchmarks. Primary results show that the combined use of LIR and TVR improves downstream task performance, with up to a 4.6% increase on common pre-training benchmarks, while also reducing extreme activation values. Principal implication for AI practioners is that managing weight variance using LIR and TVR during LLM pre-training can lead to improved model performance and stability, while mitgating some issues as massive activations. |
| V-Seek: Accelerating LLM Reasoning on Open-hardware Server-class RISC-V | ||
| Platforms (Read more on arXiv or HuggingFace) | Luca Benini, Daniele Jahier Pagliari, Alessio Burrello, Mohamed Amine Ahmdi, Javier J. Poveda Rodrigo | This paper optimizes LLM inference on a many-core RISC-V CPU, achieving significant speedups compared to baseline implementations. The main research objective is to optimize the performance of LLM inference on the Sophon SG2042 RISC-V platform. Key methodologies include developing optimized quantized kernels, choosing a suitable compilation toolchain (Xuantie GCC 10.4 for kernels, Clang 19 for the framework), and optimizing model mapping with NUMA policies. On a DeepSeek R1 Distill Llama 8B model, the authors achieved 4.32 tokens/s for token generation and 6.54 tokens/s for prompt processing, representing speedups of up to 2.9x/3.0x over the baseline. The principal implication is to use, on RISC-V architecture, Clang 19 compiler, to disable NUMA Balancing and activate Memory Interleaving, to improve LLM inference performance. |
| MetaSpatial: Reinforcing 3D Spatial Reasoning in VLMs for the Metaverse (Read more on arXiv or HuggingFace) | Han Liu, zhenyupan | MetaSpatial is an RL-based framework that enhances 3D spatial reasoning in vision-language models (VLMs) for 3D scene generation. The main research objective is to address the lack of internalized 3D spatial reasoning in VLMs and the limitations of supervised fine-tuning for 3D layout generation. The key methodology is a multi-turn reinforcement learning (RL) optimization that uses format detection, physical detection, and rendering-based evaluation to provide reward signals, optimized via Group Relative Policy Optimization (GRPO). Results show that on a Qwen-VL 7B model, MetaSpatial improves format correctness from 0.85 to 0.98 and reduces the object collision rate by 24.5%. For AI practitioners, this provides a method to train VLMs to generate coherent, physically plausible 3D scenes without needing extensive “perfect” layout annotations or manual post-processing. |
| Diffusion-4K: Ultra-High-Resolution Image Synthesis with Latent | ||
| Diffusion Models (Read more on arXiv or HuggingFace) | Junjie Liu, Jinjin Zhang, dihuang, xiefan-guo, qiuyuhuang | Diffusion-4K introduces a framework for direct ultra-high-resolution (4K) image synthesis using latent diffusion models. The main research objective is to enable direct training and generation of 4K images with diffusion models, addressing the lack of a 4K image synthesis benchmark. The key methodology involves a wavelet-based fine-tuning approach for latent diffusion models and the creation of a new benchmark, Aesthetic-4K, including a curated 4K dataset with GPT-40-generated captions. Results show that Diffusion-4K, particularly when powered by models like SD3-2B and Flux-12B, achieves a FID score of 39.49 and the model performs well and improves GLCM Score up to 0.79 on the Aesthetic-Eval@2048 benchmark, outperforming the previous scores. AI practitioners can use Diffusion-4K and the Aesthetic-4K benchmark for training and evaluating models capable of generating high-quality, ultra-high-resolution images with detailed textures and improved text prompt adherence. |
| RDTF: Resource-efficient Dual-mask Training Framework for Multi-frame | ||
| Animated Sticker Generation (Read more on arXiv or HuggingFace) | Yeshuang Zhu, Jiapei Zhang, Ying Deng, Ting Zhang, Zhiqiang Yuan | i) This paper introduces RDTF, a resource-efficient training framework for generating multi-frame animated stickers using a dual-mask approach and curriculum learning. ii) The main research objective is to demonstrate that training a smaller video generation model from scratch with limited data can outperform parameter-efficient tuning of larger models under resource constraints. iii) Key methodologies include a discrete frame generation network with a spatial-temporal interaction layer, a dual-mask data utilization strategy (condition mask and loss mask), and a difficulty-adaptive curriculum learning method. iv) On the I&T->V task, RDTF achieved an FVD of 442.18 and a VQA of 0.502, outperforming methods like I2V-Adapter and SimDA. v) For AI practitioners, RDTF shows that effective data utilization and curriculum strategies can enable smaller models trained from scratch to achieve superior performance in resource-constrained settings, suggesting an alternative to fine-tuning large pre-trained models. |
| Optimized Minimal 3D Gaussian Splatting (Read more on arXiv or HuggingFace) | Jong Hwan Ko, epark, maincold2 | Optimized Minimal 3D Gaussian Splatting (OMG) significantly reduces the storage and computational costs of 3D Gaussian Splatting while maintaining rendering quality. The main objective is to minimize the number of Gaussian primitives and storage requirements for 3D Gaussian Splatting (3DGS) without significantly degrading rendering quality. The key methodology involves using a compact attribute representation with sub-vector quantization, integrating per-Gaussian features with a lightweight neural field, and introducing a local distinctiveness metric for Gaussian pruning. The primary result is that OMG achieves nearly a 50% storage reduction compared to the previous state-of-the-art on the Mip-NeRF 360 dataset, requiring only 4.06 MB while preserving comparable rendering quality. The principal implication for AI practitioners is that they can utilize OMG for real-time, high-fidelity rendering on resource-constrained devices and accelerate training through reduced Gaussians and optimized attribute representation. |
| Verbal Process Supervision Elicits Better Coding Agents (Read more on arXiv or HuggingFace) | Jui-Ming Yao, Cheng-Pong Huang, MarkChenX | CURA, a novel code reasoning agent with verbal process supervision (VPS), enhances code generation performance. The main research objective is to examine if iterative verbal process supervision, combined with an agentic reasoning pipeline like Code Understanding and Reasoning Agent (CURA), improves code generation over baseline models. The key methodology involves a process-supervised reasoning framework called CURA, using VPS to generate verbal reward signals at each reasoning step, incorporating iterative feedback within a code-testing sandbox. The primary result is that CURA with VPS achieved a 3.65% improvement over baseline models on BigCodeBench. For AI practitioners, integrating agentic reasoning with iterative, step-level verbal process supervision offers a new, effective approach for enhancing code generation and software engineering tasks, with a direct, measurable performance improvement. |
Papers for 2025-03-24
| Title | Authors | Summary |
|---|---|---|
| MAPS: A Multi-Agent Framework Based on Big Seven Personality and | ||
| Socratic Guidance for Multimodal Scientific Problem Solving (Read more on arXiv or HuggingFace) | Xinyu Zhang, Zhangqi Wang, Zhiyuan Wang, Qika, VentureZJ | MAPS is a multi-agent framework for multimodal scientific problem-solving, leveraging the Big Seven Personality theory and Socratic questioning to improve reasoning and reflection in AI systems. The main research question is how to leverage and elicit off-the-shelf Multimodal Large Language Models (MLLMs) to address challenging Multimodal Scientific Problems (MSPs). The key methodology involves a multi-agent framework with seven distinct agents, each based on a Big Seven personality trait, using a progressive four-agent solving strategy and a Critic agent for Socratic feedback. The primary results show that MAPS outperforms the current state-of-the-art model by 15.84% across all tasks on the EMMA, Olympiad, and MathVista datasets, and slightly exceeds human expert by 3.58%. The principal implication is that AI practitioners can use this framework to enhance multi-model comprehensive reasoning and provide continous feedback mechanism to improve the accuracy in complex, multimodal scientific problem-solving scenarios. |
| MARS: A Multi-Agent Framework Incorporating Socratic Guidance for | ||
| Automated Prompt Optimization (Read more on arXiv or HuggingFace) | Jun Liu, Haiping Zhu, Zhangqi Wang, Qika, VentureZJ | MARS is a multi-agent framework for automated prompt optimization (APO) that uses Socratic guidance and autonomous planning. The main research objective is to address the limited flexibility of fixed templates and inefficient search in prompt spaces that are present in existing APO methods. The key methodology involves a multi-agent architecture with seven agents, including a Planner, and a Teacher-Critic-Student Socratic dialogue pattern for iterative prompt refinement. Primary results show that MARS outperforms the previous state-of-the-art by 6.04% on general tasks and achieves 85.11% accuracy on 12 general tasks. The use of MARS can help AI practitioners by enabling more efficient and precise prompt refinement, leading to better performance of LLMs across various tasks without needing to create complex meta prompts. |
| RoboFactory: Exploring Embodied Agent Collaboration with Compositional | ||
| Constraints (Read more on arXiv or HuggingFace) | Xiaohong Liu, Zhenfei Yin, Xiufeng Song, FACEONG, IranQin | RoboFactory introduces a framework for generating safe and efficient collaborative data for multi-agent embodied systems using compositional constraints. The main research objective is to address the challenges of multi-agent collaboration in embodied systems by proposing and validating a compositional constraint-based approach. The key methodology involves using a large language model (RoboBrain) to generate sub-goals and textual constraints, constructing constraint interfaces (RoboChecker) to ensure adherence, and generating trajectories using predefined motion primitives. Primary results show that in tasks involving three agents, an average success rate of 20.5% was achieved using diffusion policy with 150 demonstrations, and the use of a “local view” with “separate policy” improves task success rates for the “Food Place” task from 0% to 20% in imitation learning when compared with a “shared policy”. The principal implication for AI practitioners is that they can use RoboFactory’s compositional constraints and automated data collection framework to develop and evaluate multi-agent manipulation systems more efficiently. |
| When Less is Enough: Adaptive Token Reduction for Efficient Image | ||
| Representation (Read more on arXiv or HuggingFace) | Andrey Kuznetsov, Elizaveta Goncharova, Eduard Allakhverdov | This paper introduces an adaptive token reduction method for vision encoders to improve efficiency without compromising performance. The main research objective is to determine if all visual tokens generated by vision encoders are equally valuable, or if some can be discarded to reduce computational costs. The key methodology involves integrating an autoencoder with a Gumbel-Softmax selection mechanism to identify and retain only the most informative visual tokens, based on reconstructability. Primary results show that on OCR-based tasks, over 50% of the visual context can be removed with minimal performance loss using the LLaVA-NeXT model. Principal implication for AI practitioners is that multimodal pruning can be adaptively performed, facilitating scalable and low-overhead inference without requiring additional model fine-tuning. |
| Bridging Continuous and Discrete Tokens for Autoregressive Visual | ||
| Generation (Read more on arXiv or HuggingFace) | Yuanzhi Zhu, Yao Teng, Zhijie Lin, ShuhuaiRen, Epiphqny | TokenBridge bridges continuous and discrete token representations for autoregressive visual generation, achieving high-quality image synthesis with simplified modeling. The main objective is to maintain the representational capacity of continuous tokens while preserving the modeling simplicity of discrete tokens in autoregressive visual generation. The key methodology is post-training quantization of pre-trained continuous VAE features using a dimension-wise quantization strategy, paired with a lightweight autoregressive prediction mechanism for large token spaces. The proposed method achieved an FID score of 1.55 and an IS of 313.3 on ImageNet 256x256, matching state-of-the-art continuous approaches while still using discrete token prediction. AI practitioners can leverage this approach to build high-quality autoregressive visual generation models using standard categorical prediction, bypassing the complexity of continuous distribution modeling, without compromising image quality. |
| OpenVLThinker: An Early Exploration to Complex Vision-Language Reasoning | ||
| via Iterative Self-Improvement (Read more on arXiv or HuggingFace) | Wei Wang, Nanyun Peng, Fan Yin, Hritik Bansal, Yihe Deng | OpenVLThinker explores iteratively improving vision-language reasoning in large vision-language models (LVLMs) through a combination of supervised fine-tuning (SFT) and reinforcement learning (RL). The main research objective is to investigate whether complex reasoning capabilities, similar to those in large language models, can be integrated into LVLMs and improve performance on multimodal reasoning tasks. The key methodology involves iterative SFT and RL, with each iteration’s RL-improved model generating refined SFT datasets for the next round, using distilled reasoning steps from text-only models. Primary results show that OpenVLThinker-7B achieved 70.2% accuracy on MathVista, surpassing the Qwen2.5-VL-7B baseline of 68.5%. Principal implication for AI practioners that is combining SFT with verifiable RL can enhance the multi-step reasoning in LVLMs. |
| Modifying Large Language Model Post-Training for Diverse Creative | ||
| Writing (Read more on arXiv or HuggingFace) | Max Kreminski, Yuqian Sun, Melissa Roemmele, Vishakh Padmakumar, John Joon Young Chung | The paper introduces a post-training approach that modifies large language models (LLMs) to improve output diversity in creative writing while maintaining quality. The primary objective is to enhance LLM output diversity during creative writing tasks by incorporating “deviation” (difference from other outputs for the same prompt) into the training objective. The methodology involves adapting Direct Preference Optimization (DPO) and Odds Ratio Preference Optimization (ORPO) by weighting training instances with the deviation of the winning response. Results showed that a Llama-3.1-8B-based diversified DPO model achieved on-par diversity with a human-created dataset and output quality similar to instruction-tuned models like GPT-4o. AI practitioners can leverage this approach to promote output diversity in creative writing LLMs, balancing diverse and high-quality outputs by incorporating the instance deviation during post-training. |
| ETVA: Evaluation of Text-to-Video Alignment via Fine-grained Question | ||
| Generation and Answering (Read more on arXiv or HuggingFace) | Wei Liu, Peng Zhang, Yuchong Sun, Zhengfeng Lai, Guan123 | ETVA is a new method for evaluating text-to-video alignment using question generation and answering. The main research objective is to develop a more accurate and fine-grained evaluation metric for text-to-video (T2V) alignment than existing methods. The key methodology involves a multi-agent system for generating atomic questions from text prompts using scene graphs and a knowledge-augmented multi-stage reasoning framework for answering questions about generated videos. Primary results show that ETVA achieves a Spearman’s correlation coefficient of 58.47 with human judgment, significantly outperforming existing metrics like VideoScore (31.0). Principal implication is that AI practitioners can use ETVA and its associated benchmark (ETVABench) for more reliable and human-aligned evaluation of text-to-video generation models, focusing improvements on fine-grained semantic alignment. |
| Single Image Iterative Subject-driven Generation and Editing (Read more on arXiv or HuggingFace) | Idan Schwartz, Gal Chechik, yairshp | SISO is a training-free method for personalizing image generation and editing using only a single subject image. The main objective is to develop a method for subject-driven image generation and editing from a single image without requiring encoder pre-training. SISO iteratively optimizes a similarity score between the generated image and the input subject image using pre-trained models like DINO and IR. The method achieved a CMMD score of 0.18 in image generation on a benchmark dataset, improving prompt adherence while maintaining image fidelity compared to baselines. AI practitioners can use SISO as a plug-and-play optimization technique for existing image generators, enabling efficient single-image personalization without extensive training. |
| MathFlow: Enhancing the Perceptual Flow of MLLMs for Visual Mathematical | ||
| Problems (Read more on arXiv or HuggingFace) | Jun Cen, Tao Feng, Yunqiu Xu, Felix Chen, JacobYuan | MathFlow decouples visual mathematical problem-solving in Multimodal Large Language Models (MLLMs) into perception and inference stages, improving performance. The main research objective is to evaluate and enhance MLLMs’ ability to accurately perceive and interpret diagrams in visual mathematical problems. The key methodology involves creating a new benchmark, FlowVerse, to categorize information components, and developing MathFlow, a modular pipeline with a dedicated perception model (MathFlow-P-7B) trained via multi-task pretraining and supervised fine-tuning. Primary results indicate that MathFlow*GPT-4V achieved a 56.7% accuracy on MathVerse’s testmini set, and integrated MathFlow-P-7B yields substantial performance gains with various inference models. For AI practitioners, MathFlow offers a modular problem-solving pipeline that enhances the model’s mathematical problem understanding and solving ability by decoupling the perception and inference process. |
| Enabling Versatile Controls for Video Diffusion Models (Read more on arXiv or HuggingFace) | Jiaxing Yan, Xiaobin Lu, Haoming Qin, Hao Zhou, Xu Zhang | VCtrl is a unified framework for fine-grained control over pre-trained video diffusion models using diverse control signals. The main research objective is to enable precise and flexible spatiotemporal control in text-to-video generation, addressing limitations of existing methods. The key methodology involves a unified control signal encoding pipeline and a sparse residual connection mechanism, integrated with a conditional module, to handle various control signals (Canny edges, segmentation masks, human keypoints) without modifying the base generator. Results demonstrate that, on the Canny-to-Video task, VCtrl-Canny achieves a Canny Matching score of 0.24 and an FVD score of 985.31. For AI practitioners, VCtrl provides a generalizable and efficient way to incorporate diverse user-specified controls into existing video diffusion models, improving controllability and generation quality. |
| When Preferences Diverge: Aligning Diffusion Models with Minority-Aware | ||
| Adaptive DPO (Read more on arXiv or HuggingFace) | Donghao Luo, Kai Hu, Chengming Xu, Chen Liu, Lingfan Zhang | This paper proposes Adaptive-DPO, a novel approach to align diffusion models with human preferences, addressing the challenge of minority samples in preference datasets. The main research question is how to mitigate the detrimental effects of minority preference samples (erroneous annotations and subjective divergences) on diffusion model alignment. The key methodology is a minority-instance-aware metric incorporating intra-annotator confidence and inter-annotator stability, used to adaptively reweight and adjust the DPO loss function. Primary results show that Adaptive-DPO outperforms standard DPO; for example it is found that on SD1.5 with 20% flipped labels, Adaptive-DPO achieves an ImageReward of 0.34, while DPO achieves 0.00. The principal implication for AI practitioners is that incorporating Adaptive-DPO can improve the robustness and effectiveness of preference learning in text-to-image generation tasks, especially in the presence of noisy or subjective preference data. |
| FastCuRL: Curriculum Reinforcement Learning with Progressive Context | ||
| Extension for Efficient Training R1-like Reasoning Models (Read more on arXiv or HuggingFace) | Xuan Luo, Wenjie Yang, Zheng Li, Mao Zheng, Mingyang Song | FASTCURL accelerates reinforcement learning for reasoning models by segmenting training data and progressively extending the context window. The main objective is to improve the training efficiency and performance of R1-like reasoning models, particularly with a 1.5B parameter language model, in tackling complex reasoning tasks. The key methodology, FASTCURL, involves length-aware training data segmentation based on input prompt length and curriculum reinforcement learning with a progressively increasing context window. FASTCURL-1.5B-Preview surpasses DeepScaleR-1.5B-Preview across five benchmark datasets while using only 50% of the training steps. For AI practitioners, FASTCURL demonstrates a practical and efficient strategy of segmenting training dataset, and applying curriculum reinforcement learning to reduce training resources (by 50% in training steps, the paper illustrates) for R1-like large language models. |
| From Head to Tail: Towards Balanced Representation in Large | ||
| Vision-Language Models through Adaptive Data Calibration (Read more on arXiv or HuggingFace) | Yu Cheng, Jiawei Zhou, Xiaoye Qu, hitsmy | Here’s a concise summary of the research paper, adhering to your guidelines: The paper introduces an Adaptive Data Refinement (ADR) framework to address the long-tail data distribution problem in Large Vision-Language Models (LVLMs). The main research objective is to investigate and mitigate the impact of imbalanced training data on the performance of LVLMs. The key methodology involves a two-stage approach: Data Rebalancing (DR), which filters redundant head data, and Data Synthesis (DS), which uses diffusion models to generate scarce tail data. Primary results show that ADR improves the average performance of LLaVA 1.5 by 4.36% across eleven benchmarks without increasing training data volume. Principal implication for AI practitioners is, ADR can be integrated into existing LVLMs to improve their performance on tasks with long-tail data distributions, enhancing robustness and generalization capabilities. |
| PVChat: Personalized Video Chat with One-Shot Learning (Read more on arXiv or HuggingFace) | Yuchen Li, Yumeng Li, Gang Xu, Weilong Yan, Master-Shi | PVChat is a personalized video large language model capable of subject-aware question answering from a single reference video. The main research objective is to develop a ViLLM that can understand and answer questions about specific individuals in videos after learning from only one video of each individual. The key methodology involves a Mixture-of-Heads (MoH) enhanced ViLLM optimized on a synthetically augmented video-QA dataset, using a progressive image-to-video learning strategy, and a ReLU Routing MoH attention mechanism. The primary result is that PVChat achieved an accuracy of 0.901, a BLEU score of 0.562, and a BERTScore of 0.952, outperforming state-of-the-art ViLLMs in personalized feature understanding. For AI practitioners, PVChat offers a framework for building video understanding models that can learn individual-specific information from minimal data, enabling more personalized applications in areas such as smart healthcare and home environments. |
| Generalized Few-shot 3D Point Cloud Segmentation with Vision-Language | ||
| Model (Read more on arXiv or HuggingFace) | Junlin Han, Runjia Li, Yun Liu, Guolei Sun, Zhaochong An | GFS-VL enhances generalized few-shot 3D point cloud segmentation (GFS-PCS) by integrating 3D vision-language models (VLMs) and few-shot samples. The main research objective is to improve the performance of GFS-PCS models in segmenting both base and novel object classes, particularly when limited labeled data is available for novel classes. The key methodology involves using a 3D VLM to generate pseudo-labels for novel classes, filtering these pseudo-labels with few-shot samples for accuracy, adaptively infilling unlabeled regions using a combination of pseudo-label context and few-shot data, and employing a novel-base mix strategy for data augmentation. The primary results show that on the ScanNet200 benchmark, GFS-VL achieves a 28.57% increase in harmonic mean (HM) and a 23.37% increase in mIoU-N over the existing state-of-the-art GFS-PCS methods for the 5-shot setting. The principal implication is that AI practitioners can leverage the combined strengths of 3D VLMs’ open-world knowledge and the precision of few-shot samples to achieve significantly improved segmentation in scenarios where acquiring large labeled datasets for new object classes is impractical. |
| Implicit Bias-Like Patterns in Reasoning Models (Read more on arXiv or HuggingFace) | Calvin K. Lai, l048596 | Reasoning models exhibit processing differences for association-compatible versus incompatible information, similar to human implicit bias. The research examined whether reasoning models show implicit bias-like patterns by expending differential computational effort on association-compatible versus incompatible information. The researchers adapted the Implicit Association Test (IAT) for reasoning models, called RM-IAT, measuring the number of reasoning tokens generated via API calls to OpenAI’s 03-mini model for different association tasks. The model generated significantly more reasoning tokens in the association-incompatible condition than the association-compatible condition in nine of ten RM-IATs; for example, the Instruments/Weapons + Pleasant/Unpleasant RM-IAT generated, on average, 84.29 more tokens in the incompatiable vs. compatiable condition.. AI practitioners should consider that reasoning models may have implicit bias-like patterns that increase computational effort when processing association-incompatible information, impacting efficiency and potentially leading to subtle biases. |
| FFaceNeRF: Few-shot Face Editing in Neural Radiance Fields (Read more on arXiv or HuggingFace) | Junyong Noh, Hangyeul Shin, Chaelin Kim, Kwan Yun | FFaceNeRF is a NeRF-based method for 3D-aware face editing that enables customization with few-shot training on desired mask layouts. The main research objective is to overcome the limitation of existing mask-based 3D face editing methods that rely on pre-trained segmentation masks with fixed layouts. The key methodology involves a geometry adapter with feature injection and latent mixing for tri-plane augmentation (LMTA) to enable adapting to various mask layouts using few training samples. The proposed method achieved an average mIoU of 85.33% for mask generation on a test set, outperforming NeRFFaceEditing’s 81.37%. For AI practitioners, FFaceNeRF facilitates personalized and detailed 3D face editing with limited data, reducing the dependency on extensive, specifically segmented datasets. |
| TaoAvatar: Real-Time Lifelike Full-Body Talking Avatars for Augmented | ||
| Reality via 3D Gaussian Splatting (Read more on arXiv or HuggingFace) | Tiansong Zhou, Zhonghua Jiang, Gaige Wang, Jingchuan Hu, Jianchuan Chen | TaoAvatar generates photorealistic, full-body avatars from multi-view sequences for real-time AR applications. The research objective is to create high-fidelity, lightweight, and drivable full-body talking avatars that can run in real-time on mobile and AR devices. The key methodology combines 3D Gaussian Splatting (3DGS) with a personalized clothed human parametric template (SMPLX++), using a teacher-student framework with non-rigid deformation baking and blend shapes compensation. The primary result is that TaoAvatar achieves state-of-the-art rendering quality, maintaining 90 FPS on high-definition stereo devices like the Apple Vision Pro at 2K resolution. For AI practitioners, TaoAvatar provides a lightweight and efficient approach for representing and rendering lifelike full-body avatars directly deployable to resource-constrained AR environments and mobile devices. |
Papers for 2025-03-21
| Title | Authors | Summary |
|---|---|---|
| One-Step Residual Shifting Diffusion for Image Super-Resolution via | ||
| Distillation (Read more on arXiv or HuggingFace) | agoxandr, skushneryuk, ngushchin, kekchpek, apryc1 | This paper introduces RSD, a distillation method for accelerating diffusion-based super-resolution models, achieving single-step image restoration. The main research objective is to develop a computationally efficient distillation method for ResShift that maintains high perceptual quality while significantly reducing inference time. The key methodology is based on training a student network to produce images such that a fake ResShift model trained on them coincides with the teacher model, incorporating multistep training and additional supervised losses. Primary results show that RSD outperforms the teacher ResShift model and SinSR on RealSR with a MUSIQ score of 69.172 compared to the teacher’s 61.330. Principal implication for AI practitioners is that RSD offers a way to deploy diffusion-based super-resolution models in real-time applications on consumer devices by providing faster inference and lower computational requirements. |
| Stop Overthinking: A Survey on Efficient Reasoning for Large Language | ||
| Models (Read more on arXiv or HuggingFace) | andrewwen, HongyiLiuAI, jy-yuan, JiamuZhang, yangsui | This survey systematically investigates and explores the current progress toward achieving efficient reasoning in Large Language Models (LLMs), particularly addressing the “overthinking phenomenon”. The main research question is how to optimize reasoning length in LLMs while preserving or even enhancing their reasoning capabilities. Key methodologies used include model-based (RL with length reward, SFT with varied-length CoT data), reasoning output-based (latent representation compression, dynamic reasoning), and input prompt-based (prompt-guided, attribute-driven routing) approaches. Primary results across multiple works demonstrate the feasibility of significantly shortening LLM reasoning paths, with one example, O1-Pruner, showing the effectiveness of the Length-Harmonizing reward for shortening CoT length. Principal implication for AI practitioners is that efficient reasoning strategies can substantially reduce computational costs and improve the responsiveness of LLM-based applications without significantly compromising, and sometimes improving accuracy. |
| Unleashing Vecset Diffusion Model for Fast Shape Generation (Read more on arXiv or HuggingFace) | Huiwenshi, wangfuyun, cocacola, qikahh, ZeqiangLai | FlashVDM is a framework for accelerating 3D shape generation using Vecset Diffusion Models (VDMs) by optimizing both diffusion sampling and VAE decoding. The main research objective is to address the slow inference speed of VDMs in generating high-resolution 3D shapes. The key methodology involves Progressive Flow Distillation for diffusion sampling, and a lightning vecset decoder with Adaptive KV Selection, Hierarchical Volume Decoding, and Efficient Network Design for VAE acceleration. Primary results show a 45x speedup in VAE decoding (from 22.33s to 0.491s) and an overall 32x speedup in shape generation, achieving comparable quality to state-of-the-art with significantly reduced inference time. AI practitioners can leverage FlashVDM to enable significantly faster 3D shape generation with VDMs, opening possibilities for real-time interactive applications. |
| Survey on Evaluation of LLM-based Agents (Read more on arXiv or HuggingFace) | Yilun Zhao, Guy Uziel, Lilach Eden, lihaoxin2020, Asaf-Yehudai | This paper provides a comprehensive survey of evaluation methodologies for LLM-based agents across capabilities, applications, and frameworks. The main research objective is to systematically analyze existing benchmarks and frameworks for evaluating LLM-based agents across four critical dimensions: fundamental agent capabilities, application-specific benchmarks, generalist agent benchmarks, and agent evaluation frameworks. The key methodology involves a systematic review and categorization of existing literature, benchmarks, and evaluation methods for LLM-based agents, highlighting emerging trends and research gaps. Primary results include the identification of trends toward more realistic and challenging evaluations (e.g., some top-performing models scoring as low as 2% on complex benchmarks), the continuous updating of “live benchmarks,” and a lack of standardized metrics for cost-efficiency, safety, and granular performance evaluation. A principal implication for AI practitioners is the need to adopt and develop more granular, dynamic, and safety-focused evaluation frameworks to ensure robust and responsible development of LLM-based agents, shifting beyond coarse-grained metrics to include fine-grained trajectory analysis and security aspects. |
| DiffMoE: Dynamic Token Selection for Scalable Diffusion Transformers (Read more on arXiv or HuggingFace) | Mingwu Zheng, Xintao Wang, Haotian Yang, Ziyang Yuan, MingleiShi | DiffMoE introduces a Mixture-of-Experts (MoE) architecture for diffusion transformers that enables dynamic token selection and global token accessibility. The main research objective is to address the limitations of existing MoE approaches in diffusion models, specifically their restricted token accessibility and fixed computational patterns. The key methodology incorporates a batch-level global token pool during training and a capacity predictor for dynamic resource allocation during inference. DiffMoE achieves a state-of-the-art FID score of 2.13 on ImageNet 256x256 class-conditional generation with classifier-free guidance (cfg=1.5), surpassing dense models with 1.5x the number of activated parameters. The principle implication is that AI practitioners can leverage DiffMoE to scale diffusion models more efficiently, achieving superior performance while maintaining computational efficiency compared to dense models and previous MoE implementations. |
| Scale-wise Distillation of Diffusion Models (Read more on arXiv or HuggingFace) | Dmitry Baranchuk, Artem Babenko, Denis Kuznedelev, Nikita Starodubcev | Scale-wise Distillation (SWD) is a novel method that improves diffusion model efficiency by progressively increasing spatial resolution during sampling. The paper’s main objective is to investigate whether generating images scale-by-scale across the diffusion process can improve the efficiency of diffusion distillation methods. The key methodology involves integrating a scale-wise generation approach into existing diffusion distillation frameworks, specifically DMD2, and introducing a patch distribution matching (PDM) loss. A primary result is that, within SD3.5 medium, the 6-step scale-wise configuration achieves a FID score of 23.0 on COCO 2014, while its full-scale 6-step counterpart reaches 20.4. AI practitioners can leverage SWD to achieve a balance between generation speed and quality in diffusion models, offering a practical technique to accelerate inference by operating at lower resolutions during initial sampling steps. |
| Cosmos-Reason1: From Physical Common Sense To Embodied Reasoning (Read more on arXiv or HuggingFace) | Hannah Brandon, Alisson Azzolini, NVIDIA, zhuoliny, fferroni | Cosmos-Reason1 is a family of multimodal large language models developed by NVIDIA, trained to integrate physical common sense and embodied reasoning. The main research objective is to develop models capable of understanding the physical world and generating appropriate embodied decisions using natural language through long chain-of-thought reasoning. The key methodology involves defining ontologies for physical common sense and embodied reasoning, curating datasets based on these ontologies, and training models in four stages: vision pre-training, general supervised fine-tuning (SFT), Physical AI SFT, and Physical AI reinforcement learning (RL). Evaluation results show that the Cosmos-Reason1-56B model achieves 60.2% accuracy on the physical common sense benchmark, and Physical AI RL improves performance across most benchmark components. For AI practitioners, using Physical AI SFT and RL, this work will make the code and models open-source to expedite the progress of building Physical AI systems that understand and perform complex tasks. |
| MathFusion: Enhancing Mathematic Problem-solving of LLM through | ||
| Instruction Fusion (Read more on arXiv or HuggingFace) | Honglin Lin, Yu Li, Zhuoshi Pan, Lijun Wu, Qizhi Pei | MathFusion enhances LLM mathematical problem-solving by synthesizing new training instructions from existing problem pairs. The main research objective is to improve LLMs’ mathematical reasoning capabilities through cross-problem instruction synthesis, overcoming limitations of instance-level data augmentation. The key methodology, MathFusion, employs three fusion strategies—sequential, parallel, and conditional—to combine existing mathematical problems into new, more complex ones. Experiments using DeepSeekMath-7B, Mistral-7B, and Llama3-8B show that MathFusion increases accuracy by 18.0 points on average across diverse benchmarks with only 45K additional synthetic instructions. The principal implication is that AI practitioners can improve mathematical reasoning performance in LLMs efficiently using this data synthesis technique. |
| InfiniteYou: Flexible Photo Recrafting While Preserving Your Identity (Read more on arXiv or HuggingFace) | Hao Kang, Zichuan Liu, Yumin Jia, Qing Yan, Liming Jiang | InfiniteYou (InfU) is a Diffusion Transformer (DiT)-based framework for identity-preserved image generation that recrafts photos using text descriptions while maintaining facial identity. The main research objective is to address limitations of existing methods, such as insufficient identity similarity, poor text-image alignment, and low generation quality when using DiTs. The key methodology involves InfuseNet, a generalization of ControlNet, which injects identity features into the DiT base model via residual connections, combined with a multi-stage training strategy using synthetic single-person-multiple-sample (SPMS) data. Primary results showed that InfU achieved a lower ID Loss (0.209) compared to PuLID-FLUX (0.225) and FLUX.1-dev IPA (0.772), while also achieves the highest CLIPScore and PickScore. A principal implication for AI practitioners is that they can utilize InfU’s plug-and-play design, as well as the method of residual feature connections demonstrated, to create high-fidelity and text-aligned identity-preserved images, and extend use cases beyond those presented in the paper. |
| Plug-and-Play 1.x-Bit KV Cache Quantization for Video Large Language | ||
| Models (Read more on arXiv or HuggingFace) | Huan Wang, Can Qin, Yang Sui, Haoxuan You, KD-TAO | VidKV, a plug-and-play KV cache quantization method, compresses the KV cache in Video Large Language Models (VideoLLMs) to 1.x-bit precision with minimal performance loss. The main research question is how to effectively quantize the KV cache in VideoLLMs to lower than 2 bits while preserving model performance. The key methodology involves mixed-precision quantization for the key cache (2-bit for anomalous channels, 1-bit with FFT for normal channels) and 1.58-bit quantization with optional token protection for the value cache, applied per-channel. Primary results show that VidKV compresses the KV cache to 1.5-bit and 1.58-bit precision on LLaVA-OV-7B and Qwen2.5-VL-7B, achieving a VideoChat-GPT average score of 3.06 and 3.00 respectively, which is a close to no loss to the FP16 counterparts. The principal implication for AI practitioners is that they can significantly reduce the memory footprint and computational cost of VideoLLM inference using VidKV, enabling efficient deployment of these models. |
| JARVIS-VLA: Post-Training Large-Scale Vision Language Models to Play | ||
| Visual Games with Keyboards and Mouse (Read more on arXiv or HuggingFace) | Yitao Liang, Xiaojian Ma, Kaichen He, Zihao Wang, Muyao Li | JARVIS-VLA introduces a new training paradigm, ActVLP, that enhances vision-language-action (VLA) models for decision-making in open-world environments like Minecraft. The main research objective is to investigate whether integrating visual-language tasks into the post-training phase of VLA models improves their performance. The key methodology, ActVLP, involves a three-stage training pipeline: post-training language models on text-only world knowledge, post-training both vision encoder and language models on multimodal vision-language alignment and spatial grounding datasets, then post-training language models on multimodal instruction following datasets. The primary result is that post-training on non-trajectory tasks leads to a 40% improvement over the best agent baseline in Minecraft on a diverse set of atomic tasks. For AI practitioners, this demonstrates that incorporating visual-language post-training significantly improves VLA model performance in complex decision-making tasks, offering a new, effective training approach. |
| CaKE: Circuit-aware Editing Enables Generalizable Knowledge Learners (Read more on arXiv or HuggingFace) | Shumin Deng, Jia-Chen Gu, Jizhan Fang, Yunzhi Yao, Ningyu | CaKE improves the generalization of knowledge editing in large language models by aligning edits with the models’ reasoning circuits. The main research objective is to address the poor performance of existing knowledge editing (KE) methods on downstream reasoning tasks involving updated knowledge. The key methodology, CaKE, involves generating circuit-aware training data that explicitly requires reasoning with updated knowledge and training the model to construct robust reasoning circuits integrating the new information. Experimental results show CaKE improves multi-hop reasoning accuracy on the MQUAKE dataset by an average of 20% compared to existing KE methods. AI practitioners can use CaKE to create language models that not only store updated facts but also effectively apply this knowledge in downstream reasoning tasks, improving generalizability. |
| Ultra-Resolution Adaptation with Ease (Read more on arXiv or HuggingFace) | Xinchao Wang, Zhenxiong Tan, Songhua Liu, Ruonan Yu | URAE facilitates adapting text-to-image diffusion models to ultra-high resolutions with limited data and computation. The main research objective is to identify efficient guidelines for adapting existing text-to-image models to ultra-high resolutions (2K and 4K) when training data and computational resources are limited. The key methodology involves theoretically and empirically investigating data efficiency (using synthetic data from teacher models) and parameter efficiency (tuning minor components of weight matrices), alongside examining the impact of classifier-free guidance. Primary results include that URAE achieves comparable 2K generation performance to FLUX1.1 [Pro] Ultra with only 3K samples and 2K iterations, while setting new benchmarks for 4K resolution generation. The principal implication for AI practitioners is that they can adapt diffusion models to ultra-high resolutions efficiently by using synthetic data when available, tuning minor weight matrix components, and disabling classifier-free guidance during adaptation. |
| Expert Race: A Flexible Routing Strategy for Scaling Diffusion | ||
| Transformer with Mixture of Experts (Read more on arXiv or HuggingFace) | Xun Zhou, Defa Zhu, Ziyu Wang, FetchFortune, yyk-wew | Race-DiT introduces a flexible routing strategy for scaling diffusion transformers with Mixture of Experts (MoE). The main research objective is to enhance the scalability and performance of diffusion transformers by integrating MoE methods with a new routing strategy called Expert Race. The key methodology involves allowing tokens and experts to compete and selecting the top candidates, along with per-layer regularization and router similarity loss. The primary result is that Race-DiT achieves a 7.2x speedup in iterations when reaching the same training loss compared to DiT-XL, with an equal number of activated parameters. Principal implication for AI practioners is that it provides a method to improve performance gains and scaling in diffusion models while maintaining good expert utilization, with superior ImageNet validation and image quality. |
| MagicMotion: Controllable Video Generation with Dense-to-Sparse | ||
| Trajectory Guidance (Read more on arXiv or HuggingFace) | Qi Dai, Hui Zhang, Rui Wang, Zhen Xing, quanhaol | MagicMotion is a novel image-to-video generation framework that enables trajectory control through three levels of conditions (masks, bounding boxes, and sparse boxes). The main objective is to develop a trajectory-controllable video generation model that overcomes limitations of existing methods, such as imprecise trajectory adherence and compromised visual quality, and supports multiple trajectory control formats. The key methodology involves a progressive training strategy using a Trajectory ControlNet architecture (similar to ControlNet) to inject trajectory conditions into a diffusion model, alongside a novel latent segment loss. The primary results demonstrate that MagicMotion outperforms previous methods on the MagicBench benchmark, achieving a Mask_IoU of 91.57% and a Box_IoU of 87.75% in Stage 1, and Mask_IoU=76.61 and Box_IoU=81.45 in Stage 2. AI practitioners can use MagicMotion for improved controllable video generation, allowing more precise control over object motion and facilitating the creation of high-quality videos with user-specified trajectories. |
| M3: 3D-Spatial MultiModal Memory (Read more on arXiv or HuggingFace) | Jianglong Ye, Xuanbin Peng, Ri-Zhao Qiu, Yuchen Song, Xueyan Zou | M3 is a multimodal memory system that integrates 3D Gaussian Splatting with foundation models to store and render multimodal representations of medium-sized static scenes. The main research objective is to develop a spatial memory system that efficiently stores and retrieves multi-granularity information about static scenes from video sources, addressing computational constraints and information loss in existing feature splatting methods. The key methodology involves storing high-dimensional feature maps from foundation models in a memory bank (principal scene components) and using low-dimensional queries from 3D Gaussians as indices, applying Gaussian memory attention to render foundation model embeddings. The primary results show that M3 outperforms previous methods in feature similarity and downstream tasks; for example, M3 achieved a cosine similarity of 0.6074 on the Playroom dataset using CLIP, compared to 0.4867 for F-Splat. For AI practitioners, M3 provides a more effective framework to integrate foundation models with 3D scene representations, enabling efficient memorization and query of visual and semantic information in spatial contexts. |
| Why Do Multi-Agent LLM Systems Fail? (Read more on arXiv or HuggingFace) | Bhavya Chopra, Lakshya A. Agrawal, Shuyi Yang, Melissa Z. Pan, Mert Cemri | This paper presents a comprehensive study of failure modes in Multi-Agent Systems (MAS) powered by Large Language Models (LLMs). The main research question is: Why do Multi-Agent LLM Systems fail, and what is the taxonomy of these failure modes? The key methodology involves grounded theory analysis of 150+ conversation traces from five popular MAS frameworks, with human expert annotation and iterative refinement to establish a failure taxonomy. The primary result is a taxonomy (MASFT) of 14 failure modes grouped into 3 categories, with the “Poor Specification” category appearing in 37.17% of analyzed traces. AI practitioners should use this taxonomy to identify and mitigate failures in MAS designs, focusing on enhanced specification, inter-agent coordination, and task verification, rather than relying solely on base LLM improvements. |
| 1000+ FPS 4D Gaussian Splatting for Dynamic Scene Rendering (Read more on arXiv or HuggingFace) | Xinchao Wang, Xingyi Yang, Qiuhong Shen, nopyyh | 4DGS-1K achieves over 1000 FPS in dynamic scene rendering by addressing temporal redundancy in 4D Gaussian Splatting. The main research objective is to reduce the storage requirements and improve the rendering speed of 4D Gaussian Splatting (4DGS) for dynamic scenes. The key methodology involves a two-step pruning approach: first, pruning short-lifespan Gaussians using a spatial-temporal variation score, and second, filtering inactive Gaussians using a key-frame based temporal filter. The method achieves a 41x reduction in storage and 9x faster rasterization speed compared to vanilla 4DGS on complex dynamic scenes, while maintaining comparable visual quality. For AI practitioners, this implies that they can render high-fidelity, complex dynamic scenes, in real-time with significantly less storage requirements through the implementation of temporal-aware filtering and pruning. |
| XAttention: Block Sparse Attention with Antidiagonal Scoring (Read more on arXiv or HuggingFace) | Song Han, Junxian Guo, Guangxuan Xiao, Ruyi Xu, songhan | XAttention is a plug-and-play framework that accelerates long-context Transformer inference by using block-sparse attention based on antidiagonal scoring. The paper’s main research question is: Can a block-sparse attention mechanism be designed to accelerate long-context Transformers without accuracy loss? XAttention’s methodology sums antidiagonal values in the attention matrix to estimate block importance, enabling selective computation. Evaluations on language and video benchmarks show XAttention achieves comparable accuracy to full attention, with up to 13.5x acceleration in attention computation during pre-filling. This suggests AI practitioners can deploy more efficient long-context Transformer models in real-world applications by adopting XAttention to reduce computational costs. |
| Uni-3DAR: Unified 3D Generation and Understanding via Autoregression on | ||
| Compressed Spatial Tokens (Read more on arXiv or HuggingFace) | Zhifeng Gao, Lin Yao, Haowei Lin, Shuqi Lu, guolinke | Uni-3DAR is a unified framework for 3D structural generation and understanding that uses autoregressive prediction on compressed spatial tokens. The main research objective is to develop a unified framework that seamlessly integrates 3D generation and understanding (3D GU) tasks via autoregressive prediction. The key methodology involves a hierarchical tokenization using an octree to compress 3D space, a two-level subtree compression strategy, and a masked next-token prediction mechanism. Primary results show that Uni-3DAR surpasses previous state-of-the-art diffusion models on microscopic 3D GU tasks, achieving up to 256% relative improvement on PXRD-guided crystal structure prediction and up to 21.8x faster inference speeds. AI practitioners can use Uni-3DAR as a more efficient and versatile framework for unifying diverse 3D GU tasks, potentially leading to faster and more accurate models in areas like materials science and drug discovery. |
| CLS-RL: Image Classification with Rule-Based Reinforcement Learning (Read more on arXiv or HuggingFace) | Kaipeng Zhang, Jike Zhong, Ming Li, yuxianglai117, stzhao | This paper introduces CLS-RL, a rule-based reinforcement learning approach for fine-tuning Multimodal Large Language Models (MLLMs) for image classification, demonstrating improved performance and generalization compared to supervised fine-tuning. The main research objective is to explore few-shot MLLM classification fine-tuning and address catastrophic forgetting issues observed with supervised fine-tuning (SFT). The key methodology involves using verifiable signals (class names) as rewards to fine-tune MLLMs and formatting the reward to encourage “thinking” before answering, and comparing the proposed method to No-Thinking-CLS-RL. The primary results show CLS-RL outperforms SFT in most of 11 datasets, with a base-to-new generalization setting achieving 81.17% accuracy on base classes and 79.15% on new classes for CLS-RL, compared to 67.4% and 70.73% for SFT. For AI practitioners, using rule-based reinforcement learning for fine-tuning MLLMs can lead to improved image classification performance and better generalization to new classes, even with limited labeled data. |
| LHM: Large Animatable Human Reconstruction Model from a Single Image in | ||
| Seconds (Read more on arXiv or HuggingFace) | Weichao Shen, Peihao Li, Xiaodong Gu, Lingteng Qiu, DyrusQZ | LHM is a feed-forward transformer model that generates animatable 3D human avatars from single images in seconds. The main objective is to create a generalizable model for high-fidelity 3D human reconstruction from a single image that supports real-time rendering and animation. The method utilizes a multimodal transformer architecture with a head feature pyramid encoding scheme to fuse 3D point features and 2D image features and represents the avatar as 3D Gaussian splatting. Trained on a large-scale video dataset, LHM achieves a PSNR of 25.183 on synthetic data, outperforming existing methods. For AI practitioners, LHM offers an efficient solution for generating animatable 3D human models from single images, reducing reliance on extensive optimization or post-processing. |
| Zero-1-to-A: Zero-Shot One Image to Animatable Head Avatars Using Video | ||
| Diffusion (Read more on arXiv or HuggingFace) | Chua Tat-Seng, Fan Hehe, Ma Fan, zhenglin | Zero-1-to-A is a method for generating animatable 4D head avatars from a single image using video diffusion models. The main research objective is to generate high-fidelity 4D head avatars from a single image input, overcoming the spatial and temporal inconsistencies of video diffusion models. The key methodology, Zero-1-to-A, employs Symbiotic GENeration (SymGEN) to iteratively construct a consistent video dataset and optimize the avatar, alongside a Progressive Learning strategy that separates spatial and temporal learning. Results show that Zero-1-to-A achieves an average CLIP score of 0.285 (ViT-L/14) and 0.322(ViT-B/32), and improves ID consistency and rendering speed compared to prior methods. AI practitioners can leverage this method for efficient and data-sparse creation of high-fidelity, animatable head avatars from single images, eliminating the need for extensive training data. |
| Towards Unified Latent Space for 3D Molecular Latent Diffusion Modeling (Read more on arXiv or HuggingFace) | Kenji Kawaguchi, Sihang Li, Yi Zhao, Zhiyuan Liu, Yanchen Luo | The paper introduces UAE-3D and UDM-3D, a VAE and latent diffusion model, for 3D molecule generation using a unified latent space. The main research question is whether a unified generative model can seamlessly integrate all modalities of 3D molecule generation (atom types, bonds, 3D coordinates). The key methodology is a multi-modal VAE (UAE-3D) that compresses 3D molecules into a unified latent space, using a Relational Transformer encoder and SE(3) augmentations, combined with a Diffusion Transformer (DiT) for latent diffusion modeling. The results show that UDM-3D achieves 100.0% atom and bond accuracy and 0.0002 coordinate RMSD in reconstruction, and 9.89E-03 bond length distribution in GEOM-Drugs in comparison with the second-best result of 3.91E-01. For AI practitioners, this offers a way to generate 3D molecules with improved efficiency and accuracy by leveraging a unified latent space, simplifying the complexities of handling multi-modality and equivariance. |
| Tokenize Image as a Set (Read more on arXiv or HuggingFace) | Shuyang Gu, Han Hu, Mengde Xu, Zigang Geng | This paper introduces TokenSet, a new image generation paradigm using set-based tokenization and distribution modeling to improve context aggregation and robustness. The main research objective is to develop a more effective image representation that dynamically allocates coding capacity based on regional semantic complexity, unlike fixed-position latent codes. The key methodology involves representing images as unordered token sets, using a dual transformation to convert sets into fixed-length sequences, and applying a novel Fixed-Sum Discrete Diffusion model for distribution modeling. Primary results show that the TokenSet achieves a reconstruction rFID of 2.74 on ImageNet, with an token overlap of 87.6% after adding a level-10 Gaussian noise (Signal-to-Noise Ratio (dB)), which is a superior performance as compared to prior state of arts. AI practitioners can use TokenSet’s representation and modeling approach to create image generation models that better capture global context and exhibit robustness to image perturbations for a variety of computer vision applications. |
| NuiScene: Exploring Efficient Generation of Unbounded Outdoor Scenes (Read more on arXiv or HuggingFace) | Angel X. Chang, Qinghong Han, rexleeppp | NuiScene explores efficient generation of unbounded outdoor scenes using a novel vector set representation and explicit outpainting. The main research objective is to develop an efficient method for generating large, unbounded outdoor scenes with varying heights and diverse styles. The key methodology involves compressing scene chunks into uniform vector sets using 3DShape2VecSet, training an explicit outpainting diffusion model for unbounded generation, and curating a dataset (NuiScene43) of 43 scenes with unified scales and cleaned ground geometries. The vector set diffusion model achieves an FPD score of 0.571 and KPD score of 0.951, outperforming the triplane baseline. For AI practitioners, this method provides a more efficient approach for representing and generating unbounded 3D outdoor scenes compared to methods using spatially structured latents. |
| Fin-R1: A Large Language Model for Financial Reasoning through | ||
| Reinforcement Learning (Read more on arXiv or HuggingFace) | Jinyi Niu, Lingfeng Zeng, Fangqi Lou, Xin Guo, Zhaowei Liu | Fin-R1 is a 7-billion parameter large language model designed specifically for financial reasoning, addressing data fragmentation, reasoning uncontrollability, and generalization challenges. The main research objective was to develop a model that can effectively handle complex financial problems and improve performance in financial reasoning tasks. The key methodology involved constructing a high-quality dataset (Fin-R1-Data) with 60,091 chain-of-thought entries, followed by Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL) using Group Relative Policy Optimization (GRPO). Fin-R1 achieved an average score of 75.2 across multiple financial benchmarks, outperforming other similar-sized models and ranking second overall. The principal implication is that AI practitioners can leverage Fin-R1’s two-stage training framework and specialized dataset to build more accurate and interpretable decision-making tools for financial AI applications, particularly in areas like compliance and robo-advisory. |
| SALT: Singular Value Adaptation with Low-Rank Transformation (Read more on arXiv or HuggingFace) | Mohammad Yaqub, Hu Wang, Mohammed Elseiagy, Abdelrahman Elsayed, Sarim-Hash | SALT is a parameter-efficient fine-tuning method for adapting the Segment Anything Model (SAM) to medical image segmentation. The main research objective is to develop a method that effectively adapts foundation models to the medical domain while minimizing trainable parameters and preserving pre-trained knowledge. The key methodology, SALT, combines SVD-based adaptation of dominant singular values with low-rank updates for the remaining subspace, using trainable scale, shift, and low-rank matrices. SALT outperformed state-of-the-art PEFT methods (LoRA and SVD) by 2% to 5% in Dice score on five medical datasets, with only 3.9% trainable parameters. AI practitioners can use SALT for efficient and robust adaptation of large foundation models to specialized domains like medical imaging, achieving high accuracy with significantly reduced computational overhead compared to full fine-tuning or other PEFT methods. |
| MotionStreamer: Streaming Motion Generation via Diffusion-based | ||
| Autoregressive Model in Causal Latent Space (Read more on arXiv or HuggingFace) | Liang Pan, Ke Fan, Huaijin Pi, Shunlin Lu, lxxiao | MotionStreamer is a framework for text-conditioned streaming motion generation that uses a diffusion-based autoregressive model in a causal latent space. The main research objective is to address the challenge of generating human motion sequences incrementally while dynamically adapting to online text inputs and maintaining semantic coherence. The key methodology involves incorporating a continuous causal latent space into a probabilistic autoregressive model with a diffusion head, utilizing a Causal Temporal AutoEncoder (TAE) for motion compression and online decoding, and employing Two-Forward and Mixed training strategies. The method achieves a Frechet Inception Distance (FID) of 10.724 on the HumanML3D test set, outperforming existing approaches. For AI practioners, MotionStreamer provides an effective model to generate realistic and diverse human motions that directly respond to progressive input text prompts, with low latency. |
| Make Your Training Flexible: Towards Deployment-Efficient Video Models (Read more on arXiv or HuggingFace) | Yi Wang, Xiangyu Zeng, Tianxiang Jiang, Kunchang Li, Chenting Wang | FluxViT enhances video model efficiency by optimizing input token selection and sampling for varied computational budgets. The main research question is how to maximize input information across budgets, addressing sub-optimal accuracy-computation trade-offs in video models. The key methodology, termed Flux, uses flexible video sampling and token selection, integrated with a masked alignment strategy in a teacher-student training framework. FluxViT-S outperforms InternVideo2-S by 2.2% on K400 with standard computation and achieves comparable performance with only 10% of the inference cost. AI practitioners can leverage Flux for training robust video models adaptable to diverse deployment scenarios, achieving state-of-the-art performance with significantly reduced computational requirements. |
| MagicID: Hybrid Preference Optimization for ID-Consistent and | ||
| Dynamic-Preserved Video Customization (Read more on arXiv or HuggingFace) | Hongwei Yi, Tianyang Wang, Xi Xiao, Lifan Jiang, Hengjia Li | MagicID is a framework for generating personalized videos that maintain consistent identity and exhibit natural dynamics based on user-provided reference images. The main research objective is to address identity degradation and reduced dynamics in customized video generation caused by reliance on self-reconstruction training with static images. The key methodology involves constructing pairwise preference video data with explicit identity and dynamic rewards, and a hybrid sampling strategy that prioritizes identity preservation and then enhances dynamic motion. The primary results show MagicID achieves a mean identity similarity score of 0.600, outperforming existing methods while preserving motion dynamics. The principal implication for AI practitioners is that using hybrid preference optimization with tailored rewards can improve the quality of identity-preserved video customization, enabling more realistic and personalized video generation. |
| Reinforcement Learning for Reasoning in Small LLMs: What Works and What | ||
| Doesn’t (Read more on arXiv or HuggingFace) | Chris Ngo, quyanh | This study investigates reinforcement learning (RL) for improving reasoning in small language models (LLMs) under resource constraints. The main research question is how small LLMs behave when fine-tuned with RL under strict computational and time limitations, and whether their reasoning performance can be improved using an RL approach similar to DeepSeek-R1. The key methodology involves adapting the Group Relative Policy Optimization (GRPO) algorithm and curating a compact, high-quality mathematical reasoning dataset, then training a 1.5-billion-parameter model (DeepSeek-R1-Distill-Qwen-1.5B) on 4 GPUs within 24 hours. A primary result is that the model achieved an AIME24 score of 46.7% with only 7,000 training samples and a $42 training cost, surpassing the o1-preview model. This implies AI practitioners can achieve substantial reasoning gains in small LLMs using RL with limited data and computational resources, offering a cost-effective alternative to large-scale approaches. |
| Improving Autoregressive Image Generation through Coarse-to-Fine Token | ||
| Prediction (Read more on arXiv or HuggingFace) | Michael Qizhe Shieh, Kaipeng Zhang, Ziyao Guo | This paper introduces a coarse-to-fine framework for autoregressive image generation that alleviates vocabulary redundancy in large codebooks. The main research objective is to maintain the benefits of large codebooks for high-quality image reconstruction while simplifying the autoregressive modeling task. The key methodology involves clustering similar VQ-VAE codebook tokens into coarse labels, predicting coarse labels autoregressively, and then predicting fine-grained tokens in parallel using full attention. The primary results include an average improvement of 59 points in Inception Score compared to baselines, reduced FID, and faster sampling speeds despite adding an auxiliary network. For AI practitioners, this method allows more efficient autoregressive image generation by reducing the effective vocabulary size, facilitating faster training and improved image quality when using large codebooks. |
| Deceptive Humor: A Synthetic Multilingual Benchmark Dataset for Bridging | ||
| Fabricated Claims with Humorous Content (Read more on arXiv or HuggingFace) | Sunil Saumya, Shankar Biradar, UVSKKR | Here’s a concise summary of the research paper, adhering strictly to your guidelines: This paper introduces the Deceptive Humor Dataset (DHD), a new synthetic multilingual benchmark for studying humor derived from fabricated claims and misinformation. The main research objective is to establish a structured foundation for analyzing humor in deceptive contexts and to understand how humor influences the perception and spread of misinformation. The key methodology involves generating 9,000 humor-infused comments using ChatGPT-4o, labeled with satire levels (1-3) and humor attributes (Irony, Absurdity, Social Commentary, Dark Humor, Wordplay) across multiple languages and code-mixed variants. Primary results show that mBART achieved the best performance for Satire Level Classification with an accuracy of 51.00%, while BERT performed best on Humor Attribute Classification with an accuracy of 40.44%. The principal implication for AI practitioners is the availability of a structured dataset and established baselines to benchmark and advance deceptive humor detection models, a critical aspect in mitigating the spread of harmful narratives. |
| VideoRFSplat: Direct Scene-Level Text-to-3D Gaussian Splatting | ||
| Generation with Flexible Pose and Multi-View Joint Modeling (Read more on arXiv or HuggingFace) | Hyungjin Chung, Byung-Hoon Kim, Hyelin Nam, Byeongjun Park, Hyojun Go | VideoRFSplat is a text-to-3D Gaussian Splatting model that generates real-world scenes with flexible camera poses and multi-view image consistency, eliminating the need for per-scene optimization or external refinement models. The main objective is to develop a direct text-to-3D generation model capable of handling diverse camera poses and unbounded scenes without relying on score distillation sampling (SDS) refinement. The methodology utilizes a dual-stream architecture with a video generation model and a side-attached pose generation model, communicating via cross-attention and employing an asynchronous sampling strategy. The primary result is that VideoRFSplat achieves a FID of 30.33 and CLIP score of 33.0 on MVImgNet, outperforming existing direct text-to-3D methods that use SDS refinement. The principal implication is that AI practitioners can directly generate realistic and coherent 3D scenes from text prompts without needing post-hoc refinement, simplifying the 3D generation pipeline and potentially improving efficiency. |
| Sonata: Self-Supervised Learning of Reliable Point Representations (Read more on arXiv or HuggingFace) | Chris Xie, Tianwei Shen, Duncan Frost, Daniel DeTone, Xiaoyang Wu | Sonata is a self-supervised learning framework for 3D point cloud representations that addresses limitations of existing approaches. The main research question is whether a reliable self-supervised point cloud model can be developed for diverse 3D tasks via simple linear probing, even with limited data. The key methodology involves a point self-distillation framework that obscures spatial information and emphasizes input features, training on 140k point cloud scenes. A primary result is that Sonata triples linear probing accuracy on ScanNet semantic segmentation compared to previous methods, achieving 72.5% mIoU with less than 0.2% learnable parameters. The principal implication is that AI practitioners can leverage Sonata as a reliable foundation model for various 3D perception tasks, achieving strong performance and data efficiency, even with limited labeled data, by using it as initialization and then employing simple linear probing. |
| BigO(Bench) – Can LLMs Generate Code with Controlled Time and Space | ||
| Complexity? (Read more on arXiv or HuggingFace) | Gabriel Synnaeve, Benoit Sagot, Baptiste Roziere, pierrechambon | BIGO(BENCH) is a new benchmark for evaluating the ability of large language models (LLMs) to generate code with specified time and space complexity constraints. The main objective is to assess LLMs’ capacity to understand and control computational complexity in code generation. The methodology involves a dynamic complexity inference framework to analyze Python functions, a dataset of 3,105 coding problems and 1,190,250 solutions with inferred complexity labels, and evaluations of LLMs on complexity prediction, generation, and coefficient ranking. The results show that DEEPSEEK-R1 LLAMA 70B achieved 4.8% and 3.4% All@1 on time and space complexity generation, respectively, revealing challenges in handling complexity requirements. The main implication for AI practitioners is that while LLMs show proficiency in program synthesis, controlling and reasoning about time and space complexity remains a significant challenge, indicating a need to improve models on abstract thinking about code. |
| See-Saw Modality Balance: See Gradient, and Sew Impaired Vision-Language | ||
| Balance to Mitigate Dominant Modality Bias (Read more on arXiv or HuggingFace) | YoungBin Kim, Juhwan Choi, Eunju Lee, MiHyeon Kim, JuneHyoung Kwon | Vision-language (VL) models exhibit a “dominant modality bias,” disproportionately relying on one modality, which BALGRAD mitigates by reweighting and projecting gradients. The research analyzes model behavior under dominant modality bias, showing how unaligned gradients and differences in gradient magnitudes hinder balanced loss convergence. The proposed BALGRAD framework employs inter-modality gradient reweighting (adjusting KL divergence gradient based on modality contribution) and inter-task gradient projection. Experiments on UPMC Food-101, Hateful Memes, and MM-IMDb datasets demonstrate BALGRAD’s effectiveness; on UPMC Food-101, BALGRAD improved performance on the weak (text) modality by 12.5%p compared to the baseline. AI practitioners can use BALGRAD to create more robust VL models that effectively utilize both modalities, even when one is impaired, reducing reliance on a single dominant modality. |
| AIMI: Leveraging Future Knowledge and Personalization in Sparse Event | ||
| Forecasting for Treatment Adherence (Read more on arXiv or HuggingFace) | Hassan Ghasemzadeh, Diane J. Cook, ab9mamun | AIMI, a knowledge-guided system, forecasts medication adherence by leveraging sensor data, medication history, and future knowledge. The main research objective was to determine the impact of future knowledge and personalization on the accuracy of sparse event forecasting for treatment adherence. The key methodology involved training and evaluating CNN and LSTM models with various combinations of input features, including sensor data, adherence history, and “future knowledge” (prescribed medication times), along with an incremental learning algorithm. The LSTM models achieved an accuracy of 0.932 and an F-1 score of 0.936, and leveraging future knowledge improved the F-1 score by almost 112% when only high-sampled features and future knowledge data were used. For AI practitioners, the results demonstrate that incorporating readily available future knowledge, such as scheduled events, can significantly enhance the performance of sparse event forecasting models in time-series prediction, especially in resource-constrained environments. |
Papers for 2025-03-20
| Title | Authors | Summary |
|---|---|---|
| φ-Decoding: Adaptive Foresight Sampling for Balanced Inference-Time | ||
| Exploration and Exploitation (Read more on arXiv or HuggingFace) | Qika, haitengzhao, changma, Meituannnnnn, xufangzhi | Φ-Decoding is a novel inference-time optimization algorithm that balances exploration and exploitation in large language model reasoning. The main research objective is to develop an efficient inference-time strategy that achieves globally optimal step estimation without external auxiliary models. The key methodology is “foresight sampling,” which leverages simulated future steps to derive two distributions (advantage and alignment) for optimal step selection, combined with in-width and in-depth pruning strategies for adaptive computation. Primary results show that Φ-Decoding improves the average reasoning performance of LLaMA3.1-Instruct-8B by over 14% across various reasoning benchmarks compared to auto-regressive CoT. For AI practitioners, Φ-Decoding offers a training-free method to improve LLM reasoning performance while balancing computational cost. |
| DeepMesh: Auto-Regressive Artist-mesh Creation with Reinforcement | ||
| Learning (Read more on arXiv or HuggingFace) | yikaiwang, NTU-yiwen, guangce, yejunliang23, zzzrw | DeepMesh is a framework for generating artist-like 3D triangle meshes conditioned on point clouds and images using an auto-regressive transformer and reinforcement learning. The main research objective is to generate high-quality, aesthetically pleasing meshes with precise topology that align with human preferences, overcoming limitations of existing auto-regressive methods. The key methodology involves an improved mesh tokenization algorithm that reduces sequence length by 72%, a data curation strategy, and Direct Preference Optimization (DPO) with a scoring standard combining 3D metrics and human evaluation. Results show that DeepMesh outperforms state-of-the-art methods, achieving a Chamfer Distance of 0.0884 and a user preference score of 37% on a test dataset. AI practitioners can use DeepMesh’s improved tokenization and DPO implementation to efficiently generate more aesthetically refined 3D meshes, with geometric accuracy for various applications. |
| TULIP: Towards Unified Language-Image Pretraining (Read more on arXiv or HuggingFace) | XuDong Wang, Seun Eisape, Long Lian, yala, ZinengTang | TULIP is a contrastive image-text model that enhances visual feature learning while preserving language grounding. The main research objective is to improve the learning of general-purpose visual features in contrastive image-text models, addressing limitations in fine-grained visual understanding. The methodology leverages generative data augmentation, enhanced image-image and text-text contrastive learning, and image/text reconstruction regularization. TULIP achieved a zero-shot ImageNet-1K top-1 accuracy of 85.3%, surpassing existing models like SigLIP 2. AI practitioners can use TULIP as a drop-in replacement for existing CLIP-like models to achieve state-of-the-art performance on tasks requiring fine-grained visual understanding and improved vision-language representation. |
| Cube: A Roblox View of 3D Intelligence (Read more on arXiv or HuggingFace) | Karun Channa, Nishchaie Khanna, Kiran Bhat, Foundation AI Team, marcelvanworkum | This paper introduces a 3D shape tokenization method for building a foundation model for 3D intelligence on the Roblox platform. The main research objective is to develop a method for converting 3D shapes into discrete tokens that can be used in multi-modal autoregressive sequence models. The key methodology involves a Perceiver-based transformer with Phased-Modulated Positional Encoding, optimal-transport vector quantization, and a stochastic gradient shortcut, trained with a self-supervised loss. Primary results show that the proposed method, Ours-VQ, achieves a 91.7% surface-IoU and 94.5% volumetric-IoU on the Toys4K dataset, surpassing other existing methods such as Craftsman. The principal implication for AI practitioners is that this shape tokenization method enables the development of various 3D generative applications, including text-to-shape, shape-to-text, and text-to-scene generation, allowing for better integration of 3D shapes into large language models. |
| Efficient Personalization of Quantized Diffusion Model without | ||
| Backpropagation (Read more on arXiv or HuggingFace) | Se Young Chun, Kyungryeol Lee, Wongi Jeong, Agorium | ZOODiP enables memory-efficient personalization of quantized diffusion models using only forward passes. The research objective is to reduce the memory demands of diffusion model personalization on edge devices without relying on backpropagation. The key methodology combines zeroth-order optimization with a quantized diffusion model, subspace gradient projection, and partial uniform timestep sampling. The primary results show that ZOODiP achieves comparable performance to prior methods in image and text alignment scores, while reducing training memory demand up to 8.2x (2.37GB VRAM consumption). AI practitioners can leverage this approach for diffusion model personalization in memory-constrained environments, enabling on-device training with significantly reduced resources. |
| Temporal Regularization Makes Your Video Generator Stronger (Read more on arXiv or HuggingFace) | Yajing Bai, Yexin Liu, Xianfeng Wu, Haojian Huang, Harold328 | FLUXFLOW enhances temporal coherence and diversity in video generation by applying controlled temporal perturbations during training. The main research question is whether temporal augmentation, specifically the proposed FLUXFLOW strategy, can improve the temporal quality of generated videos while maintaining spatial fidelity. FLUXFLOW introduces frame-level and block-level temporal perturbations to video data during the training of video generation models, without architectural changes. Experiments on UCF-101 and VBench show that FLUXFLOW applied to VideoCrafter2 improves the FVD score by 19.21 while improves Total Score to 82.36, a 1.92 improvement, enhancing both temporal coherence and diversity without reducing spatial fidelity. AI practitioners can integrate FLUXFLOW as a plug-and-play data augmentation strategy to improve the temporal quality of various video generation models. |
| STEVE: AStep Verification Pipeline for Computer-use Agent Training (Read more on arXiv or HuggingFace) | Chi-Wing Fu, Shu Liu, Ziqin Wei, Zhisheng Zhong, Fanbin Lu | STEVE is a step verification pipeline designed to train computer-use agents using a large, verified instruction set and trajectory data. The main research objective is to develop a scalable training pipeline for computer-use agents that overcomes the limitations of behavior cloning, which requires vast, high-quality trajectories. The key methodology involves establishing a large instruction set, collecting trajectory data with suboptimal agents, using GPT-4o to verify the correctness of each step based on before-and-after screen states, and then employing Kahneman & Tversky Optimization (KTO). A primary result is that the STEVE-trained 7B vision-language model achieved a 23% task success rate on the challenging WinAgentArena live environment using KTO, surpassing the performance of supervised finetuning. The principal implication for AI practitioners is that using step verification with KTO allows training of effective computer-use agents from sub-optimal trajectory data, which scales better and performs better. |
| LEGION: Learning to Ground and Explain for Synthetic Image Detection (Read more on arXiv or HuggingFace) | Weijia Li, Junyan Ye, Siwei Wen, zichenwen, khr0516 | The paper introduces SynthScars, a new dataset for synthetic image detection, and LEGION, a multimodal large language model-based framework for analyzing and refining synthetic images. The main research objective is to develop a model capable of detecting, localizing, and explaining artifacts in fully synthetic images, and to explore its use as a controller for improving image generation. The key methodology involves using a multimodal large language model (MLLM) to integrate artifact detection, segmentation, and explanation, and then applying this in iterative image regeneration and inpainting pipelines. Primary results show that LEGION outperforms existing methods on SynthScars, achieving a 3.31% higher mIoU and 7.75% higher F1 score than the second-best traditional expert, and demonstrates superior robustness. For AI practitioners, LEGION provides a new approach and benchmark for synthetic image analysis, and suggests how deep learning based image detection models can be integrated into the generative process to achieve higher quality of image synthesis. |
| MusicInfuser: Making Video Diffusion Listen and Dance (Read more on arXiv or HuggingFace) | Steven M. Seitz, Brian Curless, Ira Kemelmacher-Shlizerman, Susung Hong | MusicInfuser adapts existing text-to-video diffusion models to generate dance videos synchronized to music, while preserving text-based control over style. The main research objective is to adapt pre-trained text-to-video models to condition on music tracks and generate synchronized dance outputs. The key methodology involves introducing lightweight music-video cross-attention and a low-rank adapter within a video diffusion model, trained on dance videos, without requiring motion capture data. The method achieved a Dance Quality Average score of 7.95, outperforming baselines like Mochi (7.70) and MM-Diffusion (7.16) in comprehensive evaluations including factors like style and beat alignment. AI practitioners can adapt pre-existing video diffusion models for music-driven video generation by incorporating audio features via cross-attention and low-rank adapters, without extensive multimodal training. |
| GKG-LLM: A Unified Framework for Generalized Knowledge Graph | ||
| Construction (Read more on arXiv or HuggingFace) | Jun Liu, haiping Zhu, Shihao Qi, Bifan Wei, VentureZJ | This paper introduces GKG-LLM, a unified framework for constructing generalized knowledge graphs (GKGs), encompassing knowledge graphs, event knowledge graphs, and commonsense knowledge graphs. The main research objective is to develop a unified framework for constructing generalized knowledge graphs (GKGs) that overcomes task-specific differences and integrates knowledge from various graph types. The key methodology is a three-stage curriculum learning fine-tuning framework that iteratively injects knowledge from knowledge graphs (KGs), event knowledge graphs (EKGs), and commonsense knowledge graphs (CKGs) into a Large Language Model (LLM), using the LoRA+ technique. The primary result is that GKG-LLM achieved an average performance of 67.90% across all tasks, outperforming the strongest baseline by 7.49%, and specifically achieved 80.63% on the NYT sentence-level relation extraction task. AI practitioners can leverage the GKG-LLM framework for improved and generalized knowledge graph construction across various domains, achieving state-of-the-art performance with a single, unified model. |
| Mitigating Visual Forgetting via Take-along Visual Conditioning for | ||
| Multi-modal Long CoT Reasoning (Read more on arXiv or HuggingFace) | Han-Jia Ye, Houwen Peng, Zhun Sun, Allen8 | The paper introduces “Take-along Visual Conditioning” (TVC) to address visual forgetting in multi-modal large language models (MLLMs) during long-chain reasoning. The main research question is how to mitigate the decline in attention to visual information in MLLMs as reasoning progresses. The key methodology involves shifting image input to critical reasoning stages and compressing visual tokens via dynamic pruning, combined with Dynamic Visual Reaffirmation (DVR) and Periodic Visual Calibration (PVC). The primary result shows that the TVC approach achieves state-of-the-art performance, with a +3.4% average improvement over previous methods across five mathematical reasoning benchmarks. For AI practitioners, TVC offers a method to improve multi-modal reasoning performance in MLLMs by sustaining visual attention, applicable to tasks like geometric problem-solving. |
| Unlock Pose Diversity: Accurate and Efficient Implicit Keypoint-based | ||
| Spatiotemporal Diffusion for Audio-driven Talking Portrait (Read more on arXiv or HuggingFace) | Chenru Jiang, Yuyao Yan, weiguangzhao, KaiserYaoJM, ChaolongYang | KDTalker is a novel framework that generates audio-driven talking portrait videos using implicit keypoint-based spatiotemporal diffusion. The main research objective is to generate talking head videos with accurate lip synchronization and diverse head poses while maintaining computational efficiency. The methodology combines unsupervised implicit 3D keypoints with a spatiotemporal diffusion model and a custom-designed spatiotemporal attention mechanism. Primary results show that KDTalker achieves a LSE-C score of 7.326 and a head pose diversity of 0.760 on the HDTF dataset, outperforming existing methods. For AI practitioners, KDTalker offers a method for creating realistic talking portrait animations suitable for real-time applications with improved pose diversity and lip-sync accuracy. |
| ELTEX: A Framework for Domain-Driven Synthetic Data Generation (Read more on arXiv or HuggingFace) | Eugene Dmitriev, Julien Capitaine, Sofia Sedlova, Kseniia Murasheva, lavriz | ELTEX is a framework for generating high-quality synthetic training data in specialized domains, like blockchain-related cyberattack detection. The main research objective is to address the scarcity of domain-specific training data in specialized fields like cybersecurity, which limits the performance of Large Language Models (LLMs). ELTEX systematically integrates explicit domain indicator extraction with dynamic prompting to preserve critical domain knowledge during the generation process. Fine-tuning Gemma-2B with ELTEX-generated data, combined with real data, achieved an F1-score of 0.81, competitive with GPT-4. The principal implication is that AI practitioners can use domain-driven synthetic data generation to bridge the performance gap between smaller, more efficient models, and larger models, in specialized domains. |
Papers for 2025-03-19
| Title | Authors | Summary |
|---|---|---|
| RWKV-7 “Goose” with Expressive Dynamic State Evolution (Read more on arXiv or HuggingFace) | saitejautpala, Guangyu, SmerkyG, ZhangRC, BlinkDL | RWKV-7 “Goose” is a new sequence modeling architecture with pre-trained language models that introduces a generalized delta rule with vector-valued gating for improved performance. The main research objective is to develop a sequence modeling architecture that achieves state-of-the-art performance while maintaining efficiency in terms of memory usage and inference time. The key methodology involves a generalized formulation of the delta rule with vector-valued gating, in-context learning rates, and a relaxed value replacement rule, integrated into a modified RWKV-6 architecture. Primary results show that RWKV-7 models achieve state-of-the-art multilingual performance at the 3 billion parameter scale, matching current SoTA English language performance while requiring only constant memory usage and inference time per token; and on English-focused benchmarks the RWKV7-World3-2.9B achieved 71.5 average accuracy. AI practitioners can use RWKV-7 models as efficient alternatives to Transformers, benefiting from reduced inference costs and constant memory usage, particularly beneficial for long-sequence applications. |
| Impossible Videos (Read more on arXiv or HuggingFace) | Hai Ci, mikeshou, ZechenBai | This paper introduces IPV-BENCH, a benchmark for evaluating video generation and understanding models on impossible or counterfactual video content. The main research questions are whether current video generation models can create impossible videos from prompts and whether video understanding models can comprehend them. The key methodology involved creating a taxonomy of impossible video types, generating a dataset of text prompts (IPV-TXT) and videos (IPV-VID), and evaluating various models on tasks including video generation, judgment, multiple-choice question answering, and open-ended question answering. A key finding is that the top-performing video generation model, Mochi 1, generated high-quality impossible videos in only 37.3% of cases. This demonstrates the need for significant improvement in video models’ ability to generate and understand non-real-world scenarios, providing AI practitioners a clear benchmark and identified limitations to guide the development of more robust and creative video models. |
| Creation-MMBench: Assessing Context-Aware Creative Intelligence in MLLM (Read more on arXiv or HuggingFace) | Yingji Liang, Shengyuan Ding, Kai Lan, Zhijian Chen, Xinyu Fang | Creation-MMBench is a new benchmark for evaluating the visual creative capabilities of Multimodal Large Language Models (MLLMs) in real-world, image-based tasks. i) Main research question or objective: To introduce and evaluate Creation-MMBench, a multimodal benchmark designed to assess the creative capabilities of MLLMs in real-world, image-based tasks. ii) Key methodology used: Creation-MMBench comprises 765 test cases across 51 fine-grained tasks, with instance-specific evaluation criteria for assessing response quality and factual consistency with visual inputs, using MLLM-as-a-Judge (GPT-4o) methodology. iii) Primary results: Current open-source MLLMs significantly underperform compared to proprietary models in creative tasks; for instance, Qwen2.5-VL-72B-Instruct achieved a reward of -5.82 and visual factuality score of 8.33 on the overall benchmark, while Gemini-2.0-pro-exp achieved a reward of 4.48 and visual factuality of 8.53. iv) Principal implication for AI practitioners: AI practitioners should address the limitations of current MLLMs in context-aware creativity and visual-based language generation, and focus on developing more comprehensive and fine-grained evaluation criteria, recognizing that visual fine-tuning can negatively impact the base LLM’s creative abilities. |
| DAPO: An Open-Source LLM Reinforcement Learning System at Scale (Read more on arXiv or HuggingFace) | Xiaochen Zuo, Yufeng Yuan, Ruofei Zhu, Zheng Zhang, Qiying Yu | DAPO is an open-source system for large-scale reinforcement learning (RL) with language models (LLMs), achieving state-of-the-art results on mathematical reasoning. The main research objective is to develop and open-source a scalable and reproducible RL system for LLMs that addresses limitations in existing approaches and reproduces industry-level RL results. The key methodology is the Decoupled Clip and Dynamic sampling Policy Optimization (DAPO) algorithm, incorporating techniques like Clip-Higher, Dynamic Sampling, Token-Level Policy Gradient Loss, and Overlong Reward Shaping, built upon the verl framework. The primary result is that DAPO achieves 50 points on AIME 2024 using a Qwen2.5-32B base model, surpassing previous state-of-the-art results with 50% fewer training steps. Principal implications for AI practitioners include that this paper presents a fully open-sourced algorithm, training code and dataset, providing techniques to solve problems like reward noise and training instability for reinforcement learning. |
| DeepPerception: Advancing R1-like Cognitive Visual Perception in MLLMs | ||
| for Knowledge-Intensive Visual Grounding (Read more on arXiv or HuggingFace) | Zonghao Guo, Zhicong Luo, carboncoo, sdudzy, MaxyLee | DeepPerception enhances Multimodal Large Language Models (MLLMs) for knowledge-intensive visual grounding by integrating cognitive reasoning with visual perception. The research introduces and addresses the challenge of knowledge-intensive visual grounding (KVG), requiring fine-grained perception and domain knowledge integration in MLLMs. The methodology involves a two-stage training framework: supervised fine-tuning for cognitive reasoning and reinforcement learning to optimize perception-cognition synergy, using an automated data synthesis pipeline. DeepPerception achieved an 8.08% accuracy improvement on the new KVG-Bench compared to direct fine-tuning, also showcasing +4.60% superior cross-domain generalization. AI practitioners can leverage DeepPerception’s training framework and the KVG-Bench dataset to develop MLLMs with improved cognitive visual perception, enabling more human-like visual understanding in AI systems. |
| CapArena: Benchmarking and Analyzing Detailed Image Captioning in the | ||
| LLM Era (Read more on arXiv or HuggingFace) | Qiushi Sun, Zheng Ma, Jiaxin Fan, songwp, cckevinn | CapArena benchmarks detailed image captioning with large language models (LLMs) through human evaluations and analyzes automated metrics. The main research questions are how well current Vision-Language Models (VLMs) perform on detailed image captioning compared to humans, and how reliably automated metrics can assess detailed caption quality. The key methodology involved creating CapArena, a platform with over 6000 pairwise caption battles with human preference votes, and evaluating various traditional and recent captioning metrics against these human annotations. Primary results showed that top models like GPT-4o achieve or surpass human-level performance, and the VLM-as-a-Judge approach correlated with human rankings at 94.3% at $4 per test. AI practitioners should use VLM-as-a-Judge for efficient and reliable evaluation of detailed image captioning models, as it aligns better with human preference than traditional metrics. |
| Infinite Mobility: Scalable High-Fidelity Synthesis of Articulated | ||
| Objects via Procedural Generation (Read more on arXiv or HuggingFace) | Li Ray Luo, Yitong Wang, Ruiming Liang, Zichao Yu, Xinyu Lian | Infinite Mobility is a procedural pipeline for synthesizing large-scale, high-fidelity 3D articulated objects. The main research objective is to develop a method for generating high-quality articulated objects that overcomes the limitations of existing data-driven and simulation-based approaches. The key methodology utilizes a tree-growing strategy for articulation structure generation, combined with procedural mesh generation or dataset retrieval with refinement, and ensures physical plausibility through constraint rules. The primary results show that the method produces objects comparable to human-annotated datasets, with an average Tree Edit Distance of 78.62 compared to 3.88 of PartNet-Mobility, and outperforms existing generative models in both physical property and mesh quality evaluations. The principal implication for AI practitioners is that the proposed pipeline provides a scalable and high-fidelity data source for training embodied AI agents and generative models, facilitating tasks requiring interaction with articulated objects. |
| Frac-Connections: Fractional Extension of Hyper-Connections (Read more on arXiv or HuggingFace) | Jundong Zhou, Hongzhi Huang, Defa Zhu, Taoer, FetchFortune | Frac-Connections are introduced as a memory-efficient alternative to Hyper-Connections for deep learning models. The main research objective is to address the seesaw effect between gradient vanishing and representation collapse in residual connections without increasing memory access costs. The key methodology is to divide hidden states into multiple parts (fractional expansion), rather than expanding their width, and construct fractional connection strengths. Primary results show that OLMoE-7B-DFC×4 models achieve a training loss reduction of 0.012 and outperform the baseline by +0.95% on WinoGrande. The principal implication for AI practitioners is that Frac-Connections can improve training stability and downstream task performance in large language models with minimal parameter overhead. |
| Cosmos-Transfer1: Conditional World Generation with Adaptive Multimodal | ||
| Control (Read more on arXiv or HuggingFace) | Tiffany Cai, Maciej Bala, Jose Alvarez, Hassan Abu Alhaija, NVIDIA | Cosmos-Transfer1 is a diffusion-based conditional world model that generates videos based on multiple spatial control inputs with an adaptive weighting scheme. The main research objective is to develop a highly controllable world generation model that can leverage multimodal inputs (segmentation, depth, edge) to produce high-quality and diverse simulations. The key methodology involves adding multiple ControlNet branches to a diffusion transformer-based world model (Cosmos-Predict1), training these branches separately, and fusing them with spatiotemporal control maps during inference. Primary results include a Blur SSIM of 0.87 and a Quality Score of 8.54 on the TransferBench evaluation when using uniform weights across all modalities, outperforming single-modality baselines. Principal implication for AI practitioners is that Cosmos-Transfer1 provides a framework for generating high-fidelity and controllable simulations useful in applications requiring diverse and controllable environments, such as robotics Sim2Real transfer and autonomous vehicle data enrichment, where it achieves a real-time generation of a 5-second video in 4.2 seconds. |
| MPBench: A Comprehensive Multimodal Reasoning Benchmark for Process | ||
| Errors Identification (Read more on arXiv or HuggingFace) | Kai Wang, Wangbo Zhao, Jiaxin Ai, Pengfei Zhou, Zhaopan Xu | MPBench is a new benchmark for evaluating multimodal process reward models (PRMs) across diverse reasoning tasks. The main research objective is to systematically assess the effectiveness of PRMs in diverse reasoning scenarios using multi-task, multimodal data. The key methodology involves three evaluation paradigms: Step Correctness, Answer Aggregation, and Reasoning Process Search, applied to a dataset of 9,745 instances across six sub-categories. A primary result is that the state-of-the-art model, GPT-4o, achieved an overall score of 71.2, while weaker models like Qwen2.5-VL-3B scored below random chance on some assessments. The principal implication for AI practitioners is that current multimodal PRMs, even advanced ones, struggle with complex reasoning tasks, indicating a need for improved model capacity and training strategies specifically for process-level supervision and multimodal understanding. |
| Aligning Multimodal LLM with Human Preference: A Survey (Read more on arXiv or HuggingFace) | Jinda Lu, Junkang Wu, Chaoyou Fu, Tao Yu, yifanzhang114 | This survey provides a comprehensive and systematic review of alignment algorithms for multimodal large language models (MLLMs). The main research question is how to categorize and understand the current advancements in aligning MLLMs with human preferences, focusing on application scenarios, dataset construction, and evaluation benchmarks. The key methodology involves a systematic literature review, categorizing existing methods based on application scenarios (general image understanding, complex modalities, extended applications), dataset construction factors (data sources, model responses, preference annotations), and evaluation benchmarks. The primary result found 13 benchmarks used in current MLLM alignment research, and no publicly available, fully human-annotated dataset over 200,000 samples. The principal implication for AI practitioners is the need for developing more efficient methods to balance dataset scalability with quality and find new methods that efficiently use visual information in alignment, moving beyond current limitations. |
| Measuring AI Ability to Complete Long Tasks (Read more on arXiv or HuggingFace) | Katharyn Garcia, Amy Deng, Joel Becker, Ben West, Thomas Kwa | The paper introduces a metric to quantify AI capabilities on long tasks, finding exponential growth in AI task completion time horizon. The main research objective is to quantify AI capabilities in terms of human capabilities, and track the progress. The authors measured human and AI performance on a new dataset of 170 software engineering, cybersecurity, machine learning, and general reasoning tasks, and fit a logistic model to estimate the “50%-task-completion time horizon” for each AI model. Results show the 50% time horizon for frontier AI models like Claude 3.7 Sonnet is around 50 minutes, and has been doubling approximately every seven months since 2019. For AI practitioners, the time horizon metric and trend provide a quantitative framework to assess and forecast AI agent capabilities for performing complex, real-world, long-duration tasks. |
| Concat-ID: Towards Universal Identity-Preserving Video Synthesis (Read more on arXiv or HuggingFace) | Chongxuan Li, Xiaotao Gu, Jiayan Teng, Zhuoyi Yang, Yong Zhong | Concat-ID is a unified framework for identity-preserving video generation that scales to multiple identities and subjects. The main research objective is to develop a framework that achieves a balance between maintaining identity consistency and facial editability in generated videos, without needing extra modules or parameters. The key methodology uses Variational Autoencoders (VAEs) to extract image features, which are concatenated with video latents along the sequence dimension, leveraging solely 3D self-attention mechanisms, combined with a cross-video pairing strategy and a multi-stage training regimen. Primary results show that Concat-ID achieves an ArcSim score of 0.442 and a CLIPDist score of 0.325 for single-identity generation, superior to existing methods in both identity consistency and facial editiablity. Principal implication for AI practitioners is that a single and concise model is sufficient to achieve single-identity, multi-identity, and multi-subject preservation in video generation without additional modules. |
| Temporal Consistency for LLM Reasoning Process Error Identification (Read more on arXiv or HuggingFace) | Xinzhe Juan, Kaixuan Huang, Jiahao Qiu, Yue Wu, Jiacheng Guo | This paper introduces a temporal consistency method to improve large language models’ (LLMs) ability to identify errors in mathematical reasoning processes. The main research question is whether leveraging consistency in a sequence of self-reflection actions can improve verification accuracy in identifying mathematical process errors. The key methodology involves iterative self-checking by LLMs, where each LLM reviews its own verification results based on previous assessments until a stable result is achieved. Applying the method to DeepSeek R1 distilled models, improvements of 46.6% on MathCheck*, 37.9% on ProcessBench, and 29.0% on PRM800K with the 8B model. AI practitioners can use this temporal consistency approach to enhance the reliability of LLM-based verification systems, particularly for mathematical reasoning, by incorporating iterative self-reflection to reduce errors. |
| PEBench: A Fictitious Dataset to Benchmark Machine Unlearning for | ||
| Multimodal Large Language Models (Read more on arXiv or HuggingFace) | Wangbo Zhao, Jiaxin Ai, Weidong Tang, Pengfei Zhou, Zhaopan Xu | PEBench is a new benchmark for evaluating machine unlearning in multimodal large language models, focusing on personal entities and events. The main research objective is to develop a standardized framework to assess the efficacy of machine unlearning (MU) methods in removing specific visual concepts (identity and event) from Multimodal Large Language Models (MLLMs) while preserving performance on unrelated concepts. The key methodology involves creating a synthetic dataset, PEBench, with 200 fictitious individuals and 40 event scenes, coupled with six MU methods, to evaluate unlearning efficacy, generality, and scope using metrics like precision, ROUGE-L, and G-Eval. A primary result is that while most MU methods achieve nearly 100% efficacy for people unlearning, the ROUGE-L score for event descriptions drops from 0.99 to an average of 0.88, showing an impact from the unlearning process in people to events. AI practitioners can use PEBench to systematically evaluate and improve MU methods for MLLMs, ensuring effective removal of specific concepts without degrading performance on unrelated tasks, particularly in privacy-sensitive applications. |
| MM-Spatial: Exploring 3D Spatial Understanding in Multimodal LLMs (Read more on arXiv or HuggingFace) | Justin Lazarow, Haiming Gang, David Griffiths, Nina Wenzel, Erik Daxberger | MM-Spatial introduces a new dataset and benchmark, CA-VQA, to improve 3D spatial understanding in multimodal large language models (MLLMs). The main research objective is to develop an MLLM, MM-Spatial, that excels at 3D spatial reasoning tasks using large-scale 3D scene data. The key methodology involves generating a supervised fine-tuning dataset, CA-VQA, from high-quality 3D scene data, and training MM-Spatial with diverse spatial tasks, metric depth, and multi-view inputs. MM-Spatial achieves state-of-the-art performance on 3D spatial understanding benchmarks, with a 70.1 average score on the CA-VQA spatial category. The principal implication is that AI practitioners can leverage the CA-VQA dataset and MM-Spatial model to enhance MLLMs’ 3D spatial reasoning capabilities, crucial for applications like robotics and AR/VR. |
| Reflect-DiT: Inference-Time Scaling for Text-to-Image Diffusion | ||
| Transformers via In-Context Reflection (Read more on arXiv or HuggingFace) | Yusuke Kato, Arsh Koneru, Akash Gokul, Konstantinos Kallidromitis, Shufan Li | Reflect-DiT improves text-to-image generation by enabling Diffusion Transformers to iteratively refine outputs using past generations and textual feedback. The main research objective is to develop an inference-time scaling method for text-to-image diffusion models that improves image quality and text alignment without extensive retraining. The methodology, Reflect-DiT, uses a vision-language model to critique generated images and provide textual feedback, which a Diffusion Transformer then uses along with previous generations as in-context examples to refine subsequent outputs. Reflect-DiT achieved a new state-of-the-art score of 0.81 on the GenEval benchmark using only 20 samples per prompt. AI practitioners can use Reflect-DiT to improve the quality and prompt alignment of text-to-image diffusion models during inference, achieving better results with fewer samples compared to best-of-N sampling. |
| Florenz: Scaling Laws for Systematic Generalization in Vision-Language | ||
| Models (Read more on arXiv or HuggingFace) | Sven Behnke, Sebastian Houben, Spravil | Florenz investigates scaling laws for systematic generalization in vision-language models (VLMs) by training monolingual models on multilingual tasks with incomplete data coverage. The main research question is how model size and the number of seen training samples affect a monolingual VLM’s ability to generalize to unseen task-language pairs in a multilingual setting. The key methodology involves training a novel encoder-decoder VLM, Florenz, on a synthetic dataset with intentionally missing language coverage for image captioning, using a combination of pre-trained VLM (Florence-2) and LLM (Gemma-2) components. A primary result is that a 30B parameter model could achieve a cross-entropy loss of 2.31 on unseen captioning, and increasing model size has more significant effect on generalizaton than quantity of training samples. This result implies that AI practitioners can potentially achieve cross-lingual transfer in VLMs even with monolingual models by focusing on scaling model size, mitigating the need for exhaustive multilingual data collection for every task. |
| Pensez: Less Data, Better Reasoning – Rethinking French LLM (Read more on arXiv or HuggingFace) | HoangHa | Pensez 7B, a bilingual English-French language model, demonstrates competitive reasoning performance with significantly less training data than comparable models. The main research question is whether strategic fine-tuning on a small, high-quality, bilingual dataset can enhance both the reasoning capabilities and French language proficiency of a large language model. The key methodology involves supervised fine-tuning of a Qwen2.5 7B Instruct base model on a curated 2,000-example bilingual (English-French) dataset, emphasizing data quality, diversity, and explicit reasoning chains. Pensez 7B achieves a 12-point accuracy increase on a French MATH level 5 benchmark compared to the base model. The principal implication is that AI practitioners can achieve strong reasoning performance in LLMs with focused, high-quality datasets, reducing reliance on massive, resource-intensive training corpora. |
| Hyperbolic Safety-Aware Vision-Language Models (Read more on arXiv or HuggingFace) | Rita Cucchiara, Lorenzo Baraldi, Pascal Mettes, Tejaswi Kasarla, tobi1modna | HySAC introduces a novel approach to address unsafe content in vision-language models (VLMs) using hyperbolic space. The main research objective is to develop a VLM that can distinguish between safe and unsafe content without unlearning unsafe concepts, enabling controlled retrieval and classification. The key methodology involves encoding safe and unsafe image-text pairs in a hyperbolic space, employing entailment loss functions to model hierarchical relationships, and using a traversal mechanism to adjust query embeddings for safe or unsafe retrieval. Primary results show that HySAC achieves a recall of 49.8% at R@1 and 90.7% at R@20 for safe content retrieval on the ViSU test set, outperforming existing safety-unlearning CLIP and hyperbolic CLIP models. AI practitioners can use HySAC to build VLMs with enhanced safety awareness, allowing for dynamic control over content moderation and safer retrieval by design without removing information. |
| KUDA: Keypoints to Unify Dynamics Learning and Visual Prompting for | ||
| Open-Vocabulary Robotic Manipulation (Read more on arXiv or HuggingFace) | Yunzhu Li, Mingtong Zhang, Zixian Liu | KUDA is an open-vocabulary robotic manipulation system that integrates visual prompting and dynamics learning through a unified keypoint representation. The main research objective is to develop a system that can perform complex manipulation tasks based on free-form language instructions while accounting for object dynamics. The key methodology involves using a vision-language model (VLM) to generate keypoint-based target specifications from language instructions and RGBD observations, and then employing model-based planning with a learned dynamics model to achieve the specified goals. The system achieved an 80.0% success rate across 60 trials on various manipulation tasks, significantly outperforming baseline methods. AI practitioners can leverage KUDA’s unified keypoint representation to bridge vision-language models and dynamics models, enabling more flexible and robust robotic manipulation systems that can handle a wider variety of objects and tasks. |
| RoCo-Sim: Enhancing Roadside Collaborative Perception through Foreground | ||
| Simulation (Read more on arXiv or HuggingFace) | Junhao Ge, Yifan Lu, Zichen Chao, Anning Hu, yuwendu | RoCo-Sim is a simulation framework for improving roadside collaborative perception by generating diverse, multi-view consistent simulated data. The main research objective is to address data limitations in roadside collaborative perception, such as calibration errors, sparse data, and multi-view inconsistency, by developing a simulation framework. The key methodology involves using dynamic foreground editing and full-scene style transfer of single images, Camera Extrinsic Optimization, a Multi-View Occlusion-Aware Sampler (MOAS), DepthSAM, and a Scalable Post-Processing Toolkit. RoCo-Sim outperforms state-of-the-art methods on the Rcooper-Intersection dataset by 83.74% for AP70. AI practitioners can use RoCo-Sim to generate realistic and diverse roadside perception datasets, substantially enhancing the performance of camera-only 3D detection models without needing extensive real-world data collection or model architecture changes. |
Papers for 2025-03-18
| Title | Authors | Summary | |
|---|---|---|---|
| DropletVideo: A Dataset and Approach to Explore Integral Spatio-Temporal | |||
| Consistent Video Generation (Read more on arXiv or HuggingFace) | Runze Zhang, NeilXu, EllenAP, lixiaochuan, georgedu | DropletVideo introduces a new dataset and model for generating videos with integral spatio-temporal consistency, addressing plot coherence and visual consistency across viewpoints. The main research question is how to ensure integral spatio-temporal consistency in video generation, considering the interplay between plot progression, camera techniques, and prior content impact. The key methodology involves constructing a large-scale dataset (DropletVideo-10M) with detailed captions and developing a diffusion model (DropletVideo) with motion-adaptive generation. Primary results show DropletVideo achieves 37.93% in Camera Motion and 98.94% in Motion Smoothness on VBench++-ISTP benchmarks, indicating a strong ability of DropletVideo to generate videos with integral spatiotemporal consistency. AI practitioners can utilize the open-sourced DropletVideo dataset and model to advance video generation research and applications requiring robust spatio-temporal coherence, particularly multi-plot narratives. | |
| Being-0: A Humanoid Robotic Agent with Vision-Language Models and | |||
| Modular Skills (Read more on arXiv or HuggingFace) | tellarin, SherryXu, takenpeanut, fuyh, Yaya041 | i) Being-0, a hierarchical framework, effectively controls a full-sized humanoid robot for complex embodied tasks by integrating a Foundation Model (FM) with a modular skill library. ii) The research aims to develop a humanoid robotic agent that can perform complex, long-horizon tasks efficiently and robustly in real-world environments. iii) The methodology involves using an FM for high-level planning, a VLM-based Connector module for bridging the gap between the FM and low-level skills, and a modular skill library for locomotion and manipulation. iv) Experiments demonstrate Being-0 achieves an 84.4% average completion rate on long-horizon tasks and 4.2x efficiency in navigation compared to fully FM-based agents when modules (except the FM) are deployed on onboard computation devices. v) The principal implication for AI practitioners is the demonstration of a hierarchical architecture using a lightweight VLM Connector which significantly enhances the embodied decision-making capabilities of humanoid robots and efficiently coordinates locomotion and manipulation. | |
| DreamRenderer: Taming Multi-Instance Attribute Control in Large-Scale | |||
| Text-to-Image Models (Read more on arXiv or HuggingFace) | Yi Yang, z-x-yang, aiJojosh, limuloo1999 | DreamRenderer is a training-free approach for controlling attributes of multiple instances in image-conditioned text-to-image generation. The research aims to enable precise control over the content of individual instances or regions within images generated from textual descriptions and conditioning inputs like depth or canny maps. The key methodology involves “Bridge Image Tokens” for Hard Text Attribute Binding to correctly associate text embeddings with visual attributes, and selective application of “Hard Image Attribute Binding” in vital layers of the FLUX model. DreamRenderer improves the Image Success Ratio by 17.7% over FLUX on the COCO-POS benchmark and enhances performance of layout-to-image models like GLIGEN by up to 26.8%. AI practitioners can leverage DreamRenderer as a plug-and-play controller for fine-grained control over multi-instance image generation without additional training, enhancing controllability in applications like animation and game development. | |
| Edit Transfer: Learning Image Editing via Vision In-Context Relations (Read more on arXiv or HuggingFace) | Qi Mao, AnalMom, guyuchao, Orannue | Edit Transfer introduces a new image editing paradigm that learns transformations from single source-target examples and applies them to new images. The main research question is whether an image editing transformation can be learned from a single source-target example and applied to a new query image. The key methodology is visual relation in-context learning, adapting a DiT-based text-to-image model with a four-panel composite input and lightweight LoRA fine-tuning. The primary result is that Edit Transfer outperforms state-of-the-art TIE and RIE methods in non-rigid editing scenarios, achieving a user preference rate exceeding 80% across all aspects in user studies. The principal implication is that AI practitioners can achieve sophisticated non-rigid image editing using minimal data (42 training images total) and a visual relation in-context learning approach, reducing the need for large-scale datasets and extensive training. | |
| Personalize Anything for Free with Diffusion Transformer (Read more on arXiv or HuggingFace) | Lu Sheng, Lin Li, Haoran Feng, lvhairong, huanngzh | Personalize Anything is a training-free framework for personalized image generation in Diffusion Transformers (DiTs) that achieves high-fidelity subject reconstruction and flexible editing. The research aims to develop a training-free method for personalized image generation in DiTs that preserves identity and supports diverse editing scenarios. The key methodology involves timestep-adaptive token replacement with patch perturbation, injecting reference subject tokens in early denoising steps and transitioning to multi-modal attention in later steps. Evaluations on DreamBench demonstrate state-of-the-art performance, with the method achieving a CLIP-I score of 0.876 and a DreamSim score of 0.179 in single-subject personalization, surpassing existing approaches. AI practitioners can leverage this framework for efficient, high-fidelity personalized image generation and editing in DiTs without the need for training or fine-tuning, achieving superior identity preservation. | |
| WideRange4D: Enabling High-Quality 4D Reconstruction with Wide-Range | |||
| Movements and Scenes (Read more on arXiv or HuggingFace) | mingbao, zbhpku, Juanxi, czkk566, Lingaaaaaaa | WideRange4D enables high-quality 4D scene reconstruction, including wide-range spatial movements of objects, by introducing a new benchmark and a two-stage reconstruction method. The main research objective is to address the limitations of existing 4D reconstruction methods and datasets in handling scenes with significant object spatial variations. The key methodology involves curating a new benchmark, WideRange4D, and proposing a two-stage 4D reconstruction method, Progress4D, which first initializes a high-quality 3D scene and then progressively fits 4D dynamics. Primary results show that Progress4D achieves a PSNR of 28.86 on the WideRange4D benchmark, outperforming existing state-of-the-art methods. The principal implication for AI practitioners is that WideRange4D provides a more challenging and comprehensive benchmark for evaluating 4D generation methods, while Progress4D offers a more stable and higher-quality approach for reconstructing complex 4D scenes with wide-range object movement. | |
| BlobCtrl: A Unified and Flexible Framework for Element-level Image | |||
| Generation and Editing (Read more on arXiv or HuggingFace) | HongxiangLi, daoyuan98, ZyZcuhk, l-li, Yw22 | BlobCtrl is a unified framework for element-level image generation and editing using a probabilistic blob-based representation. The main research objective is to develop a method for precise and flexible manipulation of visual elements in images, overcoming limitations of current diffusion-based methods. The key methodology involves a dual-branch diffusion model with a blob-based representation, self-supervised training with data augmentation, and controllable dropout strategies. BlobCtrl achieves a significantly higher average CLIP-I score of 87.48 for identity preservation tasks, relative to the next best result. AI practitioners can use BlobCtrl for element-level image generation and editing, benefiting from its precise control over visual appearance and spatial layout that improves fidelity. | |
| reWordBench: Benchmarking and Improving the Robustness of Reward Models | |||
| with Transformed Inputs (Read more on arXiv or HuggingFace) | Yoon Kim, Andrew Cohen, mghazvininejad, michiyasunaga, ZhaofengWu | Reward models (RMs) are brittle and their performance degrades substantially when inputs are transformed in meaning- or ranking-preserving ways. The main research objective is to evaluate and improve the robustness of state-of-the-art reward models against input transformations. Key methodology used involves creating reWordBench, a benchmark of transformed RewardBench instances, and regularizing RM training by encouraging similar scores for paraphrased inputs. Primary results show that RM ranking accuracy on RewardBench can drop by 15.3% on the Chat subset when transformed with reWordBench, and regularization reduces the drop to 7.9%. Principal implication for AI practitioners is that RMs need to be explicitly trained for robustness, such as through paraphrase regularization, to ensure reliable performance and avoid potential reward hacking in downstream alignment tasks. | |
| MicroVQA: A Multimodal Reasoning Benchmark for Microscopy-Based | |||
| Scientific Research (Read more on arXiv or HuggingFace) | lundbergemma, chadliu, shcohen, suyc21, jmhb | MicroVQA is a new benchmark for evaluating multimodal reasoning in AI, specifically for microscopy-based biological research. The main research objective is to assess AI models’ ability to perform expert visual understanding, hypothesis generation, and experiment proposal using microscopy images and associated questions. The key methodology involves curating a dataset of 1,042 multiple-choice questions (MCQs) created by biology experts, with a two-stage MCQ generation pipeline involving optimized LLM prompting and an agent-based “RefineBot” to remove language shortcuts. The primary result is that state-of-the-art multimodal large language models (MLLMs) achieve a peak performance of only 53% accuracy on the benchmark. For AI practitioners, this benchmark highlights the need for improved multimodal reasoning capabilities beyond language understanding, specifically in integrating visual information, prior scientific knowledge, and complex reasoning, suggesting that current models are far from expert-level scientific reasoning in this domain. | |
| Free-form language-based robotic reasoning and grasping (Read more on arXiv or HuggingFace) | Matteo Bortolon, Alice Fasoli, Runyu Jiao, SPovoli, FGiuliari | FreeGrasp enables robots to perform grasping tasks based on free-form language instructions by leveraging Vision-Language Models (VLMs) for spatial reasoning. The research explores how pre-trained VLMs can interpret human instructions and understand spatial relationships for robotic grasping in a zero-shot setting. The proposed method, FreeGrasp, uses mark-based visual prompting and object keypoints to facilitate GPT-4o’s spatial reasoning about object arrangements and obstructions. Experiments on the new FreeGraspData dataset show FreeGrasp achieves a Reasoning Success Rate (RSR) of 0.83 without object ambiguity, outperforming the ThinkGrasp baseline. AI practitioners can use FreeGrasp’s approach, combining VLMs with visual prompting, to enhance robotic manipulation tasks requiring complex language understanding and spatial reasoning without the need for more training data. | |
| R1-VL: Learning to Reason with Multimodal Large Language Models via | |||
| Step-wise Group Relative Policy Optimization (Read more on arXiv or HuggingFace) | Jingyi Zhang, Xikun, liushunyu, HuanjinYao, huangjiaxing | R1-VL introduces Step-wise Group Relative Policy Optimization (StepGRPO) to enhance reasoning in Multimodal Large Language Models (MLLMs). The research aims to improve MLLMs’ reasoning abilities beyond simply imitating successful reasoning paths, addressing the sparse reward issue in online reinforcement learning. StepGRPO uses online reinforcement learning with two novel rule-based rewards: Step-wise Reasoning Accuracy Reward (StepRAR) and Step-wise Reasoning Validity Reward (StepRVR), evaluating intermediate reasoning steps and logical structure. R1-VL, developed with StepGRPO, achieved a 63.5% accuracy on the MathVista benchmark, outperforming the baseline Qwen2-VL-7B by 3.8%. AI practitioners can use StepGRPO to train MLLMs with improved reasoning capabilities, achieving more reliable and structured outputs through a process that mitigates sparse reward issues without needing process reward models. | |
| V-STaR: Benchmarking Video-LLMs on Video Spatio-Temporal Reasoning (Read more on arXiv or HuggingFace) | Wei Li, Ziquan Liu, ChenyangSi, lwpyh, Cade921 | This paper introduces V-STaR, a new benchmark for evaluating Video-LLMs’ spatio-temporal reasoning abilities, including a dataset and evaluation metrics. The main research objective is to assess how well Video-LLMs can integrate spatial, temporal, and causal relationships in video understanding, moving beyond simple object recognition. The key methodology is a Reverse Spatio-Temporal Reasoning (RSTR) task that decomposes video understanding into “what”, “when”, and “where” questions, evaluated with coarse-to-fine Chain-of-Thought (CoT) questions generated by a semi-automated GPT-4-powered pipeline. Primary results show that while some models like GPT-4o perform well on “what” questions (60.78% accuracy), their performance on integrated spatio-temporal reasoning is significantly lower, with the best LGM score of 39.51 on the “what-when-where” chain. The principal implication is that current Video-LLMs have significant limitations in consistent spatio-temporal reasoning, requiring AI practitioners to develop methods that enhance causal and relational understanding in video processing models. | |
| VideoMind: A Chain-of-LoRA Agent for Long Video Reasoning (Read more on arXiv or HuggingFace) | Chang Wen Chen, Ye Liu, AnalMom, KevinQHLin | Here’s a concise summary of the research paper: i) VideoMind is a video-language agent that uses a Chain-of-LoRA strategy for temporal-grounded video understanding. ii) The main research objective is to develop an agent that can effectively reason about long videos by identifying and integrating essential capabilities for temporal reasoning. iii) Key methodology involves a role-based agentic workflow (Planner, Grounder, Verifier, Answerer) and a Chain-of-LoRA strategy for efficient role-switching using lightweight LoRA adaptors on a single base model (Qwen2-VL). iv) Primary results: On the CG-Bench long video benchmark, the 2B VideoMind model achieved a 5.94 mIoU, surpassing GPT-40-mini (3.75) and approaching GPT-40 (5.62). v) Principal implication for AI practitioners: The Chain-of-LoRA approach enables the creation of efficient and flexible video reasoning agents, reducing the computational overhead of using multiple models while demonstrating strong performance on grounded video question-answering. | |
| Rewards Are Enough for Fast Photo-Realistic Text-to-image Generation (Read more on arXiv or HuggingFace) | Jing Tang, Kenji Kawaguchi, Weijian Luo, whatlegequ, Luo-Yihong | This paper introduces R0, a novel approach for fast text-to-image generation that relies solely on reward maximization, challenging the necessity of diffusion distillation. The main research question is whether reward signals alone, without diffusion losses, are sufficient for high-quality, few-step text-to-image generation. The key methodology is R0, a conditional generation approach via regularized reward maximization, that treats image generation as an optimization problem in data space. The results show that R0 outperforms previous methods such as RG-LCM and DI++, achieving a HPS of 34.37 and Image Reward of 1.27 using SD-v1.5 in 4 steps. AI practitioners can develop fast and high-quality text-to-image models by focusing on proper reward functions and regularization, without relying on computationally expensive diffusion distillation, and may adapt the framework to other conditional image generation tasks. | |
| MTV-Inpaint: Multi-Task Long Video Inpainting (Read more on arXiv or HuggingFace) | CeciliaJL, XiaodongChen, magicwpf, lianghou, GuZheng | MTV-Inpaint is a unified video inpainting framework that supports multiple tasks, including text/image-guided object insertion and scene completion, and handles long videos. The main research objective is to develop a video inpainting model capable of handling both scene completion and controllable object insertion tasks in long videos, unifying these tasks and with enhanced input controllability. The key methodology involves a dual-branch spatial attention mechanism in a T2V diffusion U-Net, integration of image inpainting models via an I2V mode, and a two-stage pipeline (keyframe plus in-between frame propagation) for long videos. In object insertion, the method achieved a mIOU of 85.00%, surpassing existing baselines. For AI practitioners, MTV-Inpaint offers a single framework capable of various video inpainting tasks and their derivates like multi-modal inpainting, editing and object removal with state-of-art performance, avoiding the needs of training specialized models. | |
| Error Analyses of Auto-Regressive Video Diffusion Models: A Unified | |||
| Framework (Read more on arXiv or HuggingFace) | duchao, TIanyupang, xiaolili, Fengzhuo, k-nick | This paper develops a theoretical framework for analyzing errors in auto-regressive video diffusion models (ARVDMs) and uses the analysis to propose architectural improvements. The main research question is what types of errors are shared by most ARVDMs, why do those errors appear, and how can they be mitigated. The key methodology involves developing a unified framework, Meta-ARVDM, analyzing the KL-divergence between generated and true videos to identify error sources, and deriving an information-theoretic impossibility result related to the error. A primary result is the identification of “error accumulation” and “memory bottleneck”, with the KL-divergence bound including terms for noise initialization, score estimation, discretization errors, and a memory bottleneck term specifically I(Output; Past | Input). The principal implication is that AI practitioners can mitigate the memory bottleneck by modifying the network structure, such as using prepending and channel concatenation, leading to improved trade-offs between error and computational cost. |
| Sightation Counts: Leveraging Sighted User Feedback in Building a | |||
| BLV-aligned Dataset of Diagram Descriptions (Read more on arXiv or HuggingFace) | Jaime-Choi, sangryul, namin0202, eunkey, soarhigh | SIGHTATION, a novel dataset, enhances diagram descriptions for blind and low-vision (BLV) users by incorporating sighted user feedback on Vision Language Model (VLM) outputs. The main research objective is to create a BLV-aligned dataset of diagram descriptions that addresses the misalignment between sighted annotators and BLV user preferences. The key methodology involves a two-pass VLM inference with latent supervision using a guide generated, followed by sighted-user assessments of the VLM-generated descriptions in terms of preference, completion, retrieval, and question answering. Primary results reveal that preference-tuning a 2B model on the dataset increased usefulness ratings by BLV educators by an average of 1.670 standard deviations. Principal implication for AI practitioners is that leveraging sighted user assessments of VLM-generated content, guided by a multi-pass inference, provides a scalable and effective method to develop datasets that meet the needs of BLV users. | |
| Long-Video Audio Synthesis with Multi-Agent Collaboration (Read more on arXiv or HuggingFace) | Li Liu, Xiaojie Xu, yingcongchen, Xxlbigbrother, Buzz-lightyear | i) The paper introduces LVAS-Agent, a novel multi-agent framework for end-to-end long-video audio synthesis. ii) The primary research objective is to address the challenges of long-video dubbing, including semantic shifts and temporal misalignment, by mimicking professional dubbing workflows. iii) The methodology decomposes the synthesis process into scene segmentation, script generation, sound design, and audio synthesis, utilizing VLM and LLM-based agents with discussion-correction and generation-retrieval-optimization mechanisms. iv) The study demonstrates superior audio-visual alignment over baseline methods using LVAS-Bench, a new benchmark dataset with 207 professionally curated long videos, and achieves state-of-the-art performance across distribution matching, audio quality, semantic alignment, and temporal alignment metrics. v) The principal implication for AI practitioners is the provision of a structured, collaborative framework and corresponding dataset that enables higher-quality, contextually aware audio synthesis in long-form video content creation, potentially enhancing viewer immersion and narrative coherence. | |
| Basic Category Usage in Vision Language Models (Read more on arXiv or HuggingFace) | KyleMoore, JesseTNRoberts, HTSawyer | Vision Language Models (VLMs) exhibit human-like basic-level categorization preferences, distinctions between biological/non-biological objects, and expert-level shifts. The main research question is whether basic-level categorization behaviors observed in humans transfer to large language models. The key methodology involved prompting two VLMs (Llama 3.2 Vision Instruct and Molmo 7B-D) with images and comparing model-generated descriptions to a dataset of basic-level image labels, using two-proportion Z-tests for statistical analysis. Primary results showed that Llama 3.2 produced basic-level categorizations in 60.2% of outputs, and both models used basic-level terms significantly less (p<0.01) for non-biological items. The principal implication is that understanding how LLMs represent object categories, mirroring human cognition, is essential for developing models that align more closely with human behavior and interpretability. | |
| Investigating Human-Aligned Large Language Model Uncertainty (Read more on arXiv or HuggingFace) | Pamela Wisniewski, Daryl Watson, Kyle Moore, JesseTNRoberts | This work investigates how well various large language model (LLM) uncertainty measures correlate with human uncertainty. The main research question is what LLM uncertainty measures best align with human group-level uncertainty on non-factual questions. The methodology involves comparing LLM uncertainty on a curated dataset of survey questions against human response distributions, using measures like self-reporting, entropy, and ensemble methods. The primary result is that top-k entropy correlates negatively with human uncertainty and decreases in human-similarity with increased model size (r > 0.3 for many models), but combining multiple measures produces a generalizable model (r ≈ 0.5 for cross-validation and r>0.6 on full data) . AI practitioners can use mixtures of uncertainty quantification methods, and potentially combining methods such as nucleus size and top-k entropy, to create LLMs that better reflect human-like uncertainty, especially for applications requiring calibrated trust and human-AI collaboration. |
Papers for 2025-03-17
| Title | Authors | Summary |
|---|---|---|
| ReCamMaster: Camera-Controlled Generative Rendering from A Single Video (Read more on arXiv or HuggingFace) | Zuozhu, Mu437, Xintao, menghanxia, jianhongbai | ReCamMaster is a framework for re-rendering a given video with novel camera trajectories using a generative model. The main research objective is to develop a camera-controlled generative video re-rendering framework that can reproduce the dynamic scene of an input video at novel camera trajectories. The key methodology involves conditioning a pre-trained text-to-video diffusion model on both the source video and target camera poses using a frame-dimension concatenation technique, and training on a new multi-camera synchronized video dataset created with Unreal Engine 5. The method achieved a FID score of 57.10 and FVD of 122.74 on visual quality, outperforming existing state-of-the-art approaches. AI practitioners can use this framework for video editing tasks like stabilization, super-resolution, and outpainting, offering improved control over camera movements in generated videos. |
| Adversarial Data Collection: Human-Collaborative Perturbations for | ||
| Efficient and Robust Robotic Imitation Learning (Read more on arXiv or HuggingFace) | AutobotZero, hsli-cuhk, Eralien, morninghaze, SiyuanH | Here’s a concise summary of the paper: i) Adversarial Data Collection (ADC) framework improves robotic imitation learning by introducing human-collaborative perturbations during data acquisition. ii) The main research objective is to maximize per-demonstration information density and improve the efficiency and robustness of robotic imitation learning. iii) Key methodology involves a “Two-Humans-in-the-Loop” approach where an adversarial operator dynamically introduces visual and linguistic perturbations during teleoperation by a primary operator. iv) Models trained with 20% of ADC-collected data volume achieved superior generalization and robustness compared to models trained with 100% of traditionally collected data. v) For AI practitioners, ADC provides a practical strategy for enhancing data quality over quantity, reducing the reliance on large datasets for training robust robotic policies in real-world, dynamic environments. |
| Technologies on Effectiveness and Efficiency: A Survey of State Spaces | ||
| Models (Read more on arXiv or HuggingFace) | yuchenFan, xuekai, iseesaw, Youbang, XingtaiHF | i) This survey provides a structured overview of State Space Models (SSMs), comparing their effectiveness and efficiency against transformers. ii) The main objective is to present a coherent and systematic analysis of SSMs, covering their theoretical underpinnings, mathematical formulations, and applications. iii) The survey categorizes SSMs into three main sections: original SSMs, structured SSMs (S4), and selective SSMs (Mamba), emphasizing the technical aspects and key techniques. iv) The paper highlights techniques such as Euler’s method, ZOH, and bilinear transform discretization for enabling the transformation of SSMs from continuous-time to discrete-time, and references the Mamba model achieving a 20-40 time speedup by performing SSM parameter discretization and recurrence computation directly in the GPU SRAM rather than the GPU HBM. v) AI practitioners can use this survey to understand the trade-offs between different SSM architectures, enabling them to make informed decisions when selecting models for sequential data processing and long-context tasks where efficiency is critical. |
| API Agents vs. GUI Agents: Divergence and Convergence (Read more on arXiv or HuggingFace) | Eliblo1969, SiQin88, liqul, shilhe, vyokky | i) This paper comparatively analyzes API-based and GUI-based LLM agents for software automation, examining their divergence and potential convergence. ii) The main objective is to systematically analyze the architectural differences, development workflows, and user interaction models of API-based versus GUI-based LLM agents. iii) The methodology involves a comparative study across key dimensions such as modality, reliability, efficiency, availability, flexibility, security, transparency, human-like interaction, and maintainability, along with illustrative use cases. iv) The primary result shows API agents offer efficiency and security with stable endpoints while GUI agents provide broader applicability, with the finding being that hybrid approaches can combine UI-based steps where APIs are unavailable with direct calls for data-heavy tasks. v) The principal implication for AI practitioners is the need to consider hybrid agent architectures that leverage the strengths of both API- and GUI-based approaches to achieve comprehensive automation across diverse software ecosystems. |
| Large-scale Pre-training for Grounded Video Caption Generation (Read more on arXiv or HuggingFace) | Josef Sivic, Cordelia Schmid, ekazakos | This paper introduces a method for generating video captions with objects grounded via temporally dense bounding boxes, including a new model, datasets, and pre-training approach. The main research objective is to generate video-level captions with corresponding bounding boxes that consistently localize key noun phrases across the video frames. The key methodology includes an automatic annotation method that aggregates frame-level grounded captions into temporally consistent video annotations, coupled with a Grounded Video Caption Generation model (GROVE) that uses spatio-temporal adapters and a temporal objectness head. The primary results show that GROVE, pre-trained on the new, automatically-annotated HowToGround1M dataset (1M videos) and fine-tuned on the manually-annotated iGround dataset, achieves a CIDEr score of 85.4 on the iGround test set. The principal implication is that AI practitioners can leverage large-scale automatic annotation and pre-training, followed by fine-tuning on smaller, high-quality datasets, to achieve state-of-the-art results in grounded video caption generation. |
| FlowTok: Flowing Seamlessly Across Text and Image Tokens (Read more on arXiv or HuggingFace) | Liang-Chieh Chen, QHL067, QihangYu, turkeyju | FlowTok is a framework that enables direct flow matching between text and images by encoding both into compact 1D tokens. The main research question is whether multimodal understanding and generation can be unified by enabling direct transitions within a shared, compact 1D latent space. The key methodology involves projecting both text and images into a unified 1D latent space using an enhanced image tokenizer and a text projector, then applying flow matching. FlowTok reduces the latent space size by 3.3x compared to prior methods at 256 resolution and achieves a COCO FID-30K score of 9.67 while completing training in 26.1 8-A100 days. For AI practitioners, FlowTok offers a more memory-efficient and faster approach to text-to-image and image-to-text generation, by leveraging a compact 1D token representation. |
| Kolmogorov-Arnold Attention: Is Learnable Attention Better For Vision | ||
| Transformers? (Read more on arXiv or HuggingFace) | Xin Li, Killian Hitsman, aritradutta, maitysubhajit | This paper investigates learnable attention mechanisms based on Kolmogorov-Arnold Networks (KANs) for Vision Transformers (ViTs). The main research question is whether a learnable multi-head self-attention (MHSA) module, specifically a Kolmogorov-Arnold Attention (KArAt), can improve the performance of vanilla ViTs. The key methodology involves designing a general KArAt, and a specific variant, Fourier-KArAt, and evaluating them against vanilla ViTs on CIFAR-10, CIFAR-100, and ImageNet-1K datasets, analyzing loss landscapes, weight distributions, and attention maps. The primary result shows ViT-Tiny+Fourier KArAt outperforms ViT-Tiny on CIFAR-10 by 5.40% in Top-1 accuracy, but larger ViT models with KArAt show diminished gains or worse performance. The implication is that directly replacing softmax with learnable activations in ViT’s attention mechanism does not guarantee improved performance, requiring careful design due to increased model complexity and optimization challenges, although in some instances, smaller models can improve their performance. |
| Cockatiel: Ensembling Synthetic and Human Preferenced Training for | ||
| Detailed Video Caption (Read more on arXiv or HuggingFace) | Hao Li, Zhiyu Tan, xiaomengyang, Kobeshegu, Fr0zencr4nE | Cockatiel-13B is a video captioning model that ensembles synthetic and human-aligned training to generate detailed and human-preferred video descriptions. The main research objective is to address the imbalanced video-caption alignment and misalignment with human preferences in existing video detailed captioning (VDC) models. The key methodology involves a three-stage training pipeline that curates data using a human-aligned caption quality scorer, trains a 13B parameter model (Cockatiel-13B) on the curated data, and distills an 8B parameter model (Cockatiel-8B) from it. Primary results show Cockatiel-13B achieving a new state-of-the-art VDCSCORE average of 43.80, outperforming existing models. The principal implication is that AI practitioners can achieve more human-aligned and dimension-balanced video descriptions by utilizing a training procedure that selectively combines diverse model strengths, guided by structured human preferences. |
| Neighboring Autoregressive Modeling for Efficient Visual Generation (Read more on arXiv or HuggingFace) | Hong Zhou, Feng Chen, Shaoxuan He, Yuanyu He, Yefei He | Neighboring Autoregressive Modeling (NAR) is a new paradigm for efficient visual generation that formulates autoregressive visual generation as a progressive outpainting procedure. The main research objective is to develop an autoregressive visual generation method that improves efficiency and preserves spatial/temporal locality, unlike raster-order “next-token prediction” approaches. The key methodology is a near-to-far “next-neighbor prediction” mechanism, using dimension-oriented decoding heads to predict multiple adjacent tokens in parallel along orthogonal dimensions. Results show that on ImageNet 256x256, NAR-L achieves a lower FID (3.06) than LlamaGen-XXL (3.09) with 87.8% fewer steps and 13.8x higher throughput. AI practitioners can use NAR to achieve more efficient autoregressive visual generation with improved fidelity compared to traditional next-token prediction and existing parallel approaches, particularly beneficial for high-resolution image and video tasks. |
| ProJudge: A Multi-Modal Multi-Discipline Benchmark and | ||
| Instruction-Tuning Dataset for MLLM-based Process Judges (Read more on arXiv or HuggingFace) | Fanrui Zhang, Ming Li, Zhaopan Xu, Pengfei Zhou, Jiaxin Ai | ProJudge is a benchmark and instruction-tuning dataset for evaluating multi-modal large language models (MLLMs) as automated process judges for scientific problem-solving. The main research objective is to assess and enhance the capability of MLLMs to perform fine-grained evaluation of step-by-step reasoning in scientific problems, including error detection, classification, and diagnosis. The key methodology involves creating ProJudgeBench, a benchmark of 2,400 multi-modal scientific problems with 50,118 step-level annotations, and ProJudge-173k, a large-scale instruction-tuning dataset, accompanied by a Dynamic Dual-Phase fine-tuning strategy. A key finding is that after fine-tuning on ProJudge-173k, InternVL2.5-8B showed a 58.92% increase in step correctness accuracy. Principal implication for AI practioners is that open-source models, through the ProJudge, can significantly enhance their performance to match that of many state-of-art closed-source, enabling more reliable and nuanced process evaluation in multi-modal reasoning tasks. |
| ARMOR v0.1: Empowering Autoregressive Multimodal Understanding Model | ||
| with Interleaved Multimodal Generation via Asymmetric Synergy (Read more on arXiv or HuggingFace) | Zizhen Li, Fanrui Zhang, Chuanhao Li, Yukang Feng, Jianwen Sun | ARMOR v0.1 is a resource-efficient framework that upgrades existing multimodal large language models (MLLMs) to unified models (UniMs) capable of both understanding and interleaved text-image generation. The main research objective is to enable MLLMs to perform multimodal generation while preserving their understanding capabilities and minimizing computational overhead. The key methodology involves an asymmetric encoder-decoder architecture with a forward-switching mechanism, a curated interleaved dataset, and a three-stage “What or How to Generate” (WoHG) training algorithm. Experimental results show that ARMOR outperforms existing UniMs on multimodal understanding benchmarks (78.8 score on MMB versus 62.6 for Janus-pro) while achieving comparable generation performance. AI practitioners can leverage ARMOR to build UniMs by fine-tuning existing MLLMs, thereby reducing training costs and enabling natural text-image interleaved generation. |
| Learning Few-Step Diffusion Models by Trajectory Distribution Matching (Read more on arXiv or HuggingFace) | Yujun Cai, jingtang, JIACSUN96, whatlegequ, Luo-Yihong | Learning Few-Step Diffusion Models by Trajectory Distribution Matching (TDM) introduces a unified distillation paradigm for accelerating diffusion model sampling. The main research objective is to develop a few-step diffusion model distillation method that combines the strengths of distribution and trajectory matching, overcoming their individual limitations. The key methodology is a data-free score distillation objective that aligns the student’s trajectory with the teacher’s at the distribution level, coupled with a sampling-steps-aware objective for flexible multi-step adaptation. The method distills PixArt-α into a 4-step generator that outperforms its teacher on real user preference at 1024 resolution, accomplishing this with only 500 iterations and 2 A800 hours. For AI practitioners, TDM offers a highly efficient way to train fast and high-quality few-step diffusion models, significantly reducing training cost while surpassing teacher model performance, as demonstrated on text-to-image tasks. |
| ETCH: Generalizing Body Fitting to Clothed Humans via Equivariant | ||
| Tightness (Read more on arXiv or HuggingFace) | Yuliang Xiu, Michael J. Black, Zeyu Cai, Haiwen Feng, Boqian-Li | ETCH is a novel framework for fitting a 3D body model to point clouds of clothed humans by modeling cloth-to-body mapping. The main research objective is to accurately estimate the underlying body shape and pose from 3D scans of clothed humans, generalizing across diverse poses, shapes, and garment types. The key methodology is Equivariant Tightness Fitting, which uses SE(3)-equivariant displacement vectors to represent “tightness” and leverages pose-invariant body correspondences for sparse marker regression. The method reduces directional errors by 67.2% ~ 89.8% in one-shot (out-of-distribution) settings with approximately 1% of training data. AI practitioners can use this method to obtain accurate body shape and pose estimations from 3D scans of clothed individuals, with robustness to variations in clothing and pose, even with limited training data. |
| Open-World Skill Discovery from Unsegmented Demonstrations (Read more on arXiv or HuggingFace) | Yitao Liang, Anji Liu, Shaofei Cai, Zihao Wang, Jingwen Deng | This paper introduces Skill Boundary Detection (SBD), a self-supervised algorithm for segmenting unsegmented demonstration videos into discrete skills for open-world learning. The main research question is how to automatically segment long, unsegmented demonstration videos into meaningful, skill-consistent segments without manual annotations. SBD leverages a pretrained unconditional action-prediction model and detects skill boundaries by identifying significant increases in prediction error, based on event segmentation theory. The method improved the average performance of conditioned policies in Minecraft by 63.7% and 52.1% on short-term atomic skill tasks. AI practitioners can leverage this method to train instruction-following agents from diverse, unlabeled video data, such as YouTube, without requiring manual segmentation or labeling. |
| GoalFlow: Goal-Driven Flow Matching for Multimodal Trajectories | ||
| Generation in End-to-End Autonomous Driving (Read more on arXiv or HuggingFace) | Bo Jiang, Yang Hu, Xingyu Zhang, WonderingWorld, XXXXing | GoalFlow is an end-to-end autonomous driving method that generates high-quality multimodal trajectories using goal-driven flow matching. The main research objective is to address trajectory selection complexity and reduced quality in existing multimodal trajectory generation methods for autonomous driving. The key methodology involves introducing GoalFlow, which constrains trajectory generation using a goal point selected via a novel scoring mechanism, employs Flow Matching for efficient generation, and uses a refined scoring mechanism for optimal trajectory selection. Primary results show GoalFlow achieved a PDMS of 90.3 on the Navsim benchmark, significantly outperforming other methods, and requires only a single denoising step for excellent performance. Principal implication for AI practitioners is that GoalFlow provides a method for generating high-quality, diverse, yet, safe candidate actions for autonomous driving systems enhancing robustness and real-world deployability. |
| MaRI: Material Retrieval Integration across Domains (Read more on arXiv or HuggingFace) | Yuxuan Chen, Huixiong Zhang, Yangfan He, Jianhui Wang, yangzhifei | MaRI is a framework for aligning visual and material properties in a shared embedding space for material retrieval. The main research objective is to bridge the feature space gap between synthetic and real-world materials to improve material retrieval accuracy. The key methodology involves using dual DINOv2-based encoders trained contrastively to map images and materials into a shared space, leveraging a new dataset combining synthetic and real-world material data. Primary results show that on a trained material dataset, MaRI achieves a top-1 instance accuracy of 26.0% and a top-5 instance accuracy of 90.0%. AI practitioners can use MaRI’s framework and dataset to improve the accuracy and generalization of material retrieval, enhancing 3D asset creation and applications requiring realistic material representation. |
| VGGT: Visual Geometry Grounded Transformer (Read more on arXiv or HuggingFace) | Christian Rupprecht, Andrea Vedaldi, Nikita Karaev, Minghao Chen, Jianyuan Wang | VGGT is a feed-forward transformer that directly infers 3D attributes of a scene from multiple images, achieving state-of-the-art results in several 3D tasks. The main research objective is to determine if 3D tasks can be solved directly by a neural network without visual geometry post-processing. The key methodology is a large transformer with alternating frame-wise and global self-attention, trained on multiple 3D-annotated datasets to predict camera parameters, depth maps, point maps, and 3D point tracks. The primary results show that VGGT outperforms state-of-the-art methods on RealEstate10K and CO3Dv2 datasets for camera pose estimation (AUC@30 of 93.5 and 91.8 respectively, with BA), and also achieves superior accuracy on the DTU and ETH3D datasets for multi-view depth and point map estimation, exceeding optimization-based and other feed-forward methods. Principal implication is that AI practitioners can leverage VGGT for fast and accurate 3D reconstruction, reducing or eliminating the reliance on costly iterative optimization techniques commonly used in computer vision, potentially simplifying and accelerating 3D vision pipelines. |
| From TOWER to SPIRE: Adding the Speech Modality to a Text-Only LLM (Read more on arXiv or HuggingFace) | Tsz Kin Lam, Anil Keshwani, Sonal Sannigrahi, Kshitij Ambilduke, bpop | SPIRE extends the TOWER language model to process speech by incorporating discretized speech units and continued pre-training. The main research objective is to integrate English speech processing (transcription and translation) into an existing text-only multilingual LLM, TOWER, while maintaining its original text-task performance. The methodology involves two stages: continued pre-training (CPT) on a mixture of ASR data and TOWER’s text data, and instruction tuning (IT) on MT, ASR, and ST datasets, employing HuBERT-based k-means clustering for speech discretization. SPIREFULL achieves a Word Error Rate (WER) of 4.2 on the LibriSpeech test-clean set, outperforming models like Spirit-LM and the Whisper-base, though not matching the performance of more heavily speech-trained models. AI practitioners can adapt a text-based LLM for speech tasks with preserved performance on text-based tasks by leveraging the recipe of speech discretization and CPT+IT. |
| Group-robust Machine Unlearning (Read more on arXiv or HuggingFace) | Massimiliano Mancini, Elisa Ricci, Stéphane Lathuilière, Subhankar Roy, Thomas De Min | This paper introduces group-robust machine unlearning to address performance degradation in specific demographic groups caused by non-uniformly distributed data removal requests. The main research question is how to unlearn data from a trained model while preserving performance for groups that are over-represented in the forget set. The key methodology involves sample distribution reweighting during retraining and a novel approximate unlearning method (MIU) that minimizes mutual information between model features and group information, alongside mutual information calibration with original model. Primary results show that MIU outperforms standard unlearning methods on CelebA, Waterbirds, and FairFace datasets; for example it achieves 69.0% group accuracy (GA) on CelebA compared with next best of 66.2%, preserving model robustness. The principle implication is that AI practitioners should use distribution reweighting and mutual information-based techniques to mitigate fairness issues in machine unlearning scenarios where data removal requests are not uniformly distributed across groups. |
Papers for 2025-03-14
| Title | Authors | Summary |
|---|---|---|
| CoSTAast: Cost-Sensitive Toolpath Agent for Multi-turn Image Editing (Read more on arXiv or HuggingFace) | Dang Nguyen, zhoutianyi, nandakiran09, advaitgupta | CoSTA* is a cost-sensitive toolpath agent that finds the optimal tool sequence for multi-turn image editing by combining LLMs and A* search. The main research question is how to combine the strengths of large language models (LLMs) and graph search to find cost-efficient tool paths for multi-turn image editing. The key methodology is a three-stage approach called CoSTA* that uses LLMs to create a subtask tree, prunes a graph of AI tools, and then conducts A* search on the subgraph to find a tool path, guided by a combination of cost and quality metrics. CoSTA* achieved an overall accuracy of 0.94 across all tasks, outperforming baselines such as GenArtist (0.73) and CLOVA (0.63) and offers dynamic trade-offs between the computational cost and quality. This implies that AI practitioners can leverage CoSTA* to build more efficient and adaptable image editing systems that can handle complex, multi-turn editing instructions, allowing for dynamic parameter adjustments of quality-cost trade-offs. |
| World Modeling Makes a Better Planner: Dual Preference Optimization for | ||
| Embodied Task Planning (Read more on arXiv or HuggingFace) | xpqiu, Jinlan, CyberDJ, ngc7293, sinwang | D²PO jointly optimizes state prediction and action selection in LVLMs for embodied task planning, improving performance and efficiency. The research objective is to develop a learning framework, Dual Preference Optimization (D²PO), that enhances embodied task planning in large vision-language models (LVLMs) by jointly optimizing state prediction and action selection. The key methodology involves a tree search mechanism for automatic data collection and a dual preference learning approach using preference pairs for both action and state prediction. Primary results show that D²PO significantly outperforms existing methods and GPT-4o on the VoTa-Bench, achieving a 31.4% relative improvement in success rate and a 33.0% improvement in planning efficiency compared to SFT baselines on a 7B-parameter model. The principal implication for AI practitioners is that incorporating world modeling objectives through D²PO substantially enhances the planning capabilities of LVLMs in embodied AI, offering a more effective approach for developing agents that can perform complex tasks with higher success and efficiency. |
| Silent Branding Attack: Trigger-free Data Poisoning Attack on | ||
| Text-to-Image Diffusion Models (Read more on arXiv or HuggingFace) | Sung Ju Hwang, kiminle2, harryjo97, wchoi403, agwmon | This paper introduces a novel data poisoning attack, called Silent Branding Attack, that manipulates text-to-image diffusion models to generate images with specific brand logos, without requiring any text triggers. The main research objective is to develop and validate a data poisoning method that unobtrusively embeds target logos into images generated by text-to-image diffusion models, operating without explicit text triggers. The key methodology involves an automated algorithm that personalizes logos, generates masks for logo placement, and uses inpainting and refinement techniques to seamlessly integrate logos into existing images. The attack achieved a logo inclusion rate (LIR) of 45.00% on the Midjourney dataset and 39.68% on the Tarot dataset with a 100% poisoning ratio, demonstrating successful logo embedding without specific text triggers. AI practitioners should be aware that text-to-image diffusion models are vulnerable to data poisoning attacks that can subtly embed unwanted visual elements, even without trigger words, necessitating safeguards against such manipulations. |
| Charting and Navigating Hugging Face’s Model Atlas (Read more on arXiv or HuggingFace) | yedid, LielAmar, jonkahana, nitzankur, Eliahu | The paper introduces a method for charting and navigating the vast model repository of Hugging Face by constructing a model atlas represented as a directed acyclic graph. The main research objective is to develop a method for recovering the undocumented evolutionary relationships between models in large repositories, and to explore the use cases of such an atlas. The key methodology involves representing models by their weights, calculating pairwise distances, and using temporal and structural priors to predict directed edges, accounting for model merging and quantization. The results show the proposed method recovers 78.87% of the model relations on a Qwen connected component dataset, substantially outperforming baseline methods, and reveal that 99.41% of quantised models in hugging face are leafs (don’t have children). The principal implication is that AI practitioners can use the constructed atlas to improve model discovery, attribute prediction, and heritage tracing, enabling more efficient model reuse and analysis. |
| GoT: Unleashing Reasoning Capability of Multimodal Large Language Model | ||
| for Visual Generation and Editing (Read more on arXiv or HuggingFace) | zengxingyu, shilinyan, LjHuang, gogoduan, LucasFang | This paper introduces Generation Chain-of-Thought (GoT), a new paradigm for visual generation and editing that leverages multimodal large language models (MLLMs) to perform explicit semantic-spatial reasoning before outputting images. The main research objective is to integrate reasoning mechanisms into visual generation and editing to improve the alignment of generated content with human intentions. The key methodology involves formulating GoT as a multimodal reasoning chain, constructing large-scale GoT datasets with 9M+ samples, and developing a unified framework integrating Qwen2.5-VL with a Semantic-Spatial Guidance Module enhanced diffusion model. The GoT framework achieved a 0.64 overall score on the GenEval benchmark for text-to-image generation, outperforming existing methods. For AI practitioners, GoT offers a framework to build visual generation and editing systems with enhanced reasoning capabilities, enabling improved control, more accurate results, and interactive generation based on modified reasoning steps. |
| Transformers without Normalization (Read more on arXiv or HuggingFace) | Zhuang Liu, Kaiming He, ylecun, endernewton, JiachenZhu | This paper introduces Dynamic Tanh (DyT) as a replacement for normalization layers in Transformers, achieving comparable or superior performance. The main research question is whether normalization layers are indispensable in Transformers, and can they be replaced with a simpler alternative. The key methodology involves replacing normalization layers (LayerNorm and RMSNorm) with a proposed element-wise operation, DyT(x) = tanh(αx), where α is a learnable parameter, and empirically evaluating the modified architectures. Primary results show that Vision Transformers (ViT-B) with DyT achieved 82.5% top-1 accuracy on ImageNet-1K, surpassing the 82.3% accuracy of the LN-based model. Principal implication for AI practitioners that normalization layers in Transformers may not be necessary, and simpler, computationally efficient alternatives such as DyT can provide same or better performance across multiple tasks. |
| GroundingSuite: Measuring Complex Multi-Granular Pixel Grounding (Read more on arXiv or HuggingFace) | wenyuliu, steelozazala, wondervictor, LianghuiZhu, RuiHu | GroundingSuite introduces a new benchmark and framework for evaluating and improving pixel-level visual grounding in complex and diverse scenarios. The main research objective is to address limitations in existing pixel grounding datasets, specifically their limited object categories, textual diversity, and annotation quality. The key methodology involves an automated annotation framework (GSSculpt) leveraging multiple VLM agents for entity localization, text generation, and noise filtering, alongside a curated evaluation benchmark (GSEval). A model trained on the new dataset (GSTrain-10M) achieved a cIoU of 68.9 on gRefCOCO, outperforming models trained on other datasets. AI practitioners can use GroundingSuite to train and evaluate models for more robust and generalizable pixel grounding, applicable across diverse granularities and complex referential expressions. |
| New Trends for Modern Machine Translation with Large Reasoning Models (Read more on arXiv or HuggingFace) | acecamel1977, longyuewang, minghaowu, ChenyangLyu, SNF | Large Reasoning Models (LRMs) substantially transform traditional machine translation (MT) by reframing it as a dynamic reasoning task. The main research objective is to explore the potential of LRMs in redefining MT systems and identify the foundational shifts, new opportunities, and challenges they introduce. The key methodology involves a conceptual analysis and empirical case studies of LRM capabilities in various translation scenarios, including stylized, document-level, and multimodal translation. Primary results show LRMs can perform self-reflection to correct errors, automatically utilize pivot translation, and struggle with complex encoded text; experiments on commonMT showed similar BLEURT (73.0-74.2) and COMET (84.1-84.8) scores for both the reasoning and non-reasoning models. AI practitioners should consider LRMs as a means to develop MT systems that function as multilingual cognitive agents capable of reasoning about meaning, context, culture, and intent, beyond simple text conversion. |
| Shifting Long-Context LLMs Research from Input to Output (Read more on arXiv or HuggingFace) | mingshan, tsq2000, Zhiqiang007, bys0318, mozhu | This paper advocates for a shift in long-context large language model (LLM) research, prioritizing long-output generation capabilities over the current focus on long-input processing. The main research objective is to define and address the challenges of developing LLMs capable of generating high-quality, coherent, and contextually relevant long-form text outputs. The key methodology involves analyzing existing datasets, benchmarks, and models, and identifying limitations in long-output generation through statistical analysis and qualitative assessment of model outputs. Primary results show that the demand for long-output generation (exceeding 4,000 tokens) is 2-3 times greater than for equivalent-length inputs in real-world applications, while only 2 out of 104 papers on long-context tasks at major ML/NLP conferences in 2024 directly addressed long-output generation. The principal implication for AI practitioners is the need to develop new datasets, training techniques, and evaluation metrics specifically designed for long-output LLMs to meet real-world demands in areas like creative writing and complex reasoning. |
| VisualWebInstruct: Scaling up Multimodal Instruction Data through Web | ||
| Search (Read more on arXiv or HuggingFace) | Bo Li, Xiang Yue, wenhu, jiachenli-ucsb, jymmmmm | VisualWebInstruct introduces a method for creating large-scale, multimodal instruction datasets by leveraging web search. The main research objective is to address the scarcity of high-quality, diverse training data for reasoning-focused multimodal tasks. The key methodology involves using Google Image Search with 30,000 seed images to collect over 700K unique URLs, extracting QA pairs from HTML accessibility trees, and refining the data using GPT-4O for answer synthesis and consistency filtering. Fine-tuning MAmmoTH-VL on this dataset (named VisualWebInstruct) achieves a state-of-the-art performance of 50.4% average accuracy across seven visual reasoning benchmarks. The principal implication is that AI practitioners can leverage web-scale data to improve the reasoning abilities of vision-language models, particularly on tasks requiring multi-step deliberation with visual context. |
| DiT-Air: Revisiting the Efficiency of Diffusion Model Architecture | ||
| Design in Text to Image Generation (Read more on arXiv or HuggingFace) | Rui Qian, Chen Chen, yinfeiy, tsujuifu, wenzehu | The paper introduces DiT-Air, a streamlined Diffusion Transformer architecture for text-to-image generation that achieves state-of-the-art performance with improved parameter efficiency. The main research objective is to empirically investigate the impact of architectural choices, text-conditioning strategies, and training protocols on the performance and efficiency of Diffusion Transformers (DiTs). The key methodology involves a comparative analysis of vanilla DiT, PixArt-style, and MMDiT variants, along with ablations of text encoders, layer-wise parameter sharing, and a progressive VAE training approach. Primary results show that DiT-Air achieves GenEval and T2I CompBench scores of 82.9 and 59.5, respectively, outperforming existing models while using significantly fewer parameters (66% reduction compared to MMDiT). For AI practitioners, DiT-Air offers a more parameter-efficient architecture for text-to-image diffusion models, enabling competitive performance with reduced computational resources. |
Do I look like a cat.n.01 to you? A Taxonomy Image Generation |
||
| Benchmark (Read more on arXiv or HuggingFace) | Ekaterina Neminova, Alina Lobanova, lilaspourpre, apanc, VityaVitalich | This paper introduces a benchmark for evaluating text-to-image models’ ability to generate images representing taxonomic concepts from WordNet. The main research objective is to assess how well text-to-image models can visualize concepts of varying abstraction levels within a hierarchical taxonomy. The key methodology involves evaluating 12 text-to-image models using 9 taxonomy-related metrics, human feedback, and pairwise evaluation with GPT-4 feedback. The primary results show that Playground-v2 and FLUX consistently outperform other models across metrics, with Playground ranking first in all preference-based evaluations, but the model ranking differs significantly from standard text-to-image tasks. AI practitioners can use this benchmark to evaluate and improve text-to-image models for generating images reflecting structured, hierarchical data, with a clear indication that specific models are much better at reflecting taxonomic data. |
| Open-Sora 2.0: Training a Commercial-Level Video Generation Model in | ||
| $200k (Read more on arXiv or HuggingFace) | Xinying Guo, Tom Young, Chenhui Shen, Zangwei Zheng, Xiangyu Peng | Open-Sora 2.0 is a commercially viable video generation model trained for $200k, demonstrating cost-effective techniques for high-quality video synthesis. The main research objective is to develop a top-performing video generation model at a highly controlled cost, much lower than comparable existing models. Key methodologies used include a hierarchical data filtering system, a deeply compressed video autoencoder (Video DC-AE), a diffusion transformer (DiT) architecture leveraging full attention, and an image-to-video training approach. The model achieves a win rate favorably against other top-performing models in all three aspects of human preference evaluation (visual quality, prompt adherence, and motion quality); specifically it is 5-10x cheaper to train ($200k) than comparables like MovieGen and Step-Video-T2V. Principal implication for AI practitioners is that high-quality video generation models are achievable with significantly reduced training costs through optimized data curation, model architecture, and training strategies. |
| Long Context Tuning for Video Generation (Read more on arXiv or HuggingFace) | lindahua, zhenheny, Ikuinen, Brightmzb, ziyany | Long Context Tuning (LCT) extends pre-trained video diffusion models to generate coherent multi-shot scenes by expanding their context window. The main research objective is to enable scene-level video generation with visual and dynamic consistency across multiple shots. The key methodology involves adapting full attention mechanisms to encompass all shots in a scene, incorporating interleaved 3D positional embedding, and using an asynchronous noise strategy for training. The primary results show that LCT-trained models achieve superior semantic alignment compared to baseline methods, with a user study score of 3.79 versus baselines ranging from 1.57 to 2.50. For AI practitioners, LCT offers a training paradigm to directly adapt single-shot video models for coherent, multi-shot video generation without additional parameters, enabling applications like short film production and interactive video editing. |
| 4D LangSplat: 4D Language Gaussian Splatting via Multimodal Large | ||
| Language Models (Read more on arXiv or HuggingFace) | hpfister, Qmh, wrencanfly, rpzhou, EthanTaylor | 4D LangSplat learns 4D language fields for efficient, time-sensitive, open-vocabulary querying of dynamic scenes. The main research objective is to develop a method for constructing precise 4D language fields that enable both time-agnostic and time-sensitive open-vocabulary queries in dynamic scenes. The key methodology involves using Multimodal Large Language Models (MLLMs) to generate object-wise video captions, encoding these captions into sentence embeddings for supervision, and employing a status deformable network to model continuous state changes. Results show that on the HyperNeRF dataset, for time-sensitive querying the proposed method achieves an accuracy of 89.42% and a vIoU of 66.07%. AI practitioners can use 4D LangSplat to build systems that enable open vocabulary text-based queries, which are time agnostic and time-sensitive, of the evolution and interaction of objects within a dynamic scene. |
| SANA-Sprint: One-Step Diffusion with Continuous-Time Consistency | ||
| Distillation (Read more on arXiv or HuggingFace) | Yuyang Zhao, Shuchen Xue, Junsong Chen, xieenze, sayakpaul | SANA-Sprint is a text-to-image diffusion model that achieves fast, high-quality image generation through hybrid distillation. The main research objective is to develop an efficient diffusion model capable of one-step high-quality text-to-image (T2I) generation while maintaining multi-step sampling flexibility. The key methodology involves transforming a pre-trained flow-matching model for continuous-time consistency distillation (sCM), combined with latent adversarial distillation (LADD), and includes QK-normalization and dense time-embedding. The primary results show SANA-Sprint achieves a 7.59 FID and 0.74 GenEval in only one step, outperforming FLUX-schnell while being 10x faster (0.1s vs 1.1s on H100). The principal implication for AI practitioners is that they can leverage SANA-Sprint for applications requiring real-time or near real-time image generation with significantly reduced computational overhead compared to prior diffusion models. |
| UniGoal: Towards Universal Zero-shot Goal-oriented Navigation (Read more on arXiv or HuggingFace) | Ziwei Wang, Lingqing Zhao, jiwenlu, xuxw98, hangyin | UniGoal is a framework for universal zero-shot goal-oriented navigation that unifies different goal types within a single model. The main research objective is to develop a general framework capable of handling multiple navigation tasks (object, instance-image, and text-based goals) without task-specific training or fine-tuning. The key methodology involves representing both the scene and goals as graphs, performing graph matching, and using a multi-stage exploration policy guided by the matching score and a blacklist mechanism. Results show that UniGoal achieves a 60.2% success rate on instance-image goal navigation on the HM3D benchmark, outperforming prior zero-shot methods. AI practitioners can use UniGoal to deploy navigation agents in new environments with varied goal specifications without needing environment-specific or task-specific retraining. |
| Light-R1: Curriculum SFT, DPO and RL for Long COT from Scratch and | ||
| Beyond (Read more on arXiv or HuggingFace) | tanglifu, JunchenLiu, yyy99, duan901010, cizhenshi | Light-R1 presents a training recipe for long chain-of-thought (COT) reasoning models, achieving state-of-the-art math performance with efficient training. The main research objective was to develop a method for training compact long-COT models from scratch, overcoming limitations of existing approaches. The key methodology involved a curriculum training recipe comprising two-stage supervised fine-tuning (SFT) with a curated dataset and semi-on-policy direct preference optimization (DPO), followed by reinforcement learning (specifically GRPO). The Light-R1-32B model, trained from Qwen2.5-32B-Instruct, achieved 76.6% on the AIME24 benchmark, surpassing DeepSeek-R1-Distill-Qwen-32B. AI practitioners can use this open-sourced approach, including models, data, and code, to efficiently train and deploy long-COT reasoning capabilities in resource-constrained environments, particularly for mathematical problem-solving. |
| CINEMA: Coherent Multi-Subject Video Generation via MLLM-Based Guidance (Read more on arXiv or HuggingFace) | brotherhuang, u302117, BestWishYsh, angtian, dyf | CINEMA is a framework for generating videos featuring multiple subjects, guided by reference images and text, using a Multimodal Large Language Model (MLLM) for improved coherence. The main research objective is to generate coherent multi-subject videos that maintain visual consistency of individual subjects and follow textual prompts, addressing limitations of existing methods that rely on ambiguous keyword mapping. The key methodology involves leveraging an MLLM (specifically Qwen2-VL) to encode multimodal conditions, an AlignerNet to align MLLM outputs with text features, and VAE encoding of reference images for fine-grained visual detail preservation, all integrated within a Multimodal Diffusion Transformer (MM-DiT) framework. The model was trained on 1.46 million video clips, each paired with 1 to 6 human/object references, achieving results shown qualitatively in Figures 5 and 6 with a training configuration using 128 NVIDIA H100 GPUs. For AI practitioners, CINEMA offers a scalable approach for multi-subject video generation that eliminates the need for explicit subject-text correspondences, improving subject consistency, which is beneficial for applications like personalized video content creation. |
| Quantization for OpenAI’s Whisper Models: A Comparative Analysis (Read more on arXiv or HuggingFace) | allisonandreyev | Whisper and its variants are evaluated for speech recognition, focusing on quantization’s impact on model size, latency, and accuracy. The main research objective is to analyze the similarities, differences, and capabilities of three Whisper models (Whisper, Whisper_Streaming, and whisper-timestamped) and quantify the impact of quantization on latency and its viability for edge deployment. The key methodology involves qualitative comparisons of the three models and quantitative evaluation of word error rate (WER) and latency using the LibriSpeech dataset with three quantization methods (INT4, INT5, INT8) in whispercpp. Quantization with INT4 reduced model size by 45% (from 141.11MB to 44.33MB) and decreased latency by 19%, while slightly improved word error rate (0.0199 to 0.0159). Quantization is a viable method for deploying Whisper on resource-limited devices, maintaining accuracy while significantly reducing model size and improving deployment efficiency. |
| Distilling Diversity and Control in Diffusion Models (Read more on arXiv or HuggingFace) | David Bau, RohitGandikota | Distilled diffusion models can retain the control and regain/exceed the diversity of their base models through strategic timestep management. The paper investigates how to distill both diversity and control capabilities from base diffusion models to their efficient distilled variants. The key methodology involves introducing DT-Visualization to analyze latent representations, and a hybrid inference approach that utilizes the base model for the first critical timestep and the distilled model subsequently. The primary results reveal that the hybrid approach achieves a FID score of 10.79 on COCO-30k, better than both the base (12.74) and distilled (15.52) models, while maintaining the distilled model’s inference speed. The principal implication is that AI practitioners can achieve both high diversity and efficiency in image generation using distilled diffusion models without additional training by leveraging the hybrid inference approach. |
| R1-Onevision: Advancing Generalized Multimodal Reasoning through | ||
| Cross-Modal Formalization (Read more on arXiv or HuggingFace) | Xiaoxuan He, Yi Yang, twilightsnow, dcyin, Emilia515 | R1-Onevision introduces a multimodal reasoning model, dataset, and benchmark to improve visual-language understanding and reasoning. The main research objective is to bridge the gap between visual perception and deep reasoning in large language models by employing a cross-modal reasoning pipeline. Key methodologies used include a cross-modal reasoning pipeline that transforms images into formal textural representations and a two-stage post-training strategy (supervised fine-tuning and reinforcement learning). R1-Onevision achieved 29.9% accuracy on MathVision, comparable to the closed-source model GPT-4o. The principal implication for AI practitioners is that formalizing visual information into textual representations, combined with specialized training, can significantly enhance the multimodal reasoning capabilities of large language models, as demonstrated through performance in visual reasoning benchmarks. |
| Autoregressive Image Generation with Randomized Parallel Decoding (Read more on arXiv or HuggingFace) | Huan Wang, Guoqi Li, Jinyue Yang, hp-l33 | ARPG is a visual autoregressive model that enables random-order, parallel image generation. The research objective is to develop an autoregressive image generation model that overcomes the limitations of raster-order approaches in inference efficiency and zero-shot generalization. The methodology involves a “guided decoding” framework that decouples positional guidance (queries) from content representation (key-value pairs) within the causal attention mechanism, to specify the output image token. On ImageNet-1K 256x256, ARPG achieves an FID of 1.94 with 64 sampling steps, attaining over 20x throughput increase and reducing memory use by over 75% compared to autoregressive models of similar scale. AI practitioners can use ARPG as a more efficient and versatile framework for autoregressive image generation, enabling faster and more flexible image synthesis applications. |
| The Curse of Conditions: Analyzing and Improving Optimal Transport for | ||
| Conditional Flow-Based Generation (Read more on arXiv or HuggingFace) | Alexander Schwing, hkchengrex | Conditional optimal transport (C²OT) improves conditional flow-based generative models by addressing a train-test discrepancy caused by standard optimal transport. The main research objective is to analyze and mitigate the performance degradation of minibatch optimal transport (OT) in conditional flow matching when conditions are introduced. The key methodology is the introduction of a conditional weighting term in the OT cost matrix calculation, along with adaptive weight finding and oversampling techniques. The primary results demonstrate C²OT outperforms flow matching (FM) and OT in conditional generation, e.g. achieving a 2-Wasserstein distance of 0.013±0.003 on 8gaussians→moons with continuous conditions vs FM (0.028±0.010) and OT (2.143±1.993). AI practitioners can use C²OT as a drop-in replacement for standard OT in flow matching to achieve better performance in conditional generative modeling, avoiding skewed priors during training. |
| VisualPRM: An Effective Process Reward Model for Multimodal Reasoning (Read more on arXiv or HuggingFace) | Einsiedler, Yeshenglong, Decaux, chenlj22, Weiyun1025 | VisualPRM is an 8B parameter multimodal Process Reward Model (PRM) that improves reasoning in Multimodal Large Language Models (MLLMs) using Best-of-N evaluation. The research introduces VisualPRM and evaluates its effectiveness as a critic model for enhancing MLLM reasoning. The authors construct a multimodal process supervision dataset (VisualPRM400K) and a benchmark (VisualProcessBench) with human-annotated step-wise correctness labels, then train VisualPRM on the dataset. Applying VisualPRM to InternVL2.5-78B achieves a 5.9-point improvement across seven multimodal reasoning benchmarks. AI practitioners can utilize VisualPRM as an effective critic model to enhance the reasoning performance of MLLMs through Test-Time Scaling, particularly with the Best-of-N strategy. |
| “Silent Is Not Actually Silent”: An Investigation of Toxicity on Bug | ||
| Report Discussion (Read more on arXiv or HuggingFace) | Jaydeb Sarker, imranraad | This study investigates toxicity in GitHub bug report discussions, revealing its negative impacts on collaboration and resolution. The main research objective was to analyze how toxicity manifests in bug reports and impacts developers’ bug resolution. The researchers performed a qualitative analysis of 203 bug threads (including 81 toxic ones) from GitHub, selected using stratified sampling and toxicity detection tools (ToxiCR and LLaMA). A primary result was that only 29.11% of toxic bug report issues were linked with a Pull Request, lower than percentages reported in prior studies. The principal implication for AI practitioners is that automated systems for bug severity/priority management, combined with enhanced toxicity detection tools incorporating domain-specific knowledge, are needed to improve communication and efficiency in software projects. |
| PerCoV2: Improved Ultra-Low Bit-Rate Perceptual Image Compression with | ||
| Implicit Hierarchical Masked Image Modeling (Read more on arXiv or HuggingFace) | Daniel Mueller-Gritschneder, Sascha Hauke, HerrSiebert, edukrom, Nikolai10 | PerCoV2 is an open ultra-low bit-rate perceptual image compression system built upon Stable Diffusion 3, enhancing entropy coding through explicit modeling of the discrete hyper-latent image distribution. The main research objective is to improve ultra-low bit-rate image compression while maintaining perceptual quality by using an implicit hierarchical masked image modeling approach. The key methodology involves extending the PerCo framework to Stable Diffusion 3 and comparing autoregressive methods (VAR and MaskGIT) for entropy modeling within a two-stage training protocol. Results on the MSCOCO-30k benchmark show that PerCoV2 achieves higher image fidelity at lower bit-rates than previous methods, with the QLDS masking schedule achieving a 6.34% bit-rate saving over the baseline in the ultra-low bit-rate setting. For AI practitioners, PerCoV2 offers a publicly available, state-of-the-art, ultra low bit-rate image compression approach that, in comparison to previous works, particularly excels at the ultra low-extreme bit rates (0.003-0.03bpp). |
| On the Limitations of Vision-Language Models in Understanding Image | ||
| Transforms (Read more on arXiv or HuggingFace) | Saquib Sarfraz, Hasnain Ali, Ahmad Mustafa Anis | This paper investigates the limitations of Vision-Language Models (VLMs) in comprehending basic image transformations. The main research question is: “Can Vision Language Embedding Models understand simple Image Transformations?”. The researchers created an augmented Flickr8k dataset and evaluated CLIP and SigLIP models’ ability to associate image transformations with textual descriptions and classify transformations. Key results showed that SigLIP Base 256 Multilingual achieved only 47.21% accuracy in understanding augmented descriptions (Experiment 1), and all the VLMs model cannot classify the image transformation correctly. For AI practitioners, the principal implication is that current VLMs, despite strong semantic understanding, have significant limitations in understanding fundamental image transformations which can significantly limit downstream applications of image editing. |
Papers for 2025-03-13
| Title | Authors | Summary |
|---|---|---|
| TPDiff: Temporal Pyramid Video Diffusion Model (Read more on arXiv or HuggingFace) | Mike Zheng Shou, Lingmin Ran | TPDiff is a framework that enhances video diffusion model efficiency by using progressively increasing frame rates during the diffusion process. The main research objective is to reduce the high computational demands of training and inference in video diffusion models. The key methodology is a temporal pyramid approach that divides diffusion into stages, increasing frame rate with each stage, combined with a stage-wise diffusion training framework leveraging data-noise alignment. The primary results demonstrate a 50% reduction in training cost and a 1.5x improvement in inference efficiency compared to vanilla diffusion models. For AI practitioners, TPDiff offers a method to substantially reduce computational requirements in video generation with diffusion models, enabling faster training and more efficient inference. |
| Reangle-A-Video: 4D Video Generation as Video-to-Video Translation (Read more on arXiv or HuggingFace) | Jong Chul Ye, Suhyeon Lee, hyeonho-jeong-video | Reangle-A-Video introduces a framework for generating synchronized multi-view videos from a single input video without using multi-view generative priors. The main research objective is to develop a method for synchronized multi-view video generation from a single monocular video, reframing it as a video-to-video translation task. The methodology involves two stages: (1) Multi-View Motion Learning using self-supervised fine-tuning of an image-to-video diffusion transformer on warped videos, and (2) Multi-View Consistent Image-to-Images Translation using warped and inpainted first frames guided by a multi-view stereo reconstruction network. The proposed method achieves a MEt3R score of 0.0412 for static view transport, outperforming the Vanilla CogVideoX baseline. For AI practitioners, this work provides a new approach to multi-view video generation that leverages existing image and video diffusion priors, removing the need for large-scale 4D datasets and enabling dynamic camera control and static view transport from a single video input. |
| Block Diffusion: Interpolating Between Autoregressive and Diffusion | ||
| Language Models (Read more on arXiv or HuggingFace) | Zhixuan Qi, Zhihan Yang, Justin T Chiu, Aaron Gokaslan, Marianne Arriola | Block Diffusion Language Models (BD3-LMs) interpolate between discrete denoising diffusion and autoregressive models, enabling flexible-length generation and improved inference efficiency. The main research objective is to introduce and evaluate a class of language models that overcome limitations of both autoregressive and diffusion models, specifically addressing fixed-length generation, inference inefficiency, and perplexity gaps. The key methodology involves defining an autoregressive distribution over blocks of tokens, where the conditional probability of each block is specified by a discrete denoising diffusion model, and employing custom training algorithms and data-driven noise schedules. On the LM1B benchmark, BD3-LMs achieved a test perplexity of 28.23 with a block size of 4, outperforming previous diffusion models and closing gap with the AR perplexity of 22.88 . AI practitioners can leverage BD3-LMs for generating arbitrary-length sequences with improved likelihood modeling compared to standard diffusion models, and with parallel generation capabilities beyond autoregressive models. |
| RewardSDS: Aligning Score Distillation via Reward-Weighted Sampling (Read more on arXiv or HuggingFace) | Sagie Benaim, Guy Yariv, Itay Chachy | RewardSDS is a novel score distillation approach that aligns diffusion models with user intent using reward-weighted sampling. The main research objective is to improve the alignment of score distillation sampling (SDS) outputs with user intent in tasks such as text-to-3D generation. The key methodology is RewardSDS, which weights noise samples during score distillation based on alignment scores from a reward model, prioritizing gradients from samples yielding high-reward outputs. Primary results show that RewardSDS and RewardVSD improve over SDS and VSD on text-to-image generation, with ImageReward achieving a 7.19 LLM Grader score compared to 6.74 for the SDS baseline. AI practitioners can utilize RewardSDS as a plug-and-play module to enhance existing SDS-based methods, improving generation quality and alignment with desired reward models in various tasks, including text-to-image and text-to-3D generation. |
| GTR: Guided Thought Reinforcement Prevents Thought Collapse in RL-based | ||
| VLM Agent Training (Read more on arXiv or HuggingFace) | Zongqing Lu, Yuanchun Shi, Junliang Xing, Yijun Yang, Tong Wei | GTR is a framework that prevents “thought collapse” in reinforcement learning-trained vision-language model (VLM) agents by integrating automated thought correction. The main research objective is to investigate and mitigate the phenomenon of “thought collapse” – a degradation of reasoning ability – observed when training VLM agents with RL in visually-grounded environments. The key methodology is Guided Thought Reinforcement (GTR), which uses an off-the-shelf VLM as a corrector to evaluate and refine the agent’s chain-of-thought reasoning at each RL step, combined with SFT thought cloning and PPO updates. Primary results demonstrate that GTR significantly improves performance, achieving a 3-5x higher task success rate on the Points24 card game compared to state-of-the-art methods. Principal implication for AI practioners is that incorporating process-level guidance via automated thought correction during RL training can substantially enhance the decision-making capabilities and generalization of VLM agents in complex visual environments. |
| More Documents, Same Length: Isolating the Challenge of Multiple | ||
| Documents in RAG (Read more on arXiv or HuggingFace) | Gabriel Stanovsky, Michael Hassid, Nir Mazor, Shahar Levy, LihiShalmon | Retrieval-augmented generation (RAG) performance can degrade with more documents, even with a fixed context length. The main research objective was to isolate the effect of the number of retrieved documents on LLM performance in RAG systems, while controlling for context length. Researchers used a modified multi-hop QA dataset (MuSiQue) to create inputs with varying numbers of documents, but a constant total token count, by expanding remaining documents when others were removed. Primary result was increasing documents from 2-4 to 20 can decrease performance by up to 10% on several tested models (Llama-3.1, Gemma-2). The principal implication is that AI practitioners should consider the number of retrieved documents in RAG systems, as increasing their number without also changing the context may worsen system performance. |
| Quantizing Large Language Models for Code Generation: A Differentiated | ||
| Replication (Read more on arXiv or HuggingFace) | Gabriele Bavota, Saima Afrin, Antonio Mastropaolo, mdiipenta, Devy1 | This paper investigates the impact of quantizing large language models (LLMs) on code generation performance, focusing on extreme quantization levels and code-specific calibration datasets. The main research question is how low-bit quantization, different calibration datasets, and model size affect the code generation ability of LLMs. The key methodology involves quantizing CodeLlama and DeepSeek-Coder models to 8, 4, 3, and 2 bits using AQLM, with various calibration datasets, and evaluating performance on MultiPL-E and McEval benchmarks using the pass@1 metric. A primary result is that 4-bit quantization reduces model memory footprint by 70% with no significant performance decrease, while code-specific calibration datasets improve performance at more extreme (3 and 2-bit) quantization levels. AI practitioners can deploy larger code generation models on resource-constrained devices by safely quantizing LLMs down to 4 bits without sacrificing significant performance. |
| WildIFEval: Instruction Following in the Wild (Read more on arXiv or HuggingFace) | Liat Ein-Dor, Ariel Gera, Asaf Yehudai, Gili Lior | WILDIFEVAL introduces a large-scale dataset of real user instructions with multiple constraints to evaluate LLMs’ instruction-following capabilities. i) WILDIFEVAL, a new benchmark of 12K real-world, multi-constrained user instructions, is introduced to evaluate instruction following in LLMs. ii) The main research objective is to assess how well leading LLMs can follow complex, real-world instructions with multiple constraints. iii) Key methodology involved collecting and curating real user instructions from Chatbot Arena, decomposing them into individual constraints, and evaluating LLM performance based on the fraction of fulfilled constraints. iv) The best-performing model achieved a score of 0.65, and all models experienced performance degradation with an increasing number of constraints. v) AI practitioners should focus on improving LLMs’ ability to handle multiple, diverse constraints, particularly length-related constraints, to better align with realistic user needs and expectations in complex text generation tasks. |
| VLog: Video-Language Models by Generative Retrieval of Narration | ||
| Vocabulary (Read more on arXiv or HuggingFace) | Mike Zheng Shou, KevinQHLin | VLog is a video understanding framework that defines video narrations as vocabulary and uses a generative retrieval model for efficient indexing. The main research objective is to develop a video understanding model that generates concise, contextually accurate, and efficient narrations. The key methodology involves a generative retrieval model, a hierarchical vocabulary derived from video narrations using Narration Pair Encoding, and a vocabulary update strategy leveraging generative models. VLog achieves a 20x speedup over generative models on the Vidcab-Eval dataset while maintaining comparable accuracy to retrieval models. AI practitioners can use VLog’s generative retrieval approach to create more efficient video-language models, achieving faster processing speeds with accuracy, especially when handling long videos or requiring real-time responses. |
| Cost-Optimal Grouped-Query Attention for Long-Context LLMs (Read more on arXiv or HuggingFace) | Maosong Sun, Zhiyuan Liu, Xu Han, Yutong Wu, chen-yingfa | The paper investigates cost-optimal configurations for Grouped-Query Attention (GQA) in Transformer-based large language models (LLMs), focusing on trade-offs between performance, computational cost, and memory usage. The main research question is how to optimize the number of attention heads and groups in GQA to minimize computational and memory costs of LLMs while maximizing language modeling capabilities, particularly in long-context scenarios. The key methodology involves systematically comparing LLMs with varying parameter sizes, context lengths, and attention head configurations, extending existing scaling laws to account for context length and attention head configuration. A primary result is that for Llama-3.2-1B at 128K context length, using a head configuration of H=(8,1) and increasing the model size can achieve the same loss while reducing inference memory and FLOPs usage by 48.4% and 49.6% respectively, relative to the standard GQA configuration. The principal implication for AI practitioners is that commonly used GQA configurations can be significantly suboptimal, and carefully selecting the attention head configuration, based on expected inference context length, can substantially reduce computational and memory costs, enabling more efficient deployment of long-context LLMs. |
| Alias-Free Latent Diffusion Models:Improving Fractional Shift | ||
| Equivariance of Diffusion Latent Space (Read more on arXiv or HuggingFace) | Xingang Pan, Shuai Yang, Zeqi Xiao, SingleZombie | Alias-Free Latent Diffusion Models (AF-LDM) improve the shift-equivariance of diffusion models for more consistent image generation. The main research objective is to enhance the fractional shift-equivariance of Latent Diffusion Models (LDMs) to improve consistency in applications like video editing and image-to-image translation. The key methodology involves redesigning attention modules to be shift-equivariant, proposing an equivariance loss to suppress feature bandwidth, and using cross-frame attention in both training and inference. The primary results show that AF-LDM achieves a Latent SPSNR of 40.94 and an Image SPSNR of 28.06 on the FFHQ dataset, demonstrating significantly improved shift-equivariance compared to vanilla LDM. The principal implication for AI practitioners is that they can use AF-LDM to achieve greater consistency and stability in image and video generation tasks requiring shift-equivariance, enabling improved performance in applications like video editing and image-to-image translation. |
| Self-Taught Self-Correction for Small Language Models (Read more on arXiv or HuggingFace) | Irina Nikishina, Chris Biemann, VityaVitalich | The paper introduces the Self-Taught Self-Correction (STaSC) algorithm, enabling small language models (SLMs) to improve their outputs through iterative fine-tuning on self-generated data. The main research objective is to investigate if SLMs can learn self-correction without external information or evaluators, relying solely on intrinsic knowledge. The key methodology is iterative fine-tuning of SLMs using self-generated trajectories, incorporating flexible design choices for initial answer generation, correction filtering, and fine-tuning strategy. Primary results show that on the Natural Questions dataset, the Phi3-Mini model achieved a maximum reward of 0.394 (correction, Improving filter) with Evolving Fine-tuning, with a general observation is that both models’ initial answer’s accuracy also increased by training to improve. The STaSC algorithm allows AI practitioners to develop and deploy more accurate and efficient SLMs, enhancing their reasoning and output quality even with limited external resources. |
| MoC: Mixtures of Text Chunking Learners for Retrieval-Augmented | ||
| Generation System (Read more on arXiv or HuggingFace) | Simin Niu, Hanyu Wang, Zhaoxin Fan, Zhiyuan Ji, Robot2050 | This paper introduces a framework called Mixture-of-Chunkers (MoC) to improve text chunking in Retrieval-Augmented Generation (RAG) systems. The main research objective is to optimize text chunking, a commonly overlooked component of RAG, to improve the quality of retrieved content and subsequently enhance the accuracy of generated answers. The key methodology involves a three-stage process: a multi-granularity-aware router, specialized meta-chunkers, and a post-processing algorithm, using regex-guided chunking and edit-distance rectification. Primary results show that the Meta-chunker-1.5B achieved a BLEU-1 score of 0.3754, and F1 score of 0.2387 on the DuReader dataset, outperforming several baseline methods. For AI practitioners, the proposed MoC framework and evaluation metrics offer a way to enhance RAG system performance by optimizing the text chunking process, a critical yet often under-optimized component of the architecture. |
| Multimodal Language Modeling for High-Accuracy Single Cell | ||
| Transcriptomics Analysis and Generation (Read more on arXiv or HuggingFace) | Xiang Wang, Junfeng Fang, Sihang Li, Jiaqi Yang, Yaorui Shi | scMMGPT is a multimodal pre-trained language model for joint cell and text modeling in single-cell transcriptomics. The main research objective is to develop a unified model that effectively integrates scRNA-seq data and textual descriptions to improve performance on single-cell analysis tasks. The key methodology involves integrating pre-trained cell (scGPT) and text (Llama-2) PLMs using cross-modal projectors, and pre-training on 27 million cells with tasks including cell-text representation alignment, cell description generation, and pseudo-cell generation. Primary results include an 84% relative improvement in textual discrepancy for cell description generation compared to existing methods. The principal implication for AI practitioners is that scMMGPT provides a powerful tool for single-cell analysis and generation, demonstrating superior ability to bridge the modality gap between transcriptomic data and free text descriptions. |
| When Large Vision-Language Model Meets Large Remote Sensing Imagery: | ||
| Coarse-to-Fine Text-Guided Token Pruning (Read more on arXiv or HuggingFace) | Qi Zhu, Kang Wu, Xue Yang, Yingying Zhang, Junwei Luo | This paper introduces a text-guided token pruning method for efficient processing of large remote sensing images (RSIs) by Large Vision-Language Models (LVLMs). The main research objective is to balance image detail and computational cost when LVLMs process large RSIs. The key methodology involves a Region Focus Module (RFM) for text-aware region localization and a Dynamic Image Pyramid (DIP) for coarse-to-fine image tile selection and vision token pruning. The method achieved a 32.16% average accuracy on the new LRS-VQA benchmark, outperforming existing high-resolution strategies. AI practitioners can utilize this approach to build more efficient LVLMs for high-resolution image analysis, particularly beneficial when dealing with limited computing resources or large images. |
| Multi Agent based Medical Assistant for Edge Devices (Read more on arXiv or HuggingFace) | Pragya Sahu, Jagdish Samant, Chinmay Kulkarni, Shivam Akhouri, Sakharam Gawade | This paper introduces an on-device, multi-agent healthcare assistant that leverages task-specific agents for optimized resource utilization, privacy, and scalability. The main research objective is to develop a healthcare assistant for edge devices that addresses privacy, latency, and internet dependency challenges associated with cloud-based systems. The key methodology involves a multi-agent architecture utilizing specialized, smaller models (based on Qwen Code Instruct 2.5 7B) for tasks like intelligent diagnosis, appointment booking, emergency services, vital tracking, and reminder scheduling, combined with a data creation pipeline for synthetic data generation. The fine-tuned planner and caller agents achieved an average RougeL score of 85.5 for planning and 96.5 for calling, respectively, for appointment scheduling. This architecture enables AI practitioners to deploy robust and efficient healthcare solutions on resource-constrained edge devices, enhancing user privacy and responsiveness without relying on continuous internet access. |
| Monte Carlo Diffusion for Generalizable Learning-Based RANSAC (Read more on arXiv or HuggingFace) | Tong Zhang, Wei Ke, Chen Zhao, Jiale Wang | This paper introduces a Monte Carlo diffusion mechanism to improve the generalization of learning-based RANSAC for robust model estimation. The main research objective is to address the limited generalization of existing learning-based RANSAC methods to out-of-distribution data. The key methodology involves a diffusion-based training paradigm that progressively injects noise into ground-truth data and uses Monte Carlo sampling to approximate diverse data distributions. Primary results show that on ScanNet, the proposed method improves AUC @20° by 12% on LoFTR compared to a model trained only on SIFT. For AI practitioners, this provides a training strategy to enhance the generalization ability of learning-based RANSAC estimators across various input data distributions without retraining. |
Papers for 2025-03-12
| Title | Authors | Summary |
|---|---|---|
| Crowdsource, Crawl, or Generate? Creating SEA-VL, a Multicultural | ||
| Vision-Language Dataset for Southeast Asia (Read more on arXiv or HuggingFace) | davidanugraha, rifqifarhansyah, tackhwa, holylovenia, samuelcahyawijaya | SEA-VL is an open-source initiative to develop a vision-language dataset representing Southeast Asian cultures, addressing their underrepresentation in AI research. The main objective is to create a high-quality, culturally relevant vision-language dataset for Southeast Asian (SEA) languages and assess different data collection strategies. The researchers employ a multi-pronged approach that includes crowdsourcing, crawling existing image corpora, and generating synthetic images using diffusion models, followed by human evaluation. Crawling achieves approximately 85% cultural relevance and is more cost- and time-efficient than crowdsourcing, while image generation models are currently found unreliable for accurately reflecting SEA cultures. AI practitioners can leverage this dataset to develop more inclusive vision-language models and should prioritize crawling over generation for efficient collection of culturally relevant visual data. |
| LMM-R1: Empowering 3B LMMs with Strong Reasoning Abilities Through | ||
| Two-Stage Rule-Based RL (Read more on arXiv or HuggingFace) | Jie Liu, Zhiyuan You, Miaosen Zhang, Gongrui Zhang, Yingzhe Peng | i) This paper introduces LMM-R1, a two-stage rule-based RL framework for enhancing reasoning abilities in Large Multimodal Models (LMMs). ii) The main objective is to improve the reasoning capabilities of compact 3B-parameter LMMs, particularly in multimodal contexts. iii) The methodology involves Foundational Reasoning Enhancement (FRE) using text-only data and Multimodal Generalization Training (MGT) to extend reasoning to multimodal domains. iv) Results on Qwen2.5-VL-Instruct-3B show LMM-R1 achieves 4.83% and 4.5% average improvements over baselines in multimodal and text-only benchmarks, respectively, and a 3.63% gain in Football Game tasks. v) LMM-R1 provides AI practitioners with a data-efficient approach to enhance reasoning in LMMs by leveraging text-based reasoning enhancement for effective multimodal generalization. |
| YuE: Scaling Open Foundation Models for Long-Form Music Generation (Read more on arXiv or HuggingFace) | HKUST-Audio, Liam-Liu, dododododo, zhangysk, a43992899 | YuE is a family of open foundation models for long-form, lyrics-to-song music generation based on the LLaMA2 architecture. The main research objective is to develop a system capable of generating high-quality, long-form (up to five minutes) music with coherent structure, lyrical alignment, and engaging vocal melodies from lyrics and other control signals. The key methodology involves a track-decoupled next-token prediction strategy with dual-token output (vocal and accompaniment), structural progressive conditioning using a Chain-of-Thought-like approach, a redesigned music in-context learning framework, and a multitask, multiphase pre-training recipe. Primary results include outperforming or matching several proprietary systems (e.g., Suno, Udio) in human evaluations of musicality, and achieving a mean vocal range of approximately 27 semitones, comparable to closed-source systems. The principal implication for AI practitioners is that YuE provides an open, scalable, and performant approach to full-song lyrics-to-music generation, offering improved controllability and competitive quality to existing proprietary alternatives. |
| UniF^2ace: Fine-grained Face Understanding and Generation | ||
| with Unified Multimodal Models (Read more on arXiv or HuggingFace) | Liya Guo, Linrui Xu, Xuerui Qiu, delinqu, tulvgengenr | UniF²ace is a unified multimodal model designed for fine-grained face understanding and generation tasks, trained on a new specialized dataset. The main research objective is to develop a single model capable of both understanding (image-to-text) and generating (text-to-image) fine-grained facial attributes with high accuracy. The key methodology involves a combination of autoregressive and diffusion models, optimized using a dual discrete diffusion training strategy and a two-level mixture-of-experts architecture, trained on the self-constructed UniF²ace-130K dataset. The primary results show that UniF²ace achieves a FID score of 66.005 and a VLM-score of 88.049 on the UniF²ace-130K test dataset, outperforming existing unified multimodal models and approaching state-of-the-art generative models. The principal implication for AI practitioners is that a unified model, leveraging both score-based and masked generative models with a specialized architecture, can achieve high performance in both detailed facial image understanding and generation, potentially streamlining the development of face-related AI applications. |
| SegAgent: Exploring Pixel Understanding Capabilities in MLLMs by | ||
| Imitating Human Annotator Trajectories (Read more on arXiv or HuggingFace) | Qingpei Guo, Chunluan Zhou, Hao Chen, Yuzhuo Tian, Z-MU-Z | SegAgent introduces a new segmentation framework where Multimodal Large Language Models (MLLMs) mimic human annotators using interactive tools to enhance pixel-level understanding. The main research objective is to develop and evaluate a method for MLLMs to perform fine-grained pixel-level image segmentation by imitating human annotation trajectories. The key methodology is modeling segmentation as a multi-step Markov Decision Process (HLMAT), where MLLMs generate text-based click points iteratively, and adapting policy improvement methods like StaR and process reward modeling (PRM) guided tree search. The primary result is that SegAgent-LLaVA+SAM achieved a 75.72 cIoU on the refCOCO testB dataset, demonstrating performance comparable to state-of-the-art methods. Principal implication for AI practitioners is a new protocol to train and assess the fine-grained visual understanding capabilities of MLLMs on pixel segmentation and interactive tasks. |
| MagicInfinite: Generating Infinite Talking Videos with Your Words and | ||
| Voice (Read more on arXiv or HuggingFace) | Jiantong Zhao, Xuancheng Yang, Shitong Shao, Hongwei Yi, Owen777 | MagicInfinite is a diffusion Transformer framework for generating high-fidelity, infinite-length talking head videos controlled by audio and text. The main research objective is to overcome limitations of existing portrait animation methods in handling diverse character styles, achieving accurate lip synchronization, and enabling efficient long video generation. The key methodology involves a 3D full-attention mechanism with a sliding window denoising strategy, a two-stage curriculum learning scheme (integrating audio, text, and reference images), and region-specific masks with adaptive loss functions. Primary results show that MagicInfinite achieves a 20x inference speed boost over the basemodel and can generate a 10-second 540x540p video in 10 seconds on 8 H100 GPUs without quality loss. For AI practitioners, this framework offers an efficient way to generate high-quality, controllable, and arbitrarily long talking head animations with strong temporal coherence. |
| Seedream 2.0: A Native Chinese-English Bilingual Image Generation | ||
| Foundation Model (Read more on arXiv or HuggingFace) | Liang Li, Fanshi Li, Xiaoxia Hou, Lixue Gong, wujie10 | Seedream 2.0 is a bilingual Chinese-English text-to-image diffusion model that addresses limitations of existing models in cultural understanding, text rendering, and model bias. The main research objective is to develop a foundation model capable of generating high-fidelity images aligned with both Chinese and English prompts, demonstrating superior performance in multiple aspects, including text rendering and understanding of Chinese cultural nuances. The key methodology includes a multi-level optimization framework that integrates a bilingual LLM text encoder, a Glyph-Aligned ByT5 for character-level text rendering, Scaled ROPE, multi-phase post-training (SFT, RLHF), and a data system for continuous knowledge integration. The primary result is that Seedream 2.0 achieves state-of-the-art performance, outperforming models like Midjourney v6.1 and Ideogram 2.0 in human evaluations, with a human evaluation ELO score of 1117, and demonstrating a 78% text accuracy rate and 82% hit rate in Chinese text rendering. Principal implication for AI practitioners is that Seedream 2.0 provides a robust and culturally aware foundation model for bilingual image generation, particularly effective for applications requiring accurate Chinese text rendering and culturally specific content generation, outperforming widely available text-to-image models in the field. |
| Gemini Embedding: Generalizable Embeddings from Gemini (Read more on arXiv or HuggingFace) | Madhuri Shanbhogue, Daniel Cer, Sahil Dua, Feiyang Chen, Jinhyuk Lee | Gemini Embedding is a new state-of-the-art text embedding model that leverages the Gemini large language model for improved generalizability across languages and tasks. The main research objective is to develop a unified embedding model that achieves state-of-the-art performance across a broad range of multilingual text embedding tasks. The key methodology involves initializing the embedding model from Gemini, curating a high-quality training dataset using Gemini, and employing a two-stage training pipeline (pre-finetuning and finetuning) with a contrastive learning objective, culminating with model souping. The primary result is that Gemini Embedding achieves a mean task score of 68.32 on the Massive Multilingual Text Embedding Benchmark (MMTEB), outperforming prior state-of-the-art models. The principal implication for AI practitioners is that they can leverage Gemini Embedding as a highly generalizable, off-the-shelf solution for various downstream tasks, including classification, similarity, clustering, ranking and retrieval, particularly in multilingual settings. |
| LightGen: Efficient Image Generation through Knowledge Distillation and | ||
| Direct Preference Optimization (Read more on arXiv or HuggingFace) | Yexin Liu, Harold Haodong Chen, Haoze Zheng, Yajing Bai, Xianfeng Wu | LightGen is an efficient text-to-image generation model that uses knowledge distillation and direct preference optimization to reduce computational costs. The main research objective is to develop a text-to-image generation model that achieves comparable performance to state-of-the-art (SOTA) models with significantly reduced computational resources and dataset size. The key methodology involves distilling knowledge from SOTA text-to-image models into a compact Masked Autoregressive (MAR) architecture using a synthetic dataset and refining the output with Direct Preference Optimization (DPO). The model achieves an overall performance score of 0.62 on the GenEval benchmark at 512x512 resolution using only 0.7B parameters and a 2M image dataset. AI practitioners can use LightGen to develop high-quality image generation models with limited computational resources and smaller datasets, achieving performance similar to much larger and resource intensive models. |
| Tuning-Free Multi-Event Long Video Generation via Synchronized Coupled | ||
| Sampling (Read more on arXiv or HuggingFace) | Jinwoo Shin, Joon-Young Lee, Jui-Hsien Wang, Seoung Wug Oh, Subin Kim | The paper introduces SynCoS, a tuning-free inference framework for generating multi-event long videos from text prompts using existing text-to-video diffusion models. The main research objective is to extend text-to-video diffusion models for long-form video generation with multiple events while maintaining local smoothness and global coherence. The key methodology, Synchronized Coupled Sampling (SynCoS), combines reverse and optimization-based sampling (DDIM and CSD) with a grounded timestep and fixed baseline noise to synchronize denoising paths across the entire video. SynCoS achieved a subject consistency score of 90.19% on Open-Sora Plan, outperforming baselines. AI practitioners can utilize SynCoS to extend existing diffusion models for high-quality, multi-event, and coherent, long video generation without additional model training. |
| Implicit Reasoning in Transformers is Reasoning through Shortcuts (Read more on arXiv or HuggingFace) | Deqing Yang, Siyu Yuan, Tianhe Lin, hsaest | Transformers trained for implicit multi-step reasoning rely on shortcuts rather than true step-by-step computation, limiting generalization. The main research question is how language models perform implicit reasoning in multi-step tasks, and why advanced reasoning capabilities observed in explicit reasoning do not emerge in implicit reasoning. The researchers trained GPT-2 models from scratch on a synthetic multi-step mathematical reasoning dataset and used activation patching for analysis. Results showed that models trained on data with unfixed premise order had significantly reduced accuracy; for instance, accuracy dropped to ~40% on 5-step reasoning tasks. The principal implication for AI practitioners is that current language models may achieve high performance on tasks with similar patterns through shortcut learning without genuine generalization, particularly in implicit reasoning scenarios. |
| OmniMamba: Efficient and Unified Multimodal Understanding and Generation | ||
| via State Space Models (Read more on arXiv or HuggingFace) | Xinggang Wang, Wenyu Liu, Qian Zhang, Bencheng Liao, Jialv Zou | OmniMamba is a Mamba-based unified multimodal model for both understanding and generation tasks. The main research objective is to develop a unified multimodal generation model that achieves both training and inference efficiency with limited training data. The key methodology involves using a linear-architecture-based Mamba-2, decoupled vocabularies, task-specific LoRA, and a decoupled two-stage training strategy. OmniMamba achieves competitive performance with JanusFlow and surpasses Show-o across benchmarks while using only 2M image-text pairs, demonstrating up to 119.2x speedup and 63% GPU memory reduction. AI practitioners can leverage OmniMamba’s efficient architecture and training strategies for developing multimodal models with reduced computational cost and data requirements. |
| Optimizing Test-Time Compute via Meta Reinforcement Fine-Tuning (Read more on arXiv or HuggingFace) | Edward Emanuel Beeching, Lewis Tunstall, Amrith Setlur, Matthew Y. R. Yang, CohenQu | This paper introduces Meta Reinforcement Fine-Tuning (MRT), a method to optimize test-time compute for large language models (LLMs) by minimizing cumulative regret. The main research question is whether current LLMs efficiently utilize test-time compute and whether scaling approaches continue to be effective as budget improves. The key methodology is to formalize test-time compute optimization as a meta-reinforcement learning problem, using a dense reward bonus based on “progress” quantified by the change in likelihood of eventual success. MRT leads to a 2-3x relative gain in performance and roughly a 1.5x gain in token efficiency for math reasoning compared to outcome-reward RL. For AI practitioners, MRT provides a new fine-tuning method that improves LLM performance and efficiency by optimizing for progress during inference, enabling better utilization of computational resources. |
| Video Action Differencing (Read more on arXiv or HuggingFace) | Alejandro Lozano, Anita Rau, Yuhui Zhang, nicholswang, jmhb | This paper introduces Video Action Differencing (VidDiff), a new task and benchmark for identifying subtle differences between videos of the same action. The main research question is how to identify and describe fine-grained differences between two videos of individuals performing the same action. The key methodology is a three-stage agentic workflow (VidDiff Method) that leverages large language models (LLMs) for difference proposal, CLIP for frame localization, and vision-language models (VLMs) for frame differencing. The primary result is that the proposed VidDiff Method achieves a closed-set accuracy of 56.3%, outperforming GPT-40 (53.5%) and Gemini-1.5 Pro (57.7%), and its open set recall@N is 42.1. AI practitioners can use the VidDiffBench dataset and the VidDiff Method as a benchmark and baseline for developing and evaluating models capable of fine-grained video understanding and comparison, essential for applications like skill learning, coaching and automated performance feedback. |
| ^RFLAV: Rolling Flow matching for infinite Audio Video generation (Read more on arXiv or HuggingFace) | Claudio Ferrari, Tomaso Fontanini, Filippo Botti, Giuseppe Gabriele Tarollo, MaverickAlex | RFLAV is a novel transformer-based architecture for infinite and synchronized audio-video generation. The main research objective is to address the limitations of existing audio-video generation models regarding quality, multimodal synchronization, and duration. The key methodology is a rolling rectified-flow model with a lightweight temporal cross-modality fusion module that processes audio and video in separate branches before combining them. The proposed RFLAV model achieves a FVD score of 38.36 on the AIST++ dataset with 200 denoising steps, surpassing existing state-of-the-art models. For AI practitioners, this model offers an improved method for generating arbitrarily long, high-quality audio-video sequences without the duration constraints of prior methods. |
| “Principal Components” Enable A New Language of Images (Read more on arXiv or HuggingFace) | Xiaojuan Qi, Jiankang Deng, Ismail Elezi, tennant, xwen99 | “Principal Components” Enable A New Language of Images introduces a visual tokenization framework with a provable PCA-like structure in the latent token space. The main research objective is to create a compact, structured image representation that reduces redundancy while effectively decoupling semantic information from less important low-level details in 1D visual tokenizers. The key methodology involves a dynamic nested classifier-free guidance strategy during training to induce an orderliness bias in tokens, combined with a diffusion-based decoder. The approach achieves a state-of-the-art reconstruction FID score of 0.72 on the ImageNet validation set, a 10% improvement over prior methods. For AI practitioners, this method provides a way to generate more interpretable and efficient visual representations, suitable for tasks such as image reconstruction and auto-regressive generative modeling, with fewer tokens for training and inference. |
| BiasEdit: Debiasing Stereotyped Language Models via Model Editing (Read more on arXiv or HuggingFace) | Julian McAuley, Ningyu Zhang, Wei Xu, XinXuNLPer | BIASEDIT is a model editing method for debiasing stereotyped language models by modifying model parameters with lightweight editor networks. The main research objective is to develop an efficient method to remove stereotypical biases from language models without significantly impacting their language modeling capabilities. The key methodology involves training editor hyper-networks using a debiasing loss and a retention loss to generate parameter updates that locally modify a language model’s parameters related to stereotyped biases. Results show that BIASEDIT reduces Stereotype Score (SS) to less than 57% and more than 46% on various LMs, outperforming baselines, while maintaining language modeling scores with small changes. For AI practitioners, BIASEDIT offers a computationally efficient method to mitigate societal biases within pre-trained language models, enabling the development of fairer and more robust NLP applications, and bias editing on upper blocks of language models had fewer negative impacts. |
| QuoTA: Query-oriented Token Assignment via CoT Query Decouple for Long | ||
| Video Comprehension (Read more on arXiv or HuggingFace) | Shukang Yin, Weizhong Huang, Xiawu Zheng, Wang Chen, Yongdong Luo | QuoTA is a training-free framework for long video understanding that enhances existing LVLMs by assigning visual tokens based on query relevance. The main research objective is to improve long-video comprehension in Large Video-Language Models (LVLMs) by mitigating visual redundancy and aligning visual processing with task-specific requirements. The key methodology involves query-oriented frame-level importance assessment using Chain-of-Thoughts reasoning to decouple the query, parallel video frame evaluation with a scoring LVLM, and dynamic visual token assignment based on the generated scores. Implementing QuoTA with LLaVA-Video-7B yields an average performance improvement of 3.2% across six video understanding benchmarks, including Video-MME and MLVU. The principal implication for AI practitioners is that QuoTA offers a plug-and-play module to improve existing LVLMs’ long video understanding capabilities without additional training, enabling more effective processing aligned with given query. |
| Perplexity Trap: PLM-Based Retrievers Overrate Low Perplexity Documents (Read more on arXiv or HuggingFace) | Xiao Zhang, Liang Pang, Haiyuan Zhao, Sunhao Dai, Haoyu Wang | PLM-based retrieval models exhibit a “perplexity trap,” overrating documents with low perplexity, leading to source bias that favors LLM-generated content. The main research question is why PLM-based retrievers prefer low-perplexity documents, even when semantic quality is comparable to human-written ones. The authors employ causal graphs, two-stage least squares (2SLS) regression, and theoretical analysis linking retrieval and language modeling objectives. Results show a consistently negative causal effect of perplexity on relevance scores across multiple datasets and models; for example, on the TREC-COVID dataset, ANCE showed a coefficient of -0.23 (p=0.15). A causal-inspired debiasing method, Causal Diagnosis and Correction (CDC), is proposed to mitigate this effect, which is valuable for those seeking to remove perplexity-related source bias. |
| RayFlow: Instance-Aware Diffusion Acceleration via Adaptive Flow | ||
| Trajectories (Read more on arXiv or HuggingFace) | Xing Wang, Yuxi Ren, Yuhong Yang, Xin Xia, Huiyang Shao | RayFlow is a diffusion model acceleration framework that guides each sample along a unique path to an instance-specific target distribution, improving generation speed and control. The main research objective is to address the slow generation speed, sample quality compromises, and training complexities of existing diffusion model acceleration methods. The key methodology includes guiding each sample along a unique path towards instance-specific target distributions and introducing an importance sampling technique (Time Sampler) for enhanced training efficiency. Primary results show that, on the COCO-5k dataset, the SDXL-Ray model achieved a FID score of 3.90 in a 4-step generation, outperforming several existing methods. A principal implication is that AI practitioners can use RayFlow to generate high-quality images with improved speed, control, and training efficiency compared to existing acceleration techniques. |
| Benchmarking AI Models in Software Engineering: A Review, Search Tool, | ||
| and Enhancement Protocol (Read more on arXiv or HuggingFace) | Maliheh Izadi, philippedebekker, RohamKoohestani | This paper reviews AI4SE benchmarks, introduces a search tool (BenchScout) and enhancement protocol (BenchFrame), and demonstrates improvements on HumanEval, resulting in HumanEvalNext. The main research objective is to address challenges in AI4SE benchmarking, including knowledge fragmentation, benchmark selection, lack of standardization, and existing benchmark limitations. The key methodology involves a systematic literature review of 204 benchmarks, development of a semantic search tool using clustering and dimensionality reduction, and a case study applying a proposed framework (BenchFrame) for benchmark enhancement through code review, modifications, and peer review. A primary result shows that on HumanEvalNext, language models exhibited a pass@1 score reduction of 31.22% compared to the original HumanEval. The principal implication for AI practitioners is that using refined and rigorously evaluated benchmarks like HumanEvalNext provides a more accurate assessment of model capabilities and guides future AI4SE research, emphasizing the need for continuous benchmark improvement. |
| Referring to Any Person (Read more on arXiv or HuggingFace) | Yuda Xiong, Tianhe Ren, Zhaoyang Zeng, Lin Wu, Qing Jiang | This paper introduces “Referring to Any Person,” a new task and model (RexSeek) for detecting all individuals in an image that match a natural language description, along with a new dataset (HumanRef). The main research objective is to develop a model capable of multi-instance person referring, overcoming limitations of existing models and datasets that primarily focus on one-to-one object referring. The key methodology involves integrating a multimodal large language model with an object detection framework, trained in a multi-stage process, and creating a new dataset, HumanRef with 103,028 referring statements. The primary result is that RexSeek achieves a DensityF1 score of 82.3 on the HumanRef benchmark, significantly outperforming existing models like Qwen2.5-VL (31.9 DensityF1). The principal implication is that AI practitioners should leverage this model and the HumanRef for robust referring expression comprehension, especially within the task of referring to any person to enable more precise, multi-instance person detection. |
| AnyMoLe: Any Character Motion In-betweening Leveraging Video Diffusion | ||
| Models (Read more on arXiv or HuggingFace) | Junyong Noh, Chaelin Kim, Seokhyeon Hong, kwanY | AnyMoLe is a novel method for generating 3D character motion in-betweening without character-specific datasets, by leveraging video diffusion models. The main research objective is to address the scarcity of character-specific datasets in motion in-betweening, enabling animation generation for arbitrary characters. The key methodology involves a two-stage video generation process using a fine-tuned video diffusion model (ICAdapt), and motion-video mimicking optimization with a scene-specific joint estimator. The primary results show that AnyMoLe outperforms baseline methods in all metrics, achieving an HL2Q of 0.0015 for humanoid characters, demonstrating superior motion generation. For AI practitioners, this implies a reduced reliance on extensive character-specific datasets for motion in-betweening, expanding the applicability of animation generation to a wider range of characters. |
| AI-native Memory 2.0: Second Me (Read more on arXiv or HuggingFace) | Jingbo Shang, Felix Tao, Tao Gao, Xiang Ying, Jiale Wei | SECOND ME is an AI-native memory system that acts as an intelligent, persistent memory offload for users. The main research objective is to develop and evaluate an LLM-based system that can retain, organize, and dynamically utilize user-specific knowledge to improve human-computer interaction. The key methodology involves a multi-layer hybrid architecture integrating supervised fine-tuning (SFT) and direct preference optimization (DPO) with automated data synthesis and evaluation using LLMs. A key result is that using diverse data sources with strong Chain-of-Thought (CoT) normalization achieved a 0.91 score in the Memory (Self) evaluation metric. AI practitioners can leverage this fully localizable, open-sourced system’s approach to memory parameterization and multi-agent framework to build more personalized and context-aware AI applications. |
| Mixture of Experts Made Intrinsically Interpretable (Read more on arXiv or HuggingFace) | Puneet K. Dokania, Christian Schroeder de Witt, Ashkan Khakzar, Constantin Venhoff, Xingyi Yang | This paper introduces MoE-X, a Mixture-of-Experts language model designed for intrinsic interpretability by leveraging sparsity and width. The main research objective is to design an intrinsically interpretable language model architecture that reduces polysemanticity without relying on post-hoc interpretability methods. The key methodology involves rewriting the MoE layer as an equivalent sparse, wide MLP, enforcing sparse activation within each expert using ReLU, and redesigning the routing mechanism to prioritize experts with the highest activation sparsity. MoE-X achieves a perplexity better than GPT-2 and a chess board state reconstruction score of 0.840, surpassing sparse autoencoder-based approaches. AI practitioners can leverage MoE-X’s architecture for improved interpretability in language models without sacrificing performance, offering a direct path to more transparent and understandable AI systems. |
| NullFace: Training-Free Localized Face Anonymization (Read more on arXiv or HuggingFace) | Nicu Sebe, Terence Sim, Tuomas Varanka, hkung | NullFace is a training-free method for localized face anonymization that preserves non-identity facial attributes using diffusion models. The main research objective is to develop a face anonymization technique that balances identity obscuration with the preservation of key non-identity-related attributes, without requiring model training. The key methodology involves inverting an input image using DDPM inversion to recover initial noise, then denoising it through an identity-conditioned diffusion process with modified identity embeddings, and optionally applying segmentation masks for localized control. The method achieved a re-identification rate of 0.34% on the FFHQ dataset, the lowest among compared methods. For AI practitioners, this method offers a flexible and practical approach to face anonymization, achieving competitive performance in privacy-preserving applications without the need for training or fine-tuning, and enabling controllable localized anonymization. |
| Beyond Decoder-only: Large Language Models Can be Good Encoders for | ||
| Machine Translation (Read more on arXiv or HuggingFace) | Qinghong Zhang, Bei Li, Yongyu Mu, Tong Zheng, luoyingfeng | LaMaTE uses LLMs as encoders within an encoder-decoder architecture for improved machine translation. The main research objective is to explore combining LLMs with NMT by using LLMs for encoding and NMT decoders for efficient and generalizable translation. The key methodology is a two-stage training approach: first pre-training the NMT decoder and adaptor with frozen LLM parameters, then fine-tuning all parameters on a multi-task dataset (ComMT). Primary results show that LaMaTE achieves a COMET score of 82.32 and BLEU score of 33.85, averaging across all tasks in the new ComMT benchmark dataset. Principal implication for AI practitioners is that using LLMs as encoders in encoder-decoder models offers a strong balance between high translation quality, reduced computational cost (2.4-6.5x faster decoding), and generalizability, suggesting a promising direction of research. |
| VisualSimpleQA: A Benchmark for Decoupled Evaluation of Large | ||
| Vision-Language Models in Fact-Seeking Question Answering (Read more on arXiv or HuggingFace) | Lixin Liu, Shasha Guo, Xiaodong Chen, Yihan Zhao, WYLing | VisualSimpleQA is a new benchmark for evaluating fact-seeking question-answering capabilities of large vision-language models (LVLMs). The main research objective is to introduce a multimodal fact-seeking benchmark that allows for decoupled evaluation of visual and linguistic modules in LVLMs and incorporates well-defined difficulty criteria. The key methodology involves human annotation of samples with multimodal questions, text-only questions, rationales, and difficulty scores based on visual and linguistic factors. Primary results show that even state-of-the-art LVLMs like GPT-4o achieve only 60%+ correctness on multimodal questions in VisualSimpleQA, and 30%+ on a harder subset. The principal implication for AI practitioners is that there is substantial room for improvement in both the visual and linguistic modules of LVLMs for fact-seeking QA, especially regarding challenging visual recognition tasks and knowledge identification. |
Papers for 2025-03-11
| Title | Authors | Summary |
|---|---|---|
| Feature-Level Insights into Artificial Text Detection with Sparse | ||
| Autoencoders (Read more on arXiv or HuggingFace) | Kristian Kuznetsov, natriistorm, razzant, plina2polina, Kushnareva | This paper explores enhancing interpretability in artificial text detection (ATD) using Sparse Autoencoders (SAEs) to extract features from a Gemma-2-2b model’s residual stream, categorizing them, and analyzing their effectiveness. The main research objective is to improve ATD interpretability by analyzing the semantics and relevance of SAE-extracted features. The key methodology involves applying SAEs to Gemma-2-2b’s residual stream, analyzing extracted features through domain/model-specific statistics, steering, and manual/LLM-based interpretation, and evaluating feature effectiveness using XGBoost and threshold classifiers. A primary result is that SAE-derived features at the 16th layer outperform a state-of-the-art MTL model and mean-pooled activations on the COLING dataset in detecting artificially generated text. For AI practitioners, using SAEs for feature extraction offers a valuable approach for understanding text generators and detectors and their generalization, which helps in developing more robust and interpretable ATD systems. |
| SEAP: Training-free Sparse Expert Activation Pruning Unlock the | ||
| Brainpower of Large Language Models (Read more on arXiv or HuggingFace) | Xun Liang, BO1022, Ki-Seki, siminniu, UglyToilet | SEAP is a training-free method that prunes large language models (LLMs) by dynamically selecting task-relevant parameters to reduce inference overhead. The main research objective is to develop a pruning technique that reduces computational overhead while maintaining LLM performance on various tasks. The key methodology is Sparse Expert Activation Pruning (SEAP), which identifies task-specific expert activation patterns and prunes the model based on dynamically distributed sparsity. Primary results show that at 50% pruning, SEAP surpasses WandA and FLAP by over 20% in task accuracy on the Llama-2-7B model. The principal implication for AI practitioners is that SEAP provides a scalable and effective approach for optimizing large-scale LLMs, enabling more efficient deployment in resource-constrained environments. |
| MM-Eureka: Exploring Visual Aha Moment with Rule-based Large-scale | ||
| Reinforcement Learning (Read more on arXiv or HuggingFace) | wangwhcore, friskit, hflqf88888, Cierra0506, FanqingM | MM-Eureka successfully extends large-scale rule-based reinforcement learning (RL) to multimodal reasoning, demonstrating visual “aha moments”. The main research objective was to investigate the effectiveness of large-scale RL in multimodal reasoning and open-source the pipeline. The key methodology involved applying rule-based RL without supervised fine-tuning, using a simple reward function (accuracy and format), and the REINFORCE Leave-One-Out (RLOO) algorithm. MM-Eureka-Zero-38B, trained with only 9.3k image-text data, achieved a 46.4% accuracy on the K12 math test set, surpassing the instruct model and an 8.2% improvement. AI practitioners can use this open-sourced framework and simple RL setup to efficiently improve the multimodal reasoning ability of both instruction-tuned and pre-trained models, with potentially significant data efficiency gains. |
| Taking Notes Brings Focus? Towards Multi-Turn Multimodal Dialogue | ||
| Learning (Read more on arXiv or HuggingFace) | Zongqing Lu, Jiazheng Liu, tellarin, sipeng9527 | This paper introduces MMDiag, a new multi-turn multimodal dialogue dataset, and DiagNote, a model designed to improve focus and reasoning in such dialogues. The research aims to address the challenge of maintaining focus on target regions in multi-turn multimodal dialogues, specifically “saliency tracking” and “saliency recall”. The key methodology involves a new dataset, MMDiag, generated collaboratively through rules and GPT assistance, and a two-module (Deliberate and Gaze) model, DiagNote, that interacts to perform Chain-of-Thought reasoning and annotations. DiagNote, trained on MMDiag + COCO, achieved a 0.648 average Intersection over Union (IoU) score on grounding benchmarks, outperforming baselines. For AI practitioners, the work provides a new challenging benchmark (MMDiag) and demonstrates improved multimodal grounding and reasoning abilities via the proposed DiagNote model, potentially leading to better handling multi-turn conversational settings. |
| Automated Movie Generation via Multi-Agent CoT Planning (Read more on arXiv or HuggingFace) | Zeyu Zhu, AnalMom, weijiawu | MovieAgent is a multi-agent framework for automatically generating long-form videos from a script synopsis and character bank. The main research objective is to automate the process of movie generation, including narrative planning, scene structuring, and shot composition, which traditionally requires extensive manual effort. The key methodology involves a hierarchical Chain-of-Thought (CoT) reasoning process using multiple LLM agents simulating roles like director, screenwriter, and storyboard artist, decomposing the movie generation process into manageable, sequential steps. Primary results show MovieAgent achieving a CLIP score of 22.25 and an Inception score of 9.39 in keyframe generation, with 97.84 motion smoothness in video generation. The principal implication is that AI practitioners can leverage this framework to significantly reduce the cost and time required for movie/long-video production, automating narrative and cinematic planning while ensuring character consistency and narrative coherence. |
| FedRand: Enhancing Privacy in Federated Learning with Randomized LoRA | ||
| Subparameter Updates (Read more on arXiv or HuggingFace) | Sung Ju Hwang, matbambbang, Seanie-lee, Sangsang | FedRand enhances privacy in federated learning (FL) for vision-language models (VLMs) by randomizing Low-Rank Adaptation (LoRA) subparameter updates. The main research objective is to mitigate membership inference attacks (MIAs) in FL when training VLMs, specifically addressing the vulnerability caused by exposing full client model parameters to the central server. The key methodology, FedRand, involves clients randomly selecting a subset of LoRA parameters from the server and keeping the remaining LoRA parameters private; after local training, only non-private parameters are sent back for aggregation. Experimental results on MSCOCO show FedRand achieved a CIDEr score of 110.27 while maintaining an AUROC of 53.84% against MIAs, demonstrating comparable task performance to FedAvg (CIDEr: 111.08) and improved MIA robustness. This implies that AI practitioners can improve privacy in federated learning of VLMs, without significant performance degradation, by communicating only a random subset of LoRA parameters between client and server. |
| DistiLLM-2: A Contrastive Approach Boosts the Distillation of LLMs (Read more on arXiv or HuggingFace) | Luming Liang, tding1, sungnyun, tianyic, jongwooko | DISTILLM-2 introduces a contrastive learning approach to improve knowledge distillation for compressing large language models (LLMs). Main research question or objective: Can a contrastive approach, considering both teacher and student generated outputs, improve the performance of distilled smaller language models (sLMs)? Key methodology used: DISTILLM-2 uses a contrastive loss function (combining Skew KL and reverse Skew KL) applied asymmetrically to teacher- and student-generated responses, along with optimized data curation and curriculum-based adaptive loss mechanisms. Primary results: DISTILLM-2 achieved state-of-the-art performance on instruction-following, outperforming the second-best method by +2.34%, on average for Qwen2-1.5B model. Principal implication for AI practitioners: AI practitioners can utilize DISTILLM-2 to build high-performing, compact language models suitable for deployment where computational resources are limited, using the proposed contrastive distillation. |
| EasyControl: Adding Efficient and Flexible Control for Diffusion | ||
| Transformer (Read more on arXiv or HuggingFace) | Jiaming Liu, Yirui Yuan, wanghaofan, yiren98, zzyx | i) EasyControl is presented as a lightweight, efficient, and flexible framework for condition-guided Diffusion Transformers (DiT). ii) The research objective is to enable efficient and flexible control over DiT models, addressing limitations in existing spatial and subject control mechanisms. iii) The method involves a Condition Injection LoRA Module, a Position-Aware Training Paradigm, and a Causal Attention Mechanism with KV Cache. iv) The framework achieves a 58% reduction in inference time compared to ablated versions while maintaining a 15M parameter count in single-condition settings, with the best overall performance in multi-condition configurations. v) EasyControl offers AI practitioners an efficient and adaptable approach to conditional image generation with DiT models, particularly beneficial for applications requiring precise spatial control, subject manipulation, and multi-condition integration. |
| FEA-Bench: A Benchmark for Evaluating Repository-Level Code Generation | ||
| for Feature Implementation (Read more on arXiv or HuggingFace) | Wei Li, lisijia0504, yangyu90, dawnmsg, CharonBony | FEA-Bench is a benchmark for evaluating large language models on repository-level code generation for feature implementation. The main research objective is to assess the ability of LLMs to perform incremental development within code repositories by adding new features. The key methodology involves collecting pull requests from 83 GitHub repositories, filtering them based on rules and intent, and pairing code changes with unit tests. Primary results show that the best-performing LLM (DeepSeek-R1) resolves only 9.92% of task instances in the Oracle and Detailed prompt settings. The principal implication for AI practitioners is that current LLMs face significant challenges in repository-level incremental code development, requiring improvements in handling long contexts and complex code modifications. |
| AlphaDrive: Unleashing the Power of VLMs in Autonomous Driving via | ||
| Reinforcement Learning and Reasoning (Read more on arXiv or HuggingFace) | Qian Zhang, xinggangw, wenyuliu, Atan-0221, rb93dett | AlphaDrive is a VLM-based framework for autonomous driving planning that leverages reinforcement learning and reasoning. The main research objective is to investigate how reinforcement learning (RL) and reasoning can be applied to enhance the performance of vision-language models (VLMs) in autonomous driving planning while reducing training costs. The key methodology involves a two-stage training strategy combining supervised fine-tuning (SFT) with Group Relative Policy Optimization (GRPO)-based RL, using four custom-designed rewards for planning accuracy, action weighting, diversity, and output format. Primary results show AlphaDrive significantly improves planning accuracy by 25.52% compared to an SFT-trained model, and outperforms SFT by 35.31% with only 20% of the training data. For AI practitioners, AlphaDrive demonstrates the efficacy of integrating GRPO-based RL and a two-stage training approach with planning-specific rewards, offering a method to improve planning performance and training efficiency of VLMs in autonomous driving. |
| DreamRelation: Relation-Centric Video Customization (Read more on arXiv or HuggingFace) | Shiwei Zhang, Shuaishuai0219, lloong, JacobYuan, weilllllls | DreamRelation is a novel method for customizing relational video content based on a small set of exemplar videos. The main research question is: How can we decouple relations and subject appearances while accurately modeling relational dynamics to enhance generalizability in customized video generation? The key methodology involves relational decoupling learning, using a relation LoRA triplet and hybrid mask training strategy to separate relations from appearances, and relational dynamics enhancement via a space-time relational contrastive loss. The primary results show that DreamRelation achieves a relation accuracy of 0.4452 ± 0.01, outperforming baselines like direct LoRA finetuning (0.3258 ± 0.05) and MotionInversion (0.3151 ± 0.03). The principal implication for AI practitioners is that by effectively disentangling relational dynamics from subject appearances, DreamRelation provides a more precise and generalizable approach to relational video customization, enabling applications such as creation of diverse human-like animal interactions in novel domains. |
| Agent models: Internalizing Chain-of-Action Generation into Reasoning | ||
| models (Read more on arXiv or HuggingFace) | Jitao Sang, Xinyan Wen, Jiangming Shu, tzteyang, TokerZ | Large Agent Models (LAMs) internalize Chain-of-Action generation, allowing autonomous decisions on when and how to use external tools. The research objective is to develop a framework, AutoCoA, that enables reasoning models to autonomously generate Chain-of-Action (CoA) for improved task completion. The methodology combines supervised fine-tuning (SFT) with reinforcement learning (RL), including step-level action triggering and trajectory-level CoA optimization, and utilizes an internal world model. Primary results show AutoCoA-trained agent models achieve a 33.9% Exact Match accuracy on multi-hop QA tasks like Bamboogle, significantly outperforming ReAct-based workflows (15.2%). Principal implication for AI practitioners: The AutoCoA framework provides a method to train agent models that show enhanced performance by reducing reliance on externally prompted actions. |
| WritingBench: A Comprehensive Benchmark for Generative Writing (Read more on arXiv or HuggingFace) | SHaopeng Lai, Chenliang Li, Ming Yan, Jiahao Mei, AQuarterMile | WritingBench, a new benchmark, evaluates large language models (LLMs) across diverse writing tasks, incorporating a query-dependent evaluation framework. The main objective is to create a comprehensive benchmark for evaluating LLMs on diverse, real-world generative writing tasks and to propose a query-dependent evaluation framework. Key methodology involves a four-stage query construction pipeline leveraging LLMs and human refinement, and a query-dependent evaluation framework using dynamically generated, instance-specific criteria scored by a fine-tuned critic model. Primary results show that the query-dependent evaluation framework achieves 83% human alignment, significantly surpassing static-criteria baselines (65%, 59%). Principal implication for AI practitioners is that WritingBench provides a more nuanced and robust evaluation tool for writing-focused LLMs, and the query-dependent evaluation approach can lead to more accurate and human-aligned assessment of generative writing capabilities, guiding improvements in model development. |
| SurveyForge: On the Outline Heuristics, Memory-Driven Generation, and | ||
| Multi-dimensional Evaluation for Automated Survey Writing (Read more on arXiv or HuggingFace) | Bin Wang, Renqiu Xia, Jiakang Yuan, Shiyang Feng, Xiangchao Yan | SurveyForge is a framework for automated survey paper generation using heuristic outline generation, memory-driven content creation, and multi-dimensional evaluation. The main research objective is to address the quality gap between AI-generated and human-written surveys, focusing on outline structure, citation accuracy, and content comprehensiveness. The methodology involves a two-stage process: heuristic outline generation based on human-written survey patterns and relevant literature, followed by memory-driven content generation using a scholar navigation agent with temporal-aware reranking. Key results show that SurveyForge outperforms the baseline AutoSurvey in reference coverage (0.40 vs 0.23 using Claude-3-Haiku) and overall content quality (76.34 vs 73.87). AI practitioners can use SurveyForge to create comprehensive, structured survey papers more efficiently and with higher literature coverage than existing methods. |
| Vision-R1: Incentivizing Reasoning Capability in Multimodal Large | ||
| Language Models (Read more on arXiv or HuggingFace) | Zheyu Ye, Shaosheng Cao, Zijie Zhai, Bohan Jia, Wenxuan Huang | Vision-R1, a multimodal large language model (MLLM), enhances reasoning by combining cold-start initialization with reinforcement learning (RL). The main research objective is to enhance the reasoning capability of MLLMs using RL, addressing limitations of direct RL training. Key methodology used is Modality Bridging with Progressive Thinking Suppression Training (PTST) and Group Relative Policy Optimization (GRPO) using the hard formatting result reward function. Primary results show Vision-R1-7B achieves 73.5% accuracy on the MathVista benchmark, which is only 0.4% lower than the leading model, OpenAI 01. Principal implication for AI practitioners: Using cold-start initialization with a high-quality multimodal Chain-of-Thought (CoT) dataset, combined with the PTST strategy during RL, improves the mathematical reasoning of MLLMs, providing a viable training approach. |
| LLaVE: Large Language and Vision Embedding Models with Hardness-Weighted | ||
| Contrastive Learning (Read more on arXiv or HuggingFace) | Jinsong Su, Jie Zhou, Fandong Meng, lqniu, zhibinlan | LLaVE is a multimodal embedding model framework that improves performance by focusing on hard negative pairs during contrastive learning. The main research objective is to address the challenge that existing Large Multimodal Model (LMM)-based embedding models struggle to distinguish hard negative pairs effectively when trained with the standard InfoNCE loss. The key methodology involves hardness-weighted contrastive learning, using a reward model to dynamically assign larger weights to harder negative pairs and cross-device negative sample gathering. Primary results show that LLaVE-7B achieves a 6.2 point performance improvement on the MMEB benchmark over the previous state-of-the-art model. The principal implication for AI practitioners is that employing hardness-weighted contrastive learning with LMMs can create more powerful and generalizable multimodal embedding models, with the framework applied and scaling well to diverse datasets. |
| MedAgentsBench: Benchmarking Thinking Models and Agent Frameworks for | ||
| Complex Medical Reasoning (Read more on arXiv or HuggingFace) | Jiapeng Chen, Jiwoong Sohn, Daniel Shao, wshi83, RTT1 | This paper introduces MEDAGENTSBENCH, a new benchmark for evaluating large language models (LLMs) on complex medical reasoning tasks. The main research objective is to assess the performance of advanced thinking models and agent frameworks in challenging medical scenarios requiring multi-step reasoning. The key methodology involves constructing a dataset of 862 questions from seven established medical datasets, using adversarial filtering to select difficult questions and evaluating various LLMs and agent-based methods using standardized prompts and metrics. A primary result is that DEEPSEEK-R1 achieved the highest scores on five of the datasets and the accuracy values are highlighted in the papers such as MedMCQA: 31.0%, MMLU: 43.8%, MMLU-Pro: 37.0%, MedExQA: 26.0%, and MedXpertQA-U: 26.0%. The principal implication for AI practitioners is that thinking models, like DEEPSEEK-R1, and search-based agent methods, like AFLOW, offer superior performance in complex medical reasoning and better cost-efficiency than the other LLMs and agents, guiding model selection for real-world applications. |
| PE3R: Perception-Efficient 3D Reconstruction (Read more on arXiv or HuggingFace) | Xinchao Wang, Shizun Wang, Jie Hu | PE3R is a novel framework for efficient and accurate 3D semantic reconstruction from 2D images without requiring 3D data or camera parameters. The main research objective is to develop a method for 3D semantic reconstruction that generalizes across diverse scenes and objects, achieves high perception accuracy, and operates at high speed. The key methodology involves a feed-forward architecture incorporating pixel embedding disambiguation, semantic field reconstruction, and global view perception modules. The framework achieves a minimum 9-fold speedup in 3D semantic field reconstruction compared to previous methods, along with improved accuracy and precision. For AI practitioners, PE3R provides a faster and more generalizable approach to 3D scene understanding from 2D images, enabling applications in scenarios with limited 3D data availability. |
| Effective and Efficient Masked Image Generation Models (Read more on arXiv or HuggingFace) | Jun Zhou, Jun Hu, Xiaolu Zhang, Jingyang Ou, yyyou | eMIGM unifies and improves masked image and diffusion models for efficient, high-quality image generation. The main research objective is to systematically explore the design space of training and sampling in masked image generation models, identifying key factors contributing to performance and efficiency. The key methodology involves unifying masked image modeling and masked diffusion models, then exploring variations in masking distributions, weighting functions, conditional distributions, and sampling strategies like time-interval classifier-free guidance. A primary result is that on ImageNet 512x512, eMIGM-L surpasses EDM2 with an FID of 1.77, using only 60% of the function evaluations. The principal implication is that AI practitioners can leverage eMIGM’s unified framework and optimized training/sampling strategies to achieve state-of-the-art image generation with significantly reduced computational cost. |
| Seg-Zero: Reasoning-Chain Guided Segmentation via Cognitive | ||
| Reinforcement (Read more on arXiv or HuggingFace) | Fanbin Lu, Zihao Yue, Zhisheng Zhong, Bohao Peng, Yuqi Liu | Seg-Zero is a framework for reasoning segmentation that leverages cognitive reinforcement learning to achieve zero-shot generalization. The main research objective is to develop a segmentation model that exhibits strong generalization and explicit reasoning capabilities without relying on supervised fine-tuning with explicit reasoning data. The key methodology involves a decoupled architecture with a reasoning model (MLLM) generating a chain-of-thought and positional prompts, and a segmentation model producing pixel-level masks, trained using reinforcement learning with a novel reward mechanism. Primary results show that Seg-Zero-7B achieves a zero-shot performance of 57.5 on the ReasonSeg benchmark, surpassing the prior LISA-7B by 18%. The principal implication for AI practitioners is that pure reinforcement learning, guided by a well-designed reward mechanism, can induce emergent reasoning in segmentation models, improving generalization across domains without explicit reasoning supervision. |
| BlackGoose Rimer: Harnessing RWKV-7 as a Simple yet Superior Replacement | ||
| for Transformers in Large-Scale Time Series Modeling (Read more on arXiv or HuggingFace) | xiaol, Alic-Li | Rimer replaces the transformer backbone in time series models with RWKV-7, achieving superior performance and efficiency. The research objective was to develop a more efficient and scalable time-series model compared to transformer-based approaches. The methodology involved integrating RWKV-7’s time mix and channel mix components into the transformer-based time series model, Timer. The Rimer model achieved a 1.13x to 43.3x performance improvement and a 4.5x reduction in training time with 1/23 the parameters of the original Timer model. AI practitioners can leverage Rimer for improved performance and reduced computational cost in large-scale time series modeling tasks, benefiting from its compatibility with both AMD and NVIDIA GPUs. |
| This Is Your Doge, If It Please You: Exploring Deception and Robustness | ||
| in Mixture of LLMs (Read more on arXiv or HuggingFace) | Ilija Bogunovic, Sangwoong Yoon, Llwo | Mixture of LLM Agents (MoA) architectures are vulnerable to significant performance degradation when even a single agent acts deceptively. This paper explores the robustness of Mixture of LLM Agents (MoA) against deceptive agents that provide misleading responses. The authors evaluate MoA’s performance on AlpacaEval 2.0 and QUALITY benchmarks, introducing deceptive agents into the multi-agent system. They find that introducing a single deceptive agent into a 7-agent MoA reduces the length-controlled win rate on AlpacaEval 2.0 from 49.2% to 37.9%. AI practitioners should implement defense mechanisms, such as those proposed in this paper, to mitigate the risks associated with deceptive agents in multi-agent LLM systems. |
| Efficient Distillation of Classifier-Free Guidance using Adapters (Read more on arXiv or HuggingFace) | msadat97, cristianpjensen | Adapter Guidance Distillation (AGD) efficiently simulates classifier-free guidance (CFG) in diffusion models using lightweight adapters, doubling sampling speed while maintaining quality. The main research objective is to mitigate the computational cost of CFG in conditional diffusion models, which doubles the number of neural function evaluations per inference step. The key methodology involves training lightweight adapters on CFG-guided trajectories to approximate CFG in a single forward pass, keeping the base diffusion model frozen. AGD achieves a FID score of 5.03 on class-conditional ImageNet generation using DiT, outperforming CFG (FID 5.30) and matching or exceeding the performance across various other tested architectures. For AI practitioners, AGD enables faster sampling from diffusion models with performance similar to or exceeding the use of CFG, and distilling large models such as Stable Diffusion XL on a single consumer GPU. |
| State-offset Tuning: State-based Parameter-Efficient Fine-Tuning for | ||
| State Space Models (Read more on arXiv or HuggingFace) | Hyung Il Koo, Minjae Lee, Yuchen Zeng, Kevin Galim, Wonjun Kang | State-offset Tuning is a new parameter-efficient fine-tuning method for State Space Models (SSMs) that directly modifies state-related features. The main research objective is to develop a more effective parameter-efficient fine-tuning (PEFT) method for SSMs than existing prompt-based methods. The key methodology is State-offset Tuning, which adds a learnable, constant state-offset to the hidden state at each timestep within the SSM module. Primary results show State-offset Tuning (h) achieved 59.9 execution accuracy on the Spider dataset, outperforming other PEFT methods with comparable parameter budgets. AI practitioners can use State-offset Tuning to efficiently adapt pretrained SSMs to downstream tasks, achieving performance comparable to full fine-tuning with significantly fewer trainable parameters. |
| Should VLMs be Pre-trained with Image Data? (Read more on arXiv or HuggingFace) | Igor Vasiljevic, Kushal Arora, Samir Yitzhak Gadre, Jean Mercat, Sedrick Keh | Vision-Language Models (VLMs) can be improved by incorporating image data during pre-training, before the model is fully pre-trained with text. The main research question is when and how image data should be introduced during VLM pre-training to optimize downstream performance on vision-language and text-only tasks. Researchers trained approximately 300 models, systematically varying text-only pre-training amounts, image-text ratios, and fine-tuning stages using a decoder-only transformer architecture with a frozen image encoder. A key finding is that, for a 1B parameter model, introducing visual tokens 80% of the way through pre-training leads to a 2% average improvement on vision-language tasks compared to introducing them after full pre-training. The results suggest that AI practitioners should integrate image data earlier in VLM pre-training, but not immediately, to maintain text performance, instead of following traditional separate training phases. |
| WISE: A World Knowledge-Informed Semantic Evaluation for Text-to-Image | ||
| Generation (Read more on arXiv or HuggingFace) | Peng Jin, Bin Lin, Mengren Zheng, Munan Ning, Yuwei Niu | The paper introduces WISE, a new benchmark for evaluating text-to-image (T2I) models’ ability to integrate world knowledge and complex semantics, along with a new metric called WiScore. The main research objective is to assess how well T2I models can generate images that accurately reflect complex semantic understanding and world knowledge, going beyond simple text-image alignment. The key methodology involves a benchmark of 1000 prompts across 25 sub-domains of cultural common sense, spatio-temporal reasoning, and natural science, and evaluates 20 T2I models (10 dedicated, 10 unified) using a novel quantitative metric, WiScore, which assesses knowledge-image alignment. A key result is that the FLUX.1-dev model achieved the best overall WiScore of 0.50, while dedicated T2I models generally outperformed unified multimodal models in leveraging world knowledge. The primary implication is that AI practitioners need to develop enhanced methods for incorporating and applying world knowledge in T2I models, as existing models demonstrate significant limitations in this area. |
| ProBench: Judging Multimodal Foundation Models on Open-ended | ||
| Multi-domain Expert Tasks (Read more on arXiv or HuggingFace) | Liu Liu, Bei Chen, Haoning Wu, dxli1, HelloKKMe | ProBench is a benchmark for evaluating multimodal foundation models on expert-level, open-ended tasks using MLLM-as-a-Judge. The main research objective is to assess the capabilities of multimodal large language models (MLLMs) on complex, real-world professional tasks requiring expert knowledge and advanced reasoning. The key methodology involves curating a dataset of 4,000 high-quality, open-ended user queries submitted by professionals across 10 fields and 56 sub-fields, and evaluating 24 MLLMs using an MLLM-as-a-Judge approach. The primary results reveal that while the best open-source models rival proprietary ones, ProBench presents significant challenges, and that the MLLM-as-a-Judge evaluation shows 79.9% agreement with human experts. A principal implication for AI practitioners is that current MLLMs still struggle with visual perception, textual understanding, domain knowledge, and advanced reasoning, highlighting the specific areas requiring focused development for improved performance on real-world expert tasks. |
| Zero-AVSR: Zero-Shot Audio-Visual Speech Recognition with LLMs by | ||
| Learning Language-Agnostic Speech Representations (Read more on arXiv or HuggingFace) | Yong Man Ro, Stavros Petridis, Chae Won Kim, Minsu Kim, JeongHun0716 | This paper explores zero-shot audio-visual speech recognition (AVSR) using language-agnostic speech representations and Large Language Models (LLMs). The main research objective is to enable speech recognition in target languages without any audio-visual speech data in those languages. The key methodology involves an Audio-Visual Speech Romanizer (AV-Romanizer) to predict Roman text and uses pre-trained LLMs and multi-task training to convert it into language-specific graphemes. The Zero-AVSR framework, trained on a new Multilingual Audio-Visual Romanized Corpus (MARC) of 2,916 hours, achieves a 25.2% average WER on the MuAViC dataset. AI practitioners can leverage this framework to expand language support in AVSR systems without requiring target-language speech data. |
| Words or Vision: Do Vision-Language Models Have Blind Faith in Text? (Read more on arXiv or HuggingFace) | Bryan Hooi, Tri Cao, Ailin Deng, ryanchen42 | Vision-Language Models (VLMs) exhibit a “blind faith in text” phenomenon, disproportionately trusting textual data over visual data when inconsistencies arise. The main research question is: How do VLMs handle inconsistencies between visual and textual inputs? The key methodology involves introducing textual variations (match, corruption, irrelevance) to four vision-centric tasks and evaluating ten VLMs. A primary result is that Qwen2-VL-7B’s accuracy on VQAv2, DocVQA, and MathVista drops to approximately 50% of its original levels under text corruption. The principal implication for AI practitioners is that balanced training and careful consideration of modality interactions are crucial for enhancing VLM robustness and reliability when handling multi-modal data inconsistencies, especially in safety-critical applications. |
| Detection Avoidance Techniques for Large Language Models (Read more on arXiv or HuggingFace) | Gabi Dreo Rodosek, Joao A. G. Schneider, Florian Steuber, SinclairSchneider | This research investigates methods to bypass large language model (LLM) detection systems. The main research objective is to explore the vulnerability of various LLM detection techniques to different evasion strategies. The key methodology involves modifying generative model parameters (temperature, sampling), applying reinforcement learning to fine-tune models, and using paraphrasing models. Primary results show that paraphrasing led to a >90% evasion rate of zero-shot detectors like DetectGPT, reducing detection from 88.6% to 8.7% in one experiment. Principal implication for AI practitioners is that current LLM detection classifiers can be easily bypassed, requiring further research into more robust detection and adaptive detection methods. |
| DiffCLIP: Differential Attention Meets CLIP (Read more on arXiv or HuggingFace) | Bernard Ghanem, Hasan Abed Al Kader Hammoud | DiffCLIP extends CLIP with a differential attention mechanism to improve vision-language model performance. The main research question is whether differential attention can be adapted to vision-language models to improve their ability to focus on relevant features across modalities. The key methodology involves integrating differential attention, which subtracts complementary attention distributions, into CLIP’s dual-encoder (image and text) architecture. DiffCLIP outperforms standard CLIP on image-text retrieval, with a 1.2% average improvement on image retrieval using the CC3M dataset. AI practitioners can use DiffCLIP as a lightweight, parameter-efficient addition to CLIP that enhances performance across various vision-language tasks, including few-shot, zero-shot, and robustness benchmarks. |
| Novel Object 6D Pose Estimation with a Single Reference View (Read more on arXiv or HuggingFace) | Hui Yang, Jin Zheng, Kai Zeng, Wei Sun, JianLiu99 | SinRef-6D is a framework for estimating the 6D pose of novel objects using only a single RGB-D reference view. The main research objective is to develop a CAD-model-free and dense-reference-view-free method for novel object 6D pose estimation that is scalable and efficient. The key methodology involves iteratively establishing point-wise alignment in the camera coordinate system using state space models (SSMs) for feature encoding, and RGB and points SSMs to capture spatial information. The primary results show that SinRef-6D achieves 90.3% on the LineMod dataset using the ADD-0.1d metric, which is on par with some CAD-based and superior compared to single-reference view based methods. This implies that AI practitioners can achieve accurate 6D pose estimation for novel objects without requiring CAD models or multiple reference views, reducing computational overhead and manual efforts, and enhance the practicality in real-world settings. |
| Beyond RAG: Task-Aware KV Cache Compression for Comprehensive Knowledge | ||
| Reasoning (Read more on arXiv or HuggingFace) | Fabio Petroni, Orion Weller, papotti, giulio98 | This paper introduces a task-aware KV cache compression method for large language models to improve reasoning over large external knowledge corpora. The main research objective is to develop a query-agnostic compression technique that preserves efficiency while maintaining competitive performance compared to query-aware compression and Retrieval-Augmented Generation (RAG). The key methodology involves precomputing a compressed key-value (KV) cache, guided by a task description and optionally few-shot examples, which can be reused for any query within the defined task domain. The approach improves accuracy by up to 7 absolute points over RAG on LongBench v2 with a 30x compression rate, and reduces inference latency. The principal implication is that AI practitioners can leverage task-aware KV cache compression to enable more efficient and comprehensive reasoning over large corpora in LLM applications, outperforming RAG in broad-knowledge tasks. |
| HumanMM: Global Human Motion Recovery from Multi-shot Videos (Read more on arXiv or HuggingFace) | Jing Lin, Zhuokai Zhao, Ling-Hao Chen, Guanlin Wu, Yuhong Zhang | HumanMM is a framework for reconstructing 3D human motion in world coordinates from multi-shot videos, addressing challenges like shot transitions and occlusions. The main research objective is to reconstruct long-sequence 3D human motion in world coordinates from in-the-wild videos with multiple shot transitions. The key methodology integrates enhanced camera pose estimation (using a modified LEAP-VO with human masking) with Human Motion Recovery (HMR), incorporating a shot transition detector, an alignment module for pose and orientation continuity across shots, and a custom motion integrator. The proposed method achieved a PA-MPJPE of 36.82 on the ms-AIST subset of the created ms-Motion dataset, outperforming existing methods. For AI practitioners, HumanMM provides a novel, robust method for reconstructing realistic human motion in world coordinates from multi-shot videos, enabling improved motion generation and understanding applications. |
| YOLOE: Real-Time Seeing Anything (Read more on arXiv or HuggingFace) | Jungong Han, Zijia Lin, Hui Chen, Lihao Liu, Ao Wang | YOLOE is a unified, efficient object detection and segmentation model that supports diverse open prompt mechanisms, achieving real-time performance. The main research objective is to develop a single model capable of detecting and segmenting arbitrary objects guided by text prompts, visual cues, or without prompts, with high efficiency and accuracy. The key methodology involves Re-parameterizable Region-Text Alignment (RepRTA) for text prompts, Semantic-Activated Visual Prompt Encoder (SAVPE) for visual prompts, and Lazy Region-Prompt Contrast (LRPC) for prompt-free scenarios, all built upon YOLO architectures. On LVIS, YOLOE-v8-S surpasses YOLO-Worldv2-S by 3.5 AP with 3x less training cost and 1.4x inference speedup. The principal implication for AI practitioners is that YOLOE provides a strong baseline and framework for developing real-time, open-prompt-driven vision applications, streamlining development by using a single efficient model for diverse prompt types. |
| RePO: ReLU-based Preference Optimization (Read more on arXiv or HuggingFace) | Jinyang Gao, Xue Wang, Kexin Huang, Junkang Wu, xiangwang1223 | RePO introduces a simplified offline preference optimization algorithm for aligning large language models (LLMs) with human preferences. The main research question is whether a simpler offline preference optimization algorithm can be developed that achieves comparable or better performance than existing methods. The key methodology involves using ReLU-based max-margin loss and reference-free reward margins, eliminating the need for the hyperparameter β in SimPO and simplifying the log-sigmoid activation. Primary results show that RePO outperforms DPO and SimPO across multiple base models on AlpacaEval 2, achieving a win rate of 51.1% on Llama3-8B and 66.6% on Gemma2-9B, and it require tuning only one hyperparameter, γ. For AI practitioners, RePO offers a more streamlined and efficient approach to preference optimization, requiring less hyperparameter tuning while achieving competitive or superior performance in LLM alignment. |
| Adaptive Audio-Visual Speech Recognition via Matryoshka-Based Multimodal | ||
| LLMs (Read more on arXiv or HuggingFace) | Stavros Petridis, Minsu Kim, Umberto Cappellazzo | Llama-MTSK, a Matryoshka-based Multimodal LLM, enables adaptive audio-visual speech recognition with flexible token allocation. The research objective is to create an audio-visual speech recognition (AVSR) system that dynamically adjusts computational efficiency and performance at inference time using a single model. The methodology involves encoding audio-visual representations at multiple granularities using Matryoshka Representation Learning and fine-tuning a pre-trained LLM with three LoRA-based Matryoshka strategies. On the LRS3 dataset, Llama-MTSK achieved a Word Error Rate (WER) of 2.3% using the SS configuration with an audio compression rate of 4 and video compression of 2, outperforming independently trained models. AI practitioners can use Llama-MTSK to deploy AVSR models that efficiently adapt to various computational constraints and accuracy requirements without retraining. |
| Escaping Plato’s Cave: Towards the Alignment of 3D and Text Latent | ||
| Spaces (Read more on arXiv or HuggingFace) | Qixing Huang, Diego Gomez, Luca Moschella, Souhail Hadgi, teelinsan | This paper investigates the alignment between latent spaces of 3D and text encoders, finding that subspace projection improves cross-modal performance. The main research objective is to explore the possibility of a posteriori alignment of representations obtained from uni-modal 3D encoders compared to text-based feature spaces. The key methodology involves combining Canonical Correlation Analysis (CCA) for subspace selection with affine transformation and local CKA for alignment of 3D and text features. A primary result is that the affine + subspace projection method achieves a top-5 retrieval accuracy of 42.2% between uni-modal PointBert and RoBERTa, significantly higher than without subspace projection. Principal implication for AI practitioners that aligning lower-dimensional subspaces of 3D and text representations enables cross-modal applications, like matching and retrieval tasks, without expensive joint training, and offers a new tool. |
| NeuGrasp: Generalizable Neural Surface Reconstruction with Background | ||
| Priors for Material-Agnostic Object Grasp Detection (Read more on arXiv or HuggingFace) | Xudong Zheng, Wenzhe He, Chao Li, Yinghao Cai, KianYale | NeuGrasp is a generalizable neural surface reconstruction method that uses background priors for 6-DoF robotic grasp detection of objects with various material properties. The main research objective is to develop a method for robust, material-agnostic grasp detection in scenes with transparent and specular objects from sparse views within a narrow field of view. The key methodology involves integrating transformers and global prior volumes within a neural implicit surface framework, using residual feature enhancement and an occupancy-prior volume to distinguish foreground objects. Primary results show that NeuGrasp achieved a success rate of 86.3% and declutter rate of 81.0% in simulation experiments on packed scenes with transparent and specular objects, outperforming baselines. AI practitioners can apply NeuGrasp to achieve accurate grasp detection using a small amount of RGB image input. |
Papers for 2025-03-10
| Title | Authors | Summary |
|---|---|---|
| Unified Reward Model for Multimodal Understanding and Generation (Read more on arXiv or HuggingFace) | Cheng Jin, Hao Li, Jiaqiwang, yuhangzang, CodeGoat24 | This paper proposes UNIFIEDREWARD, a unified reward model for assessing both multimodal understanding and generation, enabling pairwise ranking and pointwise scoring for vision model preference alignment. The main research objective is to develop a single reward model adaptable across diverse visual tasks (image/video generation and understanding) and to demonstrate its effectiveness in aligning vision models with human preferences. The key methodology involves training a Vision Language Model (VLM) on a newly constructed, large-scale human preference dataset, then using the trained model to curate preference data for Direct Preference Optimization (DPO) of VLMs and diffusion models. Primary results show that UNIFIEDREWARD achieves 66.5% macro accuracy on VLRewardBench for image understanding assessment, outperforming existing methods. The principal implication for AI practitioners is that they can leverage this unified reward model and associated training pipeline to improve the alignment of vision models with human preferences across a range of generation and understanding tasks, leading to better output quality and overall better evaluation. |
| EuroBERT: Scaling Multilingual Encoders for European Languages (Read more on arXiv or HuggingFace) | caiocorro, ayoubhammal, DuarteMRAlves, hgissbkh, Nicolas-BZRD | EuroBERT, a family of multilingual encoder models, outperforms existing alternatives on various tasks, spanning multiple languages, mathematics, and coding. The main research objective is to revisit the development of multilingual encoders by leveraging recent advances from decoder models and examining design choices in data composition and training. Methodology includes building a 5T-token multilingual dataset, using a masked language modeling objective, and employing a two-phase training pipeline (pre-training and annealing). EuroBERT-2.1B achieves the highest performance among all systems, ranking first on 7 of 12 multilingual benchmarks, outperforming XLM-ROBERTa-XL. This implies that AI practitioners can use EuroBERT models for improved performance in NLP tasks, especially retrieval, classification and evaluation tasks across European and other widely spoken languages, even with models smaller than pre-existing state-of-the-art. |
| Sketch-of-Thought: Efficient LLM Reasoning with Adaptive Cognitive-Inspired Sketching (Read more on arXiv or HuggingFace) | Sung Ju Hwang, jinheon, saytes | Sketch-of-Thought (SoT) is a prompting framework that improves large language model (LLM) reasoning efficiency by using concise, structured intermediate steps inspired by human cognitive processes. The main research objective is to reduce the computational cost of LLM reasoning while maintaining or improving accuracy compared to verbose methods like Chain-of-Thought (CoT). The key methodology involves three cognitive-inspired paradigms (Conceptual Chaining, Chunked Symbolism, and Expert Lexicons) dynamically selected by a lightweight router model based on query characteristics. Primary results show that SoT reduces token usage by up to 76% across 15 reasoning datasets with negligible accuracy impact, and in some cases, even improved accuracy. Principal implication for AI practitioners: SoT offers a practical method to reduce computational costs and latency in LLM-based reasoning applications without significant performance degradation, enabling deployment in resource-constrained environments. |
| Forgetting Transformer: Softmax Attention with a Forget Gate (Read more on arXiv or HuggingFace) | Aaron Courville, littleowen, nikishin, zhixuan-lin | Forgetting Transformer (FoX) introduces a forget gate into the softmax attention mechanism of Transformers to improve performance, particularly in length extrapolation and short-context tasks. The main research objective is to determine if incorporating a data-dependent forget gate into Transformers can improve their performance on both long and short-context tasks. The key methodology involves modifying the softmax attention mechanism by down-weighting unnormalized attention scores based on a learned, data-dependent forget gate, implemented efficiently using a modification of the FlashAttention algorithm. Primary results show that FoX outperforms the standard Transformer in long-context language modeling, achieving a per-token loss of approximately 1.53 compared to Transformer’s ~1.58 at the 32,000 token index (Figure 2, left) in a configuration with a 760M parameter. Principal implication for AI practitioners is that the FoX architecture could improve performance in some sequential tasks and serves as a strong baseline, especially in tasks needing to balance long- and short-context information, with the Pro architecture being the most promising. |
| VideoPainter: Any-length Video Inpainting and Editing with Plug-and-Play Context Control (Read more on arXiv or HuggingFace) | Zhaoyang Zhang, yshan2u, Ljzycmd, juxuan27, BianYx | VideoPainter introduces a dual-branch framework for text-guided video inpainting and editing that maintains ID consistency in long videos. The research objective is to develop a method for video inpainting that addresses challenges such as generating fully masked objects, balancing background preservation with foreground generation, and maintaining identity consistency over long videos. The key methodology involves a lightweight context encoder within a dual-branch Diffusion Transformer architecture, and a novel inpainting region ID resampling technique. Primary results include achieving a FVID score of 0.09 on the VPBench dataset for standard video inpainting surpassing competing methods. The principal implication is that AI practitioners can leverage this framework for more effective and controllable video inpainting and editing, with robust performance in generating long videos and maintaining object identity due to its sampling technique. |
| R1-Searcher: Incentivizing the Search Capability in LLMs via Reinforcement Learning (Read more on arXiv or HuggingFace) | jrwen, TimothyCzp, EliverQ, Boru, XXsongLALA | R1-Searcher is a two-stage outcome-based reinforcement learning (RL) framework to enhance search capabilities in large language models (LLMs). The main research objective is to enable LLMs to autonomously invoke external search systems for accessing additional knowledge during reasoning. The key methodology is a two-stage RL approach: first incentivizing retrieval invocation, then rewarding accurate answer generation using retrieved information, with RAG-based rollout and retrieval mask-based loss calculation. The primary results are, using Qwen-2.5-7B-Base, R1-Searcher outperforms ReARTeR by 48.22% on HotpotQA and by 21.72% on 2Wiki. The principal implication is that AI practitioners can use this RL method to train LLMs to effectively integrate external search, improving reasoning and generalization, even in out-of-domain and online search scenarios. |
| R1-Omni: Explainable Omni-Multimodal Emotion Recognition with Reinforcing Learning (Read more on arXiv or HuggingFace) | Xihan Wei, Liefeng, StarJiaxing | The paper introduces R1-Omni, an omni-multimodal model for emotion recognition using Reinforcement Learning with Verifiable Reward (RLVR). The main research objective is to investigate the potential of RLVR in enhancing emotion recognition performance in a video-based, omni-multimodal setting (incorporating both visual and audio data). Key methodology involves applying RLVR with Group Relative Policy Optimization (GRPO) to a HumanOmni model, using a verifiable reward function that combines accuracy and format rewards, after a cold start using the EMER dataset. Primary results show that R1-Omni achieves a UAR of 65.83% and a WAR of 56.27% on the DFEW dataset, outperforming Supervised Fine-Tuning (SFT) models. For AI practitioners, the principal implication is that RLVR can significantly improve the reasoning capability, emotion recognition accuracy, and generalization ability of multimodal large language models in tasks such as emotion recognition, without explicit reasoning-process supervision. |
| TrajectoryCrafter: Redirecting Camera Trajectory for Monocular Videos via Diffusion Models (Read more on arXiv or HuggingFace) | Mark YU, yshan2u, Doubiiu, wbhu-tc | TrajectoryCrafter redirects camera trajectories in monocular videos using diffusion models. The research objective is to generate high-fidelity videos from monocular inputs with user-defined camera trajectories, ensuring 4D consistency. The methodology uses a dual-stream conditional video diffusion model that integrates point cloud renders and source videos, trained on a hybrid dataset of monocular and multi-view data using a double-reprojection strategy. The method achieved a PSNR of 14.24 on the iPhone multi-view dataset, outperforming existing methods. AI practitioners can use this framework to generate videos with controlled camera movements from single-camera footage, enhancing video content creation and editing capabilities. |
| BEHAVIOR Robot Suite: Streamlining Real-World Whole-Body Manipulation for Everyday Household Activities (Read more on arXiv or HuggingFace) | Ruohan Zhang, jiajunwu, cgokmen, yjze, yunfanj | BEHAVIOR ROBOT SUITE (BRS) is a framework for learning whole-body manipulation for household tasks. The main research objective is to identify and address the key capabilities required for robots to perform everyday household activities successfully. The key methodology used is a combination of a cost-effective whole-body teleoperation interface (JoyLo) for data collection, and a novel imitation learning algorithm (Whole-Body VisuoMotor Attention policy, WB-VIMA) for modeling coordinated whole-body actions. The trained WB-VIMA policies achieved an average success rate of 58% and a peak success rate of 93% across five challenging household tasks. For AI practitioners, BRS provides an integrated framework for whole-body manipulation, offering open-source hardware and software to facilitate data collection and policy learning for real-world robotic applications, streamlining the development of robots capable of diverse household tasks. |
| RuCCoD: Towards Automated ICD Coding in Russian (Read more on arXiv or HuggingFace) | Vladimir Makharev, Airat Valiev, Ivan Sviridov, Andrey Sakhovskiy, Aleksandr Nesterov | This paper introduces RuCCoD, a new Russian-language dataset for automated ICD coding, and benchmarks several state-of-the-art models for this task. The main research objective is to investigate the feasibility of automating clinical coding in Russian, a language with limited biomedical resources. The key methodology involves training and evaluating BERT-based, LLaMA-based (with LoRA and RAG), models on the RuCCoD dataset, and applying the best model to a larger EHR dataset for diagnosis prediction. Primary results show that pre-training a Longformer model on automatically assigned ICD codes (using the new proposed dataset) yields a 28% higher macro-averaged F1-score for diagnosis prediction compared to using physician-assigned codes. For AI practitioners, using an automated pipeline to generate ICD codes for model training can significantly improve diagnosis prediction accuracy in resource-limited languages like Russian. |
| TinyR1-32B-Preview: Boosting Accuracy with Branch-Merge Distillation (Read more on arXiv or HuggingFace) | lwher1996, yuhanwuuu, xiaoqijiang, zhaoguangxiang, lincharliesun | TinyR1-32B-Preview is a new language model that improves accuracy on reasoning tasks using a branch-merge distillation approach. The main objective is to create a smaller, high-performing Large Language Model (LLM) with reduced computational cost and time, compared to traditional distillation methods. The key methodology involves a two-phase distillation: (1) “Branch Phase,” where a large teacher model’s knowledge is selectively distilled into specialized student models via domain-specific supervised fine-tuning, and (2) “Merge Phase,” where specialized models are combined using Arcee Fusion. The primary result is that TinyR1-32B-Preview outperforms DeepSeek-R1-Distill-Qwen-32B by 5.5 points in Mathematics on the AIME 2024 benchmark. The principal implication is to provide AI practioners, a scalable solution for creating smaller, more efficient LLMs, and a means of achieving high accuracy on specific benchmarks, while potentially reducing the computational and time resources needed. |
| ProReflow: Progressive Reflow with Decomposed Velocity (Read more on arXiv or HuggingFace) | Yu Li, Xuefei Ning, Haohang Xu, Lei Ke, Ringo1110 | ProReflow improves flow matching in diffusion models for faster image and video generation by progressively refining the diffusion process and emphasizing directional alignment in velocity prediction. The main research objective is to address the high computational cost of diffusion models by optimizing the flow matching training process. The key methodology involves progressive reflow (refining diffusion models in stages with decreasing timesteps) and aligned v-prediction (prioritizing velocity direction matching over magnitude). Primary results show that on the MSCOCO2014 validation set, ProReflow-II achieves an FID of 10.70 with only 4 sampling steps. For AI practitioners, ProReflow offers a more efficient training framework for flow-based diffusion models, achieving state-of-the-art performance with reduced sampling steps, directly benefiting applications requiring fast image/video synthesis. |
| Linear-MoE: Linear Sequence Modeling Meets Mixture-of-Experts (Read more on arXiv or HuggingFace) | Yu Cheng, Tong Zhu, Xiaoye08, landisen, weigao266 | Linear-MoE integrates linear sequence modeling (LSM) with Mixture-of-Experts (MoE) for efficient large-scale model training. The paper explores the objective of combining the benefits of LSM and MoE to improve performance and training efficiency in large models. The methodology involves developing a system with modeling and training subsystems, including sequence parallelism tailored for LSM and hybrid models with standard Transformer-MoE layers. Evaluations on A0.3B-2B and A1B-7B models show Linear-MoE achieves efficiency gains while maintaining competitive performance across various benchmarks. Linear-MoE offers AI practitioners a potential next-generation foundational model architecture by enhancing efficiency and scalability in large language models. |
| Learning from Failures in Multi-Attempt Reinforcement Learning (Read more on arXiv or HuggingFace) | Jie Fu, Stephen Chung, wydu | i) The paper introduces a multi-attempt reinforcement learning task to enhance reasoning in large language models (LLMs) by providing feedback on incorrect responses. ii) The research aims to improve LLMs’ reasoning capabilities by training them to refine responses based on feedback in a multi-attempt setting. iii) The methodology involves training an LLM with standard Proximal Policy Optimization (PPO) on a math problem dataset, modifying the task to allow multiple attempts with feedback after each incorrect answer. iv) The primary result shows that an LLM trained on the multi-attempt task improves accuracy on math benchmarks from 45.6% to 52.5% with two attempts, compared to a marginal improvement from 42.3% to 43.2% for the same LLM trained on a standard single-turn task. v) The principal implication for AI practitioners is that training LLMs with multi-attempt tasks can lead to better self-refinement capabilities and improved performance in reasoning tasks, offering a more effective approach compared to single-turn training. |
| An Empirical Study on Eliciting and Improving R1-like Reasoning Models (Read more on arXiv or HuggingFace) | daixuancheng, Boru, ToheartZhang, EliverQ, TimothyCzp | i) This paper presents an empirical study on improving reasoning capabilities in Large Language Models (LLMs) through Reinforcement Learning (RL) and tool manipulation. ii) The main objective is to investigate methods for eliciting and enhancing R1-like reasoning in LLMs, focusing on scaling RL training and using tool manipulation techniques. iii) The study employs RL training with various hyperparameter settings and reward designs, alongside supervised fine-tuning to enable tool manipulation. iv) The primary result is that RL training improves QWEN2.5-32B base models, achieving 39.33% accuracy on AIME 2024 for a fine-tuned model; furthermore, tool manipulation achieved 86.67% accuracy with greedy search on AIME 2024. v) The findings suggest that scaling RL training and incorporating tool manipulation are effective strategies for AI practitioners to enhance reasoning performance in LLMs, offering a path to improve model capabilities in complex tasks. |
| SAGE: A Framework of Precise Retrieval for RAG (Read more on arXiv or HuggingFace) | Jinyang Su, Guoliang Li, jt-zhang | i) The paper introduces SAGE, a RAG framework enhancing retrieval precision through semantic segmentation, gradient-based chunk selection, and LLM self-feedback. ii) The primary objective is to improve the accuracy and cost-efficiency of RAG systems by addressing limitations in corpus segmentation and context retrieval. iii) The methodology involves training a semantic segmentation model, developing a gradient-based chunk selection algorithm, and implementing an LLM-based self-feedback mechanism for context adjustment. iv) Experiments show SAGE outperforms baselines by 61.25% in QA quality on average and achieves a 49.41% enhancement in cost efficiency. v) SAGE offers AI practitioners a more effective and cost-efficient RAG system by improving the precision of retrieved context, which reduces LLM token consumption and increases QA accuracy. |
| LONGCODEU: Benchmarking Long-Context Language Models on Long Code Understanding (Read more on arXiv or HuggingFace) | Ge Li, Kechi Zhang, Lei Li, Xuyuan Guo, Jia Li | LONGCODEU is introduced as a new benchmark to evaluate long code understanding in LLMs. The primary objective is to assess LLMs’ abilities in code unit perception, intra-code unit understanding, inter-code unit relation understanding, and long code documentation understanding. The methodology involves curating a dataset from real-world code repositories with varying code lengths and evaluating LLMs on eight different tasks spanning the four understanding aspects. Experimental results showed that LLMs’ performance significantly degrades when processing code longer than 32K tokens, and the inter-code unit relation understanding is the most challenging aspect; for example, DeepSeek-V2.5 achieves 11.75% average improvements on the benchmarks tasks. This benchmark provides AI practitioners with a means to identify limitations and guide development of LLMs for software engineering tasks requiring long code context. |
| LoRACode: LoRA Adapters for Code Embeddings (Read more on arXiv or HuggingFace) | bindsch, amanchadha, shollercoaster | LoRACode introduces a parameter-efficient fine-tuning method for code embeddings using Low-Rank Adaptation (LoRA). The research investigates whether LoRA adapters can improve code retrieval accuracy while minimizing computational costs. The methodology involves fine-tuning CodeBERT, GraphCodeBERT, and UniXcoder with LoRA on code corpora, creating task-specific and language-specific adapters. Experiments showed an increase of up to 9.1% in Mean Reciprocal Rank (MRR) for Code2Code search and up to 86.69% for Text2Code search tasks. LoRA’s efficient fine-tuning, utilizing only 1.83%-1.85% of base model parameters, allows AI practitioners to rapidly adapt code embedding models for improved semantic code search with reduced computational resources. |
| R1-Zero’s “Aha Moment” in Visual Reasoning on a 2B Non-SFT Model (Read more on arXiv or HuggingFace) | Minhao Cheng, Ruochen Wang, zhoutianyi, AIcell, Dolphin42 | This paper demonstrates emergent visual reasoning capabilities in a 2B parameter language model through reinforcement learning, without supervised fine-tuning. The main research objective was to replicate the “aha moment” and increased response length observed in DeepSeek-R1 in a multimodal setting, specifically for visual reasoning. The key methodology involved applying the GRPO algorithm, a variant of PPO, directly to a non-SFT Qwen2-VL-2B base model, using a rule-based reward function based on response format and correctness on the SAT dataset. The primary result was that the model achieved 59.47% accuracy on CVBench, outperforming the base model by approximately 30% and the SFT model by about 2%. Principal implication for AI practioners is that reinforcement learning can induce sophisticated reasoning in multimodal models without requiring extensive supervised data, offering a more scalable approach to training. |
| AnyAnomaly: Zero-Shot Customizable Video Anomaly Detection with LVLM (Read more on arXiv or HuggingFace) | Inpyo Hong, Sein Kwon, Kijung Lee, jyy1551, SkiddieAhn | AnyAnomaly is a zero-shot customizable video anomaly detection (C-VAD) method that leverages Large Vision-Language Models (LVLMs). The main research objective is to develop a VAD system that can detect user-defined anomalies in diverse environments without requiring retraining or environment-specific data. The key methodology involves a segment-level approach using a Key frames Selection Module, a context-aware Visual Question Answering (VQA) with position and temporal contexts, and a prompt designed specifically for anomaly scoring. The proposed model, AnyAnomaly, achieved a 9.88% performance improvement over the baseline on the Customizable-ShT (C-ShT) dataset and state-of-the-art on the UBnormal dataset. AI practitioners can deploy VAD in new scenarios without additional training or data collection by providing user-defined text descriptions of anomalies. |
Papers for 2025-03-07
| Title | Authors | Summary |
|---|---|---|
| LLM as a Broken Telephone: Iterative Generation Distorts Information (Read more on arXiv or HuggingFace) | Michalis Vazirgiannis, guokan-shang, mgeng, amr-mohamed | Iterative processing of text by large language models (LLMs) degrades information, similar to the “broken telephone” game. The main research question is whether LLMs distort information through iterative generation, particularly in translation tasks. The key methodology involved simulating iterative translation chains, where an English document was repeatedly translated into and out of other languages using LLMs. Primary results show a gradual decline in factuality and relevance over iterations, with an average FActScore gradient of -0.038 ± 0.02 in the most complex translation chain setting. Principal implication for AI practitioners is that iterative generation with LLMs can lead to information distortion, making control of temperature, prompt design, and understanding the role of intermediary languages necessary when building applications relying on the iterative processing of LLM-generated content. |
| EgoLife: Towards Egocentric Life Assistant (Read more on arXiv or HuggingFace) | Zzitang, Alarak, fesvhtr, THUdyh, Jingkang | i) EgoLife introduces a comprehensive egocentric dataset and benchmark for developing AI life assistants. ii) The study aims to create life-oriented question-answering tasks designed to provide meaningful assistance in daily life through multimodal egocentric data understanding. iii) Data was collected from six participants living together for a week, using AI glasses to record multimodal egocentric video, supplemented by synchronized third-person video references and annotated for comprehensive data analysis. iv) The EgoLife Dataset comprises 300 hours of egocentric data and introduces EgoLifeQA, a benchmark for long-context question answering, alongside EgoButler, an integrated system, and their experiments verified the mechanisms, critical factors, and bottlenecks, guiding future improvements with EgoGPT achieving state-of-the-art performance on egocentric video understanding. v) The EgoLife dataset, tasks, and models offer AI practitioners a resource for advancing long-term egocentric life assistance through improved multimodal integration, identity recognition, and ultra-long-context question answering. |
| HybridNorm: Towards Stable and Efficient Transformer Training via Hybrid Normalization (Read more on arXiv or HuggingFace) | Ya Wang, Breeze0417, LLIXQ, Taoer, BryceZhuo | HybridNorm, a novel normalization strategy for Transformers, combines QKV normalization in attention and Post-Norm in the feed-forward network to improve training stability and performance. The research objective is to address the trade-offs between training stability and final model performance inherent in existing normalization techniques like Pre-Norm and Post-Norm in Transformer models. The key methodology involves proposing HybridNorm and evaluating it through extensive experiments on large-scale dense and Mixture-of-Experts (MoE) language models. The primary results show that HybridNorm consistently outperforms Pre-Norm and Post-Norm across various benchmarks; for example, HybridNorm* achieved an average accuracy of 64.15% compared to Pre-Norm’s 62.99% on downstream tasks for 1.2B dense models. Principal implication: AI practitioners can use HybridNorm to achieve more stable training dynamics and superior performance when training large Transformer models, particularly in language modeling applications. |
| PokéChamp: an Expert-level Minimax Language Agent (Read more on arXiv or HuggingFace) | Andy Luu Nguyen, chijin, milkkarten | PokéChamp is a minimax language agent that achieves expert-level performance in Pokémon battles by integrating large language models (LLMs) into the tree search algorithm. The main research objective is to develop an agent capable of strategic action proposal, accurate opponent modeling, and effective evaluation of game trajectories in Pokémon battles, without requiring LLM fine-tuning. The key methodology involves replacing three components of minimax tree search—player action sampling, opponent modeling, and value function estimation—with LLM-based generations, leveraging a world model that approximates game transitions. PokéChamp, powered by GPT-4o, achieves a 76% win rate against the best existing LLM-based bot and 84% against the strongest rule-based bot in the Generation 9 OverUsed Meta. AI practitioners can leverage this framework’s integration of LLMs with game-theoretic planning algorithms to develop agents for complex, partially observable environments without task-specific training. |
| FuseChat-3.0: Preference Optimization Meets Heterogeneous Model Fusion (Read more on arXiv or HuggingFace) | passerqxj, OnewayLab, GGLS, Wanfq, AALF | FuseChat-3.0 integrates the strengths of heterogeneous large language models (LLMs) into more compact target LLMs using a two-stage training process. The main objective is to develop a method for effectively fusing knowledge from multiple, diverse source LLMs into smaller target LLMs. The methodology involves a specialized data construction protocol followed by supervised fine-tuning (SFT) and Direct Preference Optimization (DPO), using preference pairs generated from the same source model. When using Llama-3.1-8B-Instruct as the target model, the fusion approach achieves an average improvement of 6.8 points across 14 benchmarks. AI practitioners can use this implicit model fusion technique to enhance the performance of smaller LLMs by leveraging the capabilities of larger, heterogeneous models, without requiring architectural changes. |
| Token-Efficient Long Video Understanding for Multimodal LLMs (Read more on arXiv or HuggingFace) | zhiqilinv, MuyangLI, zhijianliu, xiuyul, jdps | i) STORM is a novel architecture for efficient long video understanding in multimodal LLMs. ii) The research aims to improve video understanding in LLMs, particularly with extended temporal contexts. iii) A dedicated temporal encoder using the Mamba State Space Model is introduced between the image encoder and the LLM, enabling token reduction via sampling and spatial/temporal pooling. iv) STORM achieves state-of-the-art results with over 5% improvement on MLVU and LongVideoBench, while reducing computation costs by up to 8x and decoding latency by 2.4-2.9x for fixed input frames. v) Practitioners can leverage STORM to reduce LLM computational demands and latency without sacrificing performance. |
| The Best of Both Worlds: Integrating Language Models and Diffusion Models for Video Generation (Read more on arXiv or HuggingFace) | Xu Tan, Kai Shen, Aoxiong Yin, JunchengLi, ustcscallion | LanDiff is a hybrid text-to-video generation framework that combines language models and diffusion models for coarse-to-fine video synthesis. The main research objective is to develop a framework that leverages the strengths of both autoregressive language models (semantic understanding, causal modeling) and diffusion models (high visual quality, progressive refinement) while mitigating their limitations. The key methodology involves a two-stage process: (1) a semantic tokenizer compresses 3D visual features into 1D discrete representations, and an LLM generates semantic tokens; (2) a streaming diffusion model refines these tokens into high-fidelity video features, decoded by a VAE. LanDiff, with a 5B parameter model, achieves a score of 85.43 on the VBench T2V benchmark, surpassing state-of-the-art open-source and commercial models. AI practitioners can use LanDiff architecture as a blueprint of production-level video generation, particularly in scenarios requiring high semantic accuracy, visual quality, and long video generation capabilities. |
| IFIR: A Comprehensive Benchmark for Evaluating Instruction-Following in Expert-Domain Information Retrieval (Read more on arXiv or HuggingFace) | Mingsheng Shang, yilunzhao, guo9, songtingyu | IFIR is a new benchmark for evaluating instruction-following information retrieval in specialized domains, revealing challenges for current models. The main research objective is to evaluate how well current information retrieval (IR) systems can follow complex, domain-specific instructions in expert fields. Key methodology involves creating a new benchmark (IFIR) with 2,426 examples across finance, law, healthcare, and scientific literature, incorporating three levels of instruction complexity and a novel LLM-based evaluation metric (INSTFOL). Primary results show that while BM25 performs relatively well due to glossary terms, instruction-tuned retrievers like INSTRUCTOR don’t significantly outperform their base models, and most models’ performance declines with increasing instruction complexity; LLM-based retrievers achieve the highest INSTFOL score, as demonstrated by Promptriever-7B. Principal implication is that current retrieval models, even those fine-tuned for instruction following, struggle with long, complex instructions in specialized domains, indicating a need for improved training methodologies and architectures or hybrid systems, leveraging large language model’s superior instruction-following ability. |
| Audio Flamingo 2: An Audio-Language Model with Long-Audio Understanding and Expert Reasoning Abilities (Read more on arXiv or HuggingFace) | manocha, rafaelvalle, firecomputer, ZhifengKong, SreyanG-NVIDIA | i) Audio Flamingo 2 (AF2) is a novel audio-language model (ALM) enhancing audio understanding and reasoning. ii) The research aims to develop an ALM with advanced capabilities in understanding and reasoning over both short and long audio segments, including non-speech sounds and music. iii) AF2 leverages a custom CLAP model, synthetic Audio QA data, and a multi-stage curriculum learning strategy. iv) AF2 achieves state-of-the-art performance on over 20 benchmarks, surpassing larger models, with a 3B parameter language model achieving up to 18.9% improvement on the LongAudioBench compared to Gemini F v2. v) AF2’s ability to understand long audio segments offers AI practitioners new capabilities for real-world applications requiring contextual auditory cue processing, such as anomaly detection and assistive technologies. |
| Identifying Sensitive Weights via Post-quantization Integral (Read more on arXiv or HuggingFace) | Weiyu Huang, surfingtomchen, jt-zhang, zcliang22, yuezhouhu | The paper introduces a novel sensitivity metric and quantization framework for compressing large language models (LLMs). The primary research objective is to develop a more accurate sensitivity metric for weight quantization that addresses limitations of existing gradient and Hessian-based methods. The key methodology is Post-quantization Integral (PQI), which estimates the impact of quantized weights on the loss function, along with a Dense-and-Sparse detach framework called ReQuant. Applying ReQuant to Llama 3.2 1B with QTIP quantization reduces perplexity by 2.66, showcasing the improvement. For AI practitioners, this method provides an effective way to improve post-training quantization of LLMs, achieving better compression with minimal accuracy loss. |
| L$^2$M: Mutual Information Scaling Law for Long-Context Language Modeling (Read more on arXiv or HuggingFace) | Marin Soljačić, Di Luo, Zhuotao Jin, oriolmayne, zhuoc3 | This paper establishes a theoretical framework for understanding and improving long-context language modeling based on a bipartite mutual information scaling law. The main research question is how a language model’s capacity to handle long-range dependencies scales with its internal state size and sequence length. The key methodology involves proving a “Long-context Language Modeling (L²M)” condition, theoretically relating model state size to bipartite mutual information, and empirically validating this scaling law using transformer and state space models on text datasets. The primary result is that bipartite mutual information in natural language scales as I ~ L^β (where β is between 0 and 1) and that a model’s state size must grow at least as fast as I ~ L^β for effective long-context modeling. The principal implication for AI practitioners is that designing models for long-context tasks requires careful consideration of the history state’s scaling, with transformers naturally satisfying this condition and other architectures (like SSMs) needing model size increases to maintain performance at longer sequence lengths. |
| Dedicated Feedback and Edit Models Empower Inference-Time Scaling for Open-Ended General-Domain Tasks (Read more on arXiv or HuggingFace) | Ellie Evans, Daniel Egert, Jiaqi Zeng, Zhilin Wang, odelalleau | Dedicated Feedback and Edit Models enable inference-time scaling for open-ended tasks, achieving state-of-the-art performance by leveraging human feedback. i) Main research question or objective: How to perform inference-time scaling for open-ended general-domain tasks, inspired by human feedback, using dedicated Feedback and Edit Models. ii) Key methodology used: Trained dedicated Feedback and Edit Models on a curated dataset, leveraging human-provided feedback and edits. iii) Primary results: The optimally scaled system, based on 70B models from the Llama 3 family, achieved a state-of-the-art performance on Arena Hard at 92.7, surpassing OpenAI ol-preview-2024-09-12 (90.4) and DeepSeek R1 (92.3). iv) Principal implication for AI practitioners: This approach demonstrates a viable method for improving model performance on complex, open-ended tasks by using human feedback to train models to improve responses at inference. |
| Union of Experts: Adapting Hierarchical Routing to Equivalently Decomposed Transformer (Read more on arXiv or HuggingFace) | Linhui Li, Jing Lian, yjyangwork | Union-of-Experts (UoE) decomposes transformers into equivalent experts and implements selective routing on input data and experts to improve model performance while maintaining efficiency. The main research objective is to address limitations of existing Mixture-of-Experts (MoE) methods, specifically lack of high-quality expert interactions and inefficient extension to attention blocks. Key methodology involves equivalent expert decomposition on MLP and attention blocks via matrix partition, two routing paradigms (patch-wise data and expert selection), and parallel implementation of routing/computation. Primary results show UoE achieves an average perplexity reduction of 2.38 on language modeling tasks compared to the best-performed MoE method, using only 76% of the FLOPs. Principal implication for AI practitioners is that UoE offers a more efficient and performant approach to building transformer-based models, directly applicable to large-scale language and vision tasks. |
| Lost in Literalism: How Supervised Training Shapes Translationese in LLMs (Read more on arXiv or HuggingFace) | Leyang Cui, Huajian Zhang, Zhilin Wang, Ronghao Zhang, yaful | This paper investigates and mitigates translationese (unnatural translations) in Large Language Models (LLMs) caused by biases introduced during supervised fine-tuning (SFT). The main research objective is to evaluate the prevalence of translationese in LLM-generated translations and investigate its origins during supervised training. The key methodology involves human annotation to identify translationese spans, analysis of training data, and mitigation strategies such as refining training references and filtering unnatural instances using perplexity. The primary results show that even advanced models like GPT-4 exhibit substantial translationese, with over 40% of their translations containing substantial translationese patterns, and that refining training data with LLMs reduces perplexity by 7.8 in the English-Chinese dataset. Principal implication for AI practitioners is that addressing translationese bias in SFT data, by polishing golden references or filtering, can improve the naturalness of LLM translation outputs. |
| Combining Flow Matching and Transformers for Efficient Solution of Bayesian Inverse Problems (Read more on arXiv or HuggingFace) | Ekaterina Muravleva, oseledets, dsherki | The paper introduces a method combining Conditional Flow Matching (CFM) and transformers to efficiently solve Bayesian inverse problems. The main objective is to recover the distribution of model parameters conditioned on observed experimental data, given a series of observations and a forward model. The key methodology involves training a transformer-based CFM architecture to learn the conditional probability distribution from samples, handling a variable number of observations. Results showed that for a SEIR disease model, the average error was 2.05% ± 1.04% using a 4-point MLP model, significantly outperforming MCMC in computational efficiency. AI practitioners can leverage this approach for faster and more scalable sampling from posterior distributions in Bayesian inverse problems, particularly with datasets having variable-length observations. |
| Understanding and Predicting Derailment in Toxic Conversations on GitHub (Read more on arXiv or HuggingFace) | Rebekah Copeland, Robert Zita, kdamevski, rahat-rizvi, imranraad | This research investigates conversational derailment leading to toxicity in GitHub discussions, aiming to predict and mitigate such occurrences proactively. The main research objective is to understand the characteristics of toxic conversations on GitHub and how these conversations derail into toxicity. The key methodology involves curating a dataset of toxic and non-toxic GitHub conversations, analyzing linguistic and conversational features, and developing a Large Language Model (LLM)-based approach using conversation trajectory summaries. The LLM prompts, tailored to provide summaries of GitHub conversations, achieved a 69% F1-score in predicting conversational derailment. AI practitioners can use this proactive, domain-specific, LLM-based moderation approach to identify and address potentially harmful conversations on platforms like GitHub before they escalate to toxicity. |
Papers for 2025-03-06
| Title | Authors | Summary |
|---|---|---|
| Babel: Open Multilingual Large Language Models Serving Over 90% of Global Speakers (Read more on arXiv or HuggingFace) | LidongBing, maljunied, jhying, lukecq, Yiran0924 | Babel is an open multilingual large language model that supports 25 languages, covering over 90% of global speakers. The main objective is to develop an open-source multilingual LLM that addresses the underrepresentation of many widely spoken languages in existing models. The key methodology is layer extension, adding new layers to an existing model (Qwen2.5) and pre-training on a curated dataset emphasizing under-resourced languages. Babel-83B-Base achieves an average score of 73.2 across six multilingual benchmarks, outperforming comparable open models like Qwen2.5-72B (69.8). AI practitioners can use Babel as a strong base or chat model for multilingual applications, benefiting from enhanced performance, especially in low-resource languages, and from the use of layer extension in scaling the model. |
| ABC: Achieving Better Control of Multimodal Embeddings using VLMs (Read more on arXiv or HuggingFace) | Florian Kerschbaum, Benjamin Schneider, wenhu | ABC is a multimodal embedding model that uses a vision-language model (VLM) backbone to integrate natural language instructions with visual inputs for improved control over embeddings. The main research objective is to develop a model that can effectively utilize user instructions to control and refine multimodal embeddings, overcoming limitations of existing CLIP-based models. The key methodology involves a two-stage training process: contrastive pretraining with mined negatives and instruction fine-tuning using synthetic instructions generated from image captions. The model achieves best-for-size performance on MSCOCO image-to-text retrieval with a R@1 score of 69.2 and outperforms all other models on the Massive Multimodal Embedding Benchmark (MMEB) for classification and VQA tasks. AI practitioners can use ABC’s architecture and training approach to create multimodal embedding models with enhanced control via natural language, resulting in a flexible tool that improves performance of visual retrieval, classification, and VQA, as well as the ability to complete unique, instruction-specific tasks. |
| Enhancing Abnormality Grounding for Vision Language Models with Knowledge Descriptions (Read more on arXiv or HuggingFace) | Cosmin I. Bercea, Rossella Arcucci, Wenjia Bai, Jun Li, che111 | This paper introduces a method to improve medical abnormality grounding in vision-language models (VLMs) using decomposed knowledge descriptions. The main research objective is to enhance the performance of VLMs in detecting and localizing medical abnormalities in images by improving the alignment between textual descriptions and visual features. The key methodology involves decomposing medical concepts into fundamental attributes and visual patterns, and using these attribute-based descriptions as prompts during VLM training. The proposed method, trained on only 1.5% of the data used by larger models, achieved a RoDeO score of 54.38% on the VinDr-CXR dataset, comparable to 7B parameter models like RadVLM. AI practitioners can use this knowledge-enhanced approach to achieve competitive performance in medical image abnormality grounding with significantly smaller VLMs and less training data, and improve zero-shot generalization. |
| GEN3C: 3D-Informed World-Consistent Video Generation with Precise Camera Control (Read more on arXiv or HuggingFace) | Yifan Lu, Huan Ling, Jiahui Huang, Tianchang Shen, xrenaa | GEN3C is a generative video model with precise camera control and temporal 3D consistency. The main research objective is to develop a video generation model that allows for precise camera control and maintains 3D consistency across generated frames. The key methodology involves constructing a 3D cache (point clouds from depth estimates) and rendering it with user-provided camera trajectories to condition a fine-tuned video diffusion model. The results demonstrate that GEN3C achieves a PSNR of 18.66 and an SSIM of 0.67 on the Tanks-and-Temples dataset for single-view video generation, outperforming baselines. For AI practitioners, GEN3C offers a method for generating 3D-consistent videos with precise camera control by conditioning video generation on 3D renderings, improving controllability and consistency compared to prior video generation models. |
| KodCode: A Diverse, Challenging, and Verifiable Synthetic Dataset for Coding (Read more on arXiv or HuggingFace) | Radha Poovendran, mingyuanzhou, yyqoni, nlpyang, flydust | KODCODE is a synthetic dataset of 447K coding problems with verified solutions and unit tests, designed to enhance code LLM training. The main research objective is to create a large-scale, diverse, and verifiable coding dataset that addresses limitations in existing resources for training large language models (LLMs) for code. The methodology involves a three-step pipeline: coding question synthesis from 12 sources, solution and test generation with self-verification, and post-training data synthesis via question rewriting and test-based rejection sampling using DeepSeek-R1. Models fine-tuned on KODCODE-SFT achieved a 61.26% average score across five coding benchmarks, outperforming models like Qwen2.5-Coder-32B-Instruct and DeepSeek-R1-Distill-Llama-70B. The principal implication is that AI practitioners can use KODCODE to improve the performance of code LLMs in supervised fine-tuning and potentially RL training, with verified solutions and tests offering advantages for various code-related tasks. |
| CrowdSelect: Synthetic Instruction Data Selection with Multi-LLM Wisdom (Read more on arXiv or HuggingFace) | Pan Zhou, Wenxuan Shen, Lingfeng Yang, shuaishuaicdp, yisenL | CROWDSELECT, a novel synthetic instruction data selection framework, leverages multi-LLM responses and reward scores for improved instruction tuning. The main research objective is to investigate whether multi-dimensional signals derived from multiple LLMs can enhance the selection of synthetic instruction-response pairs for instruction tuning. The key methodology involves calculating three metrics (Difficulty, Separability, Stability) from multiple LLM responses and reward model assessments, and then integrating these with a clustering-based approach for diverse data selection. Primary results show that CROWDSELECT achieves state-of-the-art performance, improving instruction tuning by 4.81% on Arena-Hard and 11.1% on MT-bench with Llama-3.2-3b-instruct. The principal implication for AI practitioners is that leveraging multi-LLM wisdom through the proposed metrics and framework can lead to more efficient and effective instruction tuning, improving the performance of distilled smaller models. |
| QE4PE: Word-level Quality Estimation for Human Post-Editing (Read more on arXiv or HuggingFace) | Malvina Nissim, Ana Guerberof-Arenas, Grzegorz Chrupała, Vilém Zouhar, gsarti | The QE4PE study investigates the impact of word-level quality estimation (QE) on professional machine translation post-editing, finding that factors beyond QE accuracy influence its real-world usefulness. The main research objective was to measure the effect of word-level QE error span highlighting on the editing quality, productivity, and usability in a realistic post-editing workflow. The methodology involved 42 professional translators post-editing machine-translated texts in English-Italian and English-Dutch, using four highlight modalities (supervised, unsupervised, oracle, and no highlights) and logging their editing behavior. Results showed that highlight modalities are not solely predictive of editing time and that cross-modality highlight overlap ranged between 15% and 39%. This implies that AI practitioners should consider factors beyond accuracy, such as domain, language, and user-specific factors, to improve the integration of word-level QE in post-editing tools and enhance their real-world usability. |
| Exploring Rewriting Approaches for Different Conversational Tasks (Read more on arXiv or HuggingFace) | Xiang Chen, Mike Rimer, Ryan A. Rossi, Md Mehrab Tanjim, Franck-Dernoncourt | This paper systematically investigates query rewriting and fusion approaches for conversational AI tasks. The main research question is whether a single LLM-based query rewrite module can be universally effective across diverse conversational scenarios or if specialized modules are needed. The key methodology involves evaluating two parameterized query rewriting approaches (query rewrite and query fusion) on three datasets: conversational text-based Q&A and two text-to-visualization tasks (short and long conversations). The primary result is that for the conversational text-based Q&A task, the query rewrite approach achieved a 3.9% higher mean cosine similarity than query fusion, while for long text-to-vis tasks, query fusion had 7.6% high mean cosine similarity. The principal implication is that AI practitioners should select a query rewriting approach (either query rewrite and query fusion) that aligns with the specific conversational task and data characteristics, as no single approach is universally superior. |
| Process-based Self-Rewarding Language Models (Read more on arXiv or HuggingFace) | Zheheng Luo, Junxiao Liu, Xin Zhang, Shimao Zhang, lx865712528 | The paper introduces Process-based Self-Rewarding Language Models, enhancing mathematical reasoning by incorporating step-wise evaluations and preference optimization. The main research objective is to improve the mathematical reasoning capabilities of large language models (LLMs) using a self-rewarding paradigm without external human feedback. The key methodology involves iterative training with step-wise LLM-as-a-Judge evaluations and step-wise preference optimization using Direct Preference Optimization (DPO). The primary result is that the 72B model, after four iterations, achieved an average accuracy of 60.6 across several math benchmarks, an improvement over the starting accuracy. The principal implication is that AI practitioners can improve LLMs’ mathematical reasoning performance, through iterative self-improvement without human-annotated data. |
| Fine-Tuning Small Language Models for Domain-Specific AI: An Edge AI Perspective (Read more on arXiv or HuggingFace) | KartikAngadi, kruthika, SyedAbdul, RakshitAralimatti | The paper introduces the Shakti series of Small Language Models (SLMs) designed for efficient on-device AI, focusing on domain-specific applications. The main objective is to develop SLMs that can overcome resource constraints of edge devices while maintaining high performance in specialized domains. Key methodologies include a combination of efficient transformer architectures, quantization-aware training, supervised fine-tuning, and preference alignment (RLHF or DPO). Primary results show that Shakti-500-Q4 achieves 583.88 tokens per second (TPS) on an NVIDIA L40s GPU and the Shakti-250M model, after fine-tuning, achieves 0.86 answer relevance score in finance domain. The paper’s principal implication is that carefully engineered and fine-tuned compact models can effectively be deployed on edge devices, offering a practical approach for real-world, domain-specific AI applications with limited computational resources. |
| Mixture of Structural-and-Textual Retrieval over Text-rich Graph Knowledge Bases (Read more on arXiv or HuggingFace) | Ryan A. Rossi, Haoyu Han, Yongjia Lei, mhalappa, Franck-Dernoncourt | This paper proposes a Mixture of Structural-and-Textual Retrieval (MoR) framework for answering queries over text-rich graph knowledge bases (TG-KBs). The main research objective is to develop a retrieval method that effectively combines both textual and structural information from TG-KBs to improve query answering performance. The key methodology is a Planning-Reasoning-Organizing framework, where the Planning stage generates textual planning graphs, the Reasoning stage interweaves structural traversal and textual matching, and the Organizing stage reranks candidates based on their structural trajectory. The primary result shows that MoR achieved an average Hit@1 score of 48.93%, outperforming other baselines on three TG-KB datasets. The principal implication is that AI practitioners can leverage MoR’s mixture-of-experts approach to improve retrieval performance in applications that use the graph knowledge bases by harmonizing textual and structural signals, especially useful to combine and rank structural knowledge from graph data with traditional text features. |
| Retrieval Models Aren’t Tool-Savvy: Benchmarking Tool Retrieval for Large Language Models (Read more on arXiv or HuggingFace) | Shuaiqiang Wang, Pengjie Ren, Lingyong Yan, Yuhan Wang, Zhengliang Shi | The paper introduces TOOLRET, a new benchmark for evaluating information retrieval (IR) models on tool retrieval tasks for large language models (LLMs). The main research objective is to assess the performance of existing IR models in retrieving relevant tools for LLMs in diverse, real-world scenarios, and to analyze the impact of retrieval quality on end-to-end task performance. The key methodology involves collecting and curating a large-scale dataset of 7.6k retrieval tasks and 43k tools from existing datasets, evaluating various IR models (sparse, dense, and re-ranking) on this benchmark, and contributing a large scale training dataset (TOOLRET-train) to improve retrieval performance. A primary result is that the best-performing model (NV-embedd-v1) achieves an nDCG@10 of only 33.83 on the benchmark, indicating existing IR models struggle with tool retrieval. The principal implication is that AI practitioners need to develop new retrieval methods tailored for tool retrieval, or improve upon current methods using target-aware reasoning and large-scale training data, as shown in the paper using TOOLRET-train, since current strong IR models are not effective for tool retrieval. |
| FLAME: A Federated Learning Benchmark for Robotic Manipulation (Read more on arXiv or HuggingFace) | Danica Kragic, Yuchong Zhang, Miguel Vasco, Alberta Longhini, Santiago Bou Betran | FLAME is a new benchmark for federated learning in robotic manipulation, providing datasets and a framework for distributed training. The main objective is to evaluate federated learning (FL) strategies for training robotic manipulation policies in a distributed, privacy-preserving manner. The key methodology involves creating a large-scale dataset of diverse manipulation tasks across multiple simulated environments and integrating it into a FL framework using FLOWER, where local models are trained and aggregated. Primary results show that Federated Averaging (FedAvg) achieves a 2.64 ± 0.13 RMSE on the Slide Block to Target task, but performance varies significantly across tasks and FL methods. The principal implication for AI practitioners is that FLAME provides a standardized benchmark for evaluating and developing scalable, adaptive, and privacy-aware robotic learning systems, although further development in FL algorithms are necessary. |
| Benchmarking Large Language Models for Multi-Language Software Vulnerability Detection (Read more on arXiv or HuggingFace) | Hung Nguyen, Martin Weyssow, Yindu Su, Chengran Yang, Ting Zhang | This paper presents a comprehensive empirical study evaluating large language models (LLMs) on software vulnerability detection (SVD) across multiple programming languages. The main research objective is to investigate the effectiveness of various LLMs in predicting software vulnerabilities, comparing them with smaller language models (SLMs) and static application security testing (SAST) tools, and exploring strategies to improve LLM performance. The key methodology involves compiling a multi-language dataset (Python, Java, JavaScript) of vulnerable functions, evaluating five open-source LLMs using prompt engineering, instruction tuning, and sequence classification fine-tuning, and comparing them against SLMs and SAST tools. The results show that fine-tuned LLMs achieved the best F1-score of 0.443 on the JavaScript dataset, with performance varying significantly across programming languages and adaptation strategies. The principal implication for AI practitioners is that while LLMs show promise for SVD, particularly in JavaScript with fine-tuning, performance is highly dependent on data characteristics, requiring careful consideration of language, model selection, and adaptation strategies. |
| CognitiveDrone: A VLA Model and Evaluation Benchmark for Real-Time Cognitive Task Solving and Reasoning in UAVs (Read more on arXiv or HuggingFace) | Artyom Myshlyaev, Oleg Sautenkov, Muhammad Haris Khan, Valerii Serpiva, Artem Lykov | CognitiveDrone, a Vision-Language-Action (VLA) model and benchmark for real-time cognitive task solving in UAVs, is introduced. The main research objective is to develop and evaluate a UAV control system capable of performing complex cognitive tasks, including human recognition, symbol understanding, and reasoning, based on visual input and textual instructions. The methodology combines a 7B-parameter VLA model (adapted from OpenVLA) trained on a dataset of over 8,000 simulated flight trajectories with an optional 7B-parameter VLM reasoning module (Qwen2.5-VL based) for task refinement, and evaluates performance within a Gazebo-based simulation benchmark (CognitiveDroneBench). The CognitiveDrone-R1 model, incorporating the reasoning module, achieved a 77.2% overall success rate, outperforming the base CognitiveDrone model (59.6%) and a racing-oriented model (RaceVLA, 31.3%). AI practitioners can utilize the provided open-source dataset, benchmark environment, and model weights to develop and evaluate VLA models for UAVs that incorporate cognitive capabilities beyond basic navigation and control. |
| Interact, Instruct to Improve: A LLM-Driven Parallel Actor-Reasoner Framework for Enhancing Autonomous Vehicle Interactions (Read more on arXiv or HuggingFace) | Peng Hang, Chen Lv, Chengkai Xu, Jiaqi Liu, FanGShiYuu | This paper introduces an LLM-driven Actor-Reasoner framework for autonomous vehicles (AVs) to improve bidirectional interactions with human-driven vehicles (HVs). The main objective is to enhance AVs’ real-time decision-making and intent expression capabilities in complex driving scenarios with heterogeneous HVs. The methodology involves a parallel Actor-Reasoner architecture; the Reasoner uses an LLM with Chain-of-Thought (CoT) reasoning to infer HV driving styles and generate eHMI displays, while the Actor employs a two-layer memory retrieval mechanism from a database constructed during training with simulated HVs. Results show that the proposed framework achieves a 94% success rate in intersection scenarios, and a memory partition module improves retrieval speed by an average of 12%. AI practitioners can use this framework as a method to integrate LLMs into real-time decision-making systems, addressing LLM inference speed limitations by combining reasoning capabilities with memory-based fast retrieval. |
| SwiLTra-Bench: The Swiss Legal Translation Benchmark (Read more on arXiv or HuggingFace) | Yingqiang Gao, Sina Ahmadi, Luka Nenadic, Jakob Merane, Joel Niklaus | SwiLTra-Bench introduces a multilingual benchmark for evaluating LLM-based translation systems on Swiss legal texts, comprising 180K aligned translation pairs across five languages. The main research objective was to evaluate the performance of frontier LLMs and fine-tuned open SLMs on Swiss legal translations in zero-shot and fine-tuning settings, including the development of an LLM-based evaluation metric. Key methodology included systematic evaluation using lexical and model-based metrics, fine-tuning open SLMs, human expert validation, and developing a specialized LLM evaluation system (SwiLTra-Judge). Primary results showed that frontier models like Claude-3.5-Sonnet outperformed others, achieving a GEMBA-MQM score of 80.66, while fine-tuned open SLMs improved but still lagged behind. For AI practitioners, this benchmark and the associated evaluations highlight that while frontier models provide superior legal text translation, fine-tuning offers significant improvement for open SLMs, and SwiLTra-Judge can serve as a reliable automated evaluation tool that aligns well with human experts. |
Papers for 2025-03-05
| Title | Authors | Summary |
|---|---|---|
| MPO: Boosting LLM Agents with Meta Plan Optimization (Read more on arXiv or HuggingFace) | sujianli, songff, Adagio, Rsy24, xwm | The paper introduces Meta Plan Optimization (MPO), a framework that enhances large language model (LLM) agents’ planning capabilities by incorporating optimized, high-level meta plans. The main research objective is to improve LLM-based agents’ performance on interactive planning tasks without requiring retraining for each new agent, while addressing planning hallucinations. MPO leverages a meta planner that generates abstract task strategies, optimized via a combination of supervised fine-tuning, Monte Carlo sampling, and Direct Preference Optimization (DPO) using agent feedback. Experiments on ALFWorld and ScienceWorld benchmarks demonstrate that MPO significantly outperforms existing baselines, with performance improvements of up to 100% for some agents. For AI practitioners, MPO offers a plug-and-play solution to boost agent performance and generalization in planning tasks, by incorporating general guidance that is improvable. |
| Mask-DPO: Generalizable Fine-grained Factuality Alignment of LLMs (Read more on arXiv or HuggingFace) | Kai Chen, Chengqi Lyu, lindahua, ZwwWayne, vanilla1116 | Mask-DPO is a fine-grained factuality alignment method for LLMs that leverages sentence-level factuality to improve preference learning and reduce hallucinations. The main research objective is to develop a more effective and generalizable method for aligning LLMs with factual correctness, addressing limitations of response-level preference learning. The key methodology, Mask-DPO, incorporates sentence-level factuality annotations as mask signals in Direct Preference Optimization (DPO), selectively learning from correct sentences in preferred responses and avoiding penalties on factual content in non-preferred responses. Primary results show that Mask-DPO improved the factuality score of Llama3.1-8B-Instruct on the ANAH test set from 49.19% to 77.53%. Principal implication for AI practitioners is that Mask-DPO provides a more precise alignment technique that enhances factuality and generalization in LLMs, enabling the development of more reliable and trustworthy AI assistants. |
| Wikipedia in the Era of LLMs: Evolution and Risks (Read more on arXiv or HuggingFace) | Yao Wan, fjchendp, mgeng, sdzzxyl, hsm316 | This paper analyzes the impact of Large Language Models (LLMs) on Wikipedia, examining its evolution and potential risks to the broader NLP community. The primary research objective is to determine if and how LLMs have already impacted Wikipedia, and how this might influence the NLP community. The key methodology involves analyzing Wikipedia page views, article content, and simulating LLM impact on machine translation benchmarks and Retrieval-Augmented Generation (RAG) systems. Primary results indicate that Wikipedia articles have been influenced by LLMs, with an estimated impact of 1%-2% in certain categories and simulations show potential score inflations in machine translation benchmarks and performance reduction in RAG systems using LLM generated content. The principal implication for AI practitioners is that reliance on Wikipedia for training and evaluating NLP models may be affected by LLM-generated content, necessitating careful consideration of data provenance and potential biases. |
| LADDER: Self-Improving LLMs Through Recursive Problem Decomposition (Read more on arXiv or HuggingFace) | akiray1, TamasSimonds | LADDER is a framework enabling large language models (LLMs) to autonomously improve problem-solving through self-guided learning by recursively generating and solving simpler problem variants. The main research objective is to develop a method for LLMs to improve their mathematical integration capabilities without curated datasets or human feedback. The key methodology, LADDER, involves recursive generation of simpler problem variants, solution verification via numerical integration, and reinforcement learning (using GRPO) on the variant trees. LADDER improved a Llama 3.2 3B model’s accuracy on undergraduate-level integration problems from 1% to 82%, and, with test-time reinforcement learning (TTRL) a Qwen 2.5 7B model achieved 90% on MIT Integration Bee. AI practitioners can leverage self-improving systems like LADDER and TTRL to enhance model capabilities in verifiable domains without extensive human supervision or data curation, demonstrating a practical path to developing more autonomous and capable AI. |
| MultiAgentBench: Evaluating the Collaboration and Competition of LLM agents (Read more on arXiv or HuggingFace) | mikewang, ShuyiGuo, Thomas-X-Yang, zhaochenhong, Leozkl | MultiAgentBench is a benchmark designed to evaluate LLM-based multi-agent systems across diverse interactive scenarios, measuring task completion and the quality of collaboration and competition. The main research objective is to assess how well LLM-based multi-agent systems perform in collaborative and competitive environments, using novel milestone-based key performance indicators. The methodology involves evaluating various coordination protocols (star, chain, tree, graph) and strategies (group discussion, cognitive planning) in six interactive scenarios, including research, Minecraft, database, coding, bargaining, and Werewolf, developed using the MARBLE framework. Results show gpt-4o-mini achieves the highest average task score, graph structure performs best in research, and cognitive planning improves milestone achievement rates by 3%. For AI practitioners, the framework and benchmark provide a means to systematically evaluate and improve multi-agent coordination, which is critical in developing more effective and collaborative AI systems. |
| PipeOffload: Improving Scalability of Pipeline Parallelism with Memory Optimization (Read more on arXiv or HuggingFace) | Min Lin, Xinyi Wan, JialinLi, huanggx-sea, QPHutu | PipeOffload enhances pipeline parallelism (PP) scalability for large language models (LLMs) by optimizing activation memory usage through offloading. The main research objective is to address the activation memory bottleneck in PP that limits its scalability. The key methodology involves selectively offloading activations to host memory, prioritizing those with longer lifespans, and integrating a generalized interleaving strategy for balancing memory and throughput. The primary result is that PipeOffload reduces per-device activation memory in a better-than-linear manner, enabling up to a 19% acceleration compared to tensor parallelism (TP), while using less memory in applicable cases. For AI practitioners, PipeOffload provides a more scalable PP method, especially beneficial when full activation offload is feasible (k <= 1), allowing for more efficient training of large models. |
| Iterative Value Function Optimization for Guided Decoding (Read more on arXiv or HuggingFace) | Ruizhe Chen, jokephp, ab3223323, lljhbxt, zhliu | Iterative Value Function Optimization (IVO) is a novel framework for guided decoding that improves the accuracy of value estimation in language models without retraining the base model. The main research objective is to address the limitations of existing value-guided decoding methods, which suffer from inaccurate value estimation due to high variance and distribution shift. The key methodology involves two components: Monte Carlo Value Estimation, which reduces estimation variance by exploring diverse trajectories, and Iterative On-Policy Optimization, which progressively improves value estimation through collecting trajectories from value-guided policies. Primary results show that IVO achieves 77.52% GPT-4 win rates on the Multi-turn Dialogue task against the base policy, significantly outperforming baseline methods in terms of reward scores across various tasks. Principal implication for AI practitioners is that IVO offers a computationally efficient way to align language models with human values and task requirements, improving control over model outputs without expensive retraining. |
| FR-Spec: Accelerating Large-Vocabulary Language Models via Frequency-Ranked Speculative Sampling (Read more on arXiv or HuggingFace) | yuxuanli, zwl96, hyx21, ThonyPan, Achazwl | FR-Spec accelerates large-vocabulary language models by optimizing draft candidate selection in speculative sampling. The main research objective is to address the increased computational overhead of the LM Head in speculative sampling when using models with large vocabularies. The key methodology is frequency-ranked speculative sampling, which constrains the draft search to a frequency-prioritized token subset, reducing LM Head computation. Primary results show an average 1.12x speedup over the state-of-the-art speculative sampling method EAGLE-2 on multiple datasets, with optimized drafting reducing computation by 75%. For AI practitioners, this method provides a plug-and-play solution to accelerate existing speculative sampling techniques without retraining, directly improving inference speed for large-vocabulary language models. |
| SemViQA: A Semantic Question Answering System for Vietnamese Information Fact-Checking (Read more on arXiv or HuggingFace) | Thanh T. Tran, ThanhDi, TienAnh, xuandin, DavidNguyen | SemViQA is a Vietnamese language fact-checking system that enhances accuracy and efficiency through semantic understanding. The main research objective is to develop a robust fact-checking system for Vietnamese, a low-resource language, addressing challenges like semantic ambiguity and long-token sequences. The key methodology integrates Semantic-based Evidence Retrieval (SER), combining TF-IDF and a Question Answering Token Classifier (QATC), with a Two-step Verdict Classification (TVC) using Focal Loss and Cross-Entropy Loss. The system achieves a strict accuracy of 80.82% on the ViWikiFC dataset and 78.97% on the ISE-DSC01. The principal implication is that AI practitioners can leverage SemViQA’s framework, particularly its SER and TVC components, to develop more efficient, robust, and effective fact-checking systems that handle complex linguistic structures, especially in low-resource languages. |
| UFO: A Unified Approach to Fine-grained Visual Perception via Open-ended Language Interface (Read more on arXiv or HuggingFace) | windmillknight, Shawnee-bxy, Haiyang-W, chenweix7, kanashi6 | UFO unifies fine-grained visual perception tasks through an open-ended language interface, achieving state-of-the-art performance without task-specific decoders. The main research objective is to effectively integrate fine-grained perception tasks (like detection and segmentation) into multimodal large language models (MLLMs) without relying on complex, task-specific designs. The key methodology involves transforming all perception targets into the language space and using a novel embedding retrieval approach for segmentation, relying solely on the language interface. After multi-task training, UFO outperforms previous state-of-the-art generalist models by 12.3 mAP on COCO instance segmentation and 3.3 mIoU on ADE20K semantic segmentation. AI practitioners can leverage UFO’s unified framework to simplify architectural design and training, seamlessly integrating fine-grained perception capabilities into MLLMs for enhanced visual understanding and enabling more challenging vision-language tasks. |
| ATLaS: Agent Tuning via Learning Critical Steps (Read more on arXiv or HuggingFace) | Yuxuan Huang, Ming Li, Zhixun Chen, zhoutianyi, YaliDU | ATLAS finetunes large language model (LLM) agents on critical steps within expert trajectories to improve generalization and reduce training costs. The main research objective is to develop a more efficient and effective agent tuning method by identifying and focusing on critical steps in expert trajectories. The key methodology, ATLAS, uses an oracle LLM to select critical steps based on criteria like plan creation, critical observation, critical action, and self-correction, then finetunes the agent’s LLM solely on these steps. Results show that an LLM finetuned on only ~30% critical steps selected by ATLAS outperforms the LLM finetuned on all steps and recent open-source LLM agents. The principal implication is that AI practitioners can achieve better agent generalization and performance with reduced training costs by focusing LLM finetuning on semantically critical steps identified by an oracle LLM. |
| Language Models can Self-Improve at State-Value Estimation for Better Search (Read more on arXiv or HuggingFace) | rittera, emendes3 | Self-taught lookahead (STL) enables language model-based value functions to improve without ground truth rewards by leveraging state-transition dynamics. The main research objective is to demonstrate that an LLM-based value function can self-improve without labels or rewards, outperforming computationally expensive methods. The key methodology, STL, fine-tunes a value model by predicting the next best action, resulting state, and value rationale, bootstrapping from an initial value function using lookahead in tree search. Results show that STL-improved models match the performance of a GPT-4 value model, improving performance by 20% while reducing inference costs 37x compared to prior LLM-based tree search. Principal implication is that AI practitioners can utilize STL to train efficient and effective value models for search-based tasks, reducing reliance on expensive closed-source models and ground truth rewards. |
| RectifiedHR: Enable Efficient High-Resolution Image Generation via Energy Rectification (Read more on arXiv or HuggingFace) | Liang Hou, dizhang, wileewang, PaulSHEN1, YZCS | RectifiedHR is a training-free method for generating high-resolution images with diffusion models by addressing energy decay and employing noise refresh. The main objective is to enable diffusion models to efficiently generate images at resolutions higher than their training resolution without additional training. The key methodology involves a noise refresh strategy to progressively increase resolution during sampling and an energy rectification strategy that adjusts classifier-free guidance to mitigate image blurriness. The primary result is that RectifiedHR achieves a FID score of 25.347 and a CLIP score of 33.756 at 2048x2048 resolution, outperforming several baselines in image quality while using less computing time. The principal implication is that AI practitioners can generate high-quality, high-resolution images using pre-trained diffusion models without costly retraining or complex modifications, by using noise refresh and energy rectification steps during image generation. |
| SPIDER: A Comprehensive Multi-Organ Supervised Pathology Dataset and Baseline Models (Read more on arXiv or HuggingFace) | Ekaterina Ivanova, alpchel, mgvz | SPIDER is a new multi-organ histopathology dataset with baseline models for patch-level classification and whole-slide image segmentation. The main research objective is to create and evaluate a large, high-quality, multi-organ, patch-level histopathology dataset with comprehensive class coverage, along with baseline classification models. Key methodology used is a semi-automatic annotation pipeline, expert pathologist verification, feature extraction with Hibou-L foundation model, and an attention-based classification head. Primary results of SPIDER’s evaluation include, on the thorax test set, model achieved an accuracy of 0.962, precision of 0.958, and F1 score of 0.960. AI practitioners can use this dataset and models to improve digital pathology tasks like tissue classification and rapid identification, providing a new benchmark for future developments in this field. |
| Q-Eval-100K: Evaluating Visual Quality and Alignment Level for Text-to-Vision Content (Read more on arXiv or HuggingFace) | Zicheng Zhang, GTZhai, a9108, sl2782087, wcain | The paper introduces Q-Eval-100K, a large-scale dataset, and Q-Eval-Score, a unified model, for evaluating visual quality and text-image/video alignment in text-to-vision generation. The main research objective is to develop a comprehensive benchmark and method for assessing both the visual quality and text-alignment of content generated by text-to-vision models. The key methodology involves collecting 100K instances (images and videos) with 960K human annotations of Mean Opinion Scores (MOS) and developing Q-Eval-Score, a Large Multimodal Model (LMM) fine-tuned using a context-prompt format. The primary results show that Q-Eval-Score achieves a 0.943 SRCC for image visual quality at the model-level, outperforming existing methods, it also introduces Vague-to-Specific Strategy for long prompt alignment. AI practitioners can use Q-Eval-100K and Q-Eval-Score as a reliable benchmark and evaluation metric to assess and improve the performance of text-to-vision generative models, focusing on both visual quality and text-alignment. |
| IterPref: Focal Preference Learning for Code Generation via Iterative Debugging (Read more on arXiv or HuggingFace) | Ruihang, yangyu90, Jianwen2003, CharonBony, Ringo1110 | IterPref is a new preference alignment framework for code generation that improves Code LLMs through iterative debugging. The research objective is to address the limitation of existing preference learning methods that do not pinpoint specific code errors, hindering the learning of informative error correction patterns. The key methodology is IterPref, which involves creating the CodeFlow dataset where code is iteratively refined until passing tests, and using a tailored DPO algorithm to align corresponding tokens for error regions. Primary result is that, equipped with IterPref, Qwen2.5-Coder-7B achieved a 29.7% pass@1 score on BigCodeBench Complete Hard, on par with some much larger models. For AI practitioners, this implies an effective way to enhance code generation models that leverages an iterative debugging process for precise preference learning, focusing model’s learning on correcting critical errors. |
| AppAgentX: Evolving GUI Agents as Proficient Smartphone Users (Read more on arXiv or HuggingFace) | Chi Zhang, Wenjia Jiang, xuyang, ChenxiSong, yyzhuang2 | AppAgentX introduces an evolutionary framework for GUI agents that improves operational efficiency on smartphones while maintaining adaptability. The main research objective is to address the inefficiency of LLM-based GUI agents in performing routine tasks by enabling them to learn and evolve high-level actions. The key methodology involves a memory mechanism that records task execution history, allowing the agent to identify repetitive action sequences and replace them with abstract, high-level actions represented as “shortcut nodes”. Primary results show that on the AppAgent benchmark, AppAgentX reduced the average steps per task from 9.1 to 5.7 and increased the success rate from baseline 16.9% to 71.4% . For AI practitioners, this evolutionary framework offers a method to develop GUI agents that execute routine operations more efficiently while using LLM only to optimize new behavior, thus improving the balance between intelligence and efficiency in practical applications. |
Papers for 2025-03-04
| Title | Authors | Summary |
|---|---|---|
| Visual-RFT: Visual Reinforcement Fine-Tuning (Read more on arXiv or HuggingFace) | yhcao, sweetFruit, yuhangzang, Zery, ziyuliu | Visual-RFT extends Reinforcement Fine-Tuning (RFT) to visual tasks by using verifiable rewards to improve performance of Large Vision-Language Models (LVLMs). The main objective is to apply RFT, previously successful in language models, to multi-modal domains, specifically visual perception tasks, with limited data. The key methodology is using LVLMs to generate multiple responses with reasoning tokens and applying visual perception verifiable reward functions (e.g., IoU for object detection) to update the model via policy optimization algorithms like Group Relative Policy Optimization (GRPO). Visual-RFT improved accuracy by 24.3% over the baseline in one-shot fine-grained image classification and exceeded SFT baselines by 21.9 and 15.4 on COCO and LVIS, in two-shot settings, respectively. For AI practitioners, Visual-RFT offers a data-efficient, reward-driven approach to enhance reasoning and adaptability in LVLMs for domain-specific tasks, particularly when fine-tuning data is scarce. |
| Difix3D+: Improving 3D Reconstructions with Single-Step Diffusion Models (Read more on arXiv or HuggingFace) | zgojcic, AnalMom, xrenaa, hturki, jayw | DIFIX3D+ enhances 3D reconstruction and novel-view synthesis using single-step diffusion models. The main research objective is to improve the quality of 3D reconstructions, especially in under-constrained regions, by leveraging 2D diffusion model priors. The methodology involves fine-tuning a single-step image diffusion model (DIFIX) to remove artifacts in rendered novel views, and using it both during reconstruction to clean pseudo-training views and as a neural enhancer during inference. Primary results show an average 2x improvement in FID score over baselines while maintaining 3D consistency, with compatibility across both NeRF and 3DGS representations. The principal implication is that AI practitioners can leverage single-step diffusion models for real-time post-processing to improve the visual quality of 3D reconstructions and novel view synthesis. |
| Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAs (Read more on arXiv or HuggingFace) | vishravmsft, martincai, alonbenhaim, jianmin-ustc, atabakashfaqMSFT | Phi-4-Mini and Phi-4-Multimodal are 3.8-billion-parameter language and multimodal models trained on high-quality data, achieving strong performance relative to their size. Main research question or objective: To develop compact yet highly capable language and multimodal models that outperform similar-sized open-source models and rival larger models, using curated data and novel architecture techniques. Key methodology used: The researchers trained Phi-4-Mini on high-quality web and synthetic data, with emphasis on math and coding datasets, expanded the vocabulary to 200K tokens, used grouped query attention, and a fractional RoPE dimension. For Phi-4-Multimodal, they used a “Mixture of LoRAs” technique, integrating modality-specific LoRAs while freezing the base language model. Primary results: Phi-4-Mini outperformed similarly sized models and matched the performance of models twice its size on math/coding, and Phi-4-Multimodal ranked first on the OpenASR leaderboard at the time, with the speech/audio LoRA having only 460 million parameters. Phi-4-Multimodal outperformed larger vision-language models, and achieved 72.0 average score across various vision-language benchmarks. Principal implication for AI practitioners: AI/ML/Software Engineers and Data Scientists can leverage Phi-4-Mini and Phi-4-Multimodal as efficient and performant small language and multimodal models, achieving strong performance while keeping the base language model frozen, making it a practical solution in resource-constrained environments. |
| OneRec: Unifying Retrieve and Rank with Generative Recommender and Iterative Preference Alignment (Read more on arXiv or HuggingFace) | GuoruiZhou, DingWF, caikuo, oneself, OrpheusBetter | OneRec is an end-to-end generative recommendation model that unifies retrieval and ranking stages. The main research objective is to develop a single-stage generative model that surpasses the performance of traditional multi-stage recommender systems in real-world scenarios. The key methodology involves an encoder-decoder architecture with Mixture-of-Experts (MoE), session-wise generation, and Iterative Preference Alignment (IPA) combined with Direct Preference Optimization (DPO) using a reward model. Primary results show that OneRec deployed in Kuaishou’s main scene achieved a 1.68% increase in watch-time, a substantial improvement over the previous system. For AI practitioners, OneRec demonstrates the feasibility of achieving significant performance gains by replacing a cascaded ranking system with a unified generative model by utilizing techniques like MoE and IPA. |
| Liger: Linearizing Large Language Models to Gated Recurrent Structures (Read more on arXiv or HuggingFace) | Yu Cheng, JusenK, Jiaxihu2, weigao266, landisen | Liger transforms pretrained Transformer-based large language models (LLMs) into gated linear recurrent structures for efficient deployment. The main research objective is to linearize LLMs into gated recurrent structures without adding extra parameters and with minimal performance loss. The key methodology involves repurposing pretrained key matrix weights to construct gating mechanisms and using Low-Rank Adaptation (LoRA) for lightweight fine-tuning. The primary result is that Liger recovers 93% of the Transformer-based Llama-3 8B model’s performance using only 0.02% of pre-training tokens during linearization. AI practitioners can deploy LLMs more efficiently with linear-time inference and constant memory usage by converting them to gated recurrent structures using Liger. |
| When an LLM is apprehensive about its answers – and when its uncertainty is justified (Read more on arXiv or HuggingFace) | Alexey Zaytsev, Edvard Khalafyan, DanielVyazhev, aigoncharov, sspetya | The paper investigates uncertainty estimation in Large Language Models (LLMs) for multiple-choice question answering, focusing on entropy and model-as-judge (MASJ) approaches. The main research question is how well token-wise entropy and MASJ estimates reflect LLM error and question difficulty across different domains and reasoning requirements. The key methodology involves evaluating three LLMs (Phi-4, Mistral, Qwen) on the MMLU-Pro dataset, using an auxiliary LLM to label questions by reasoning/knowledge needs and comparing uncertainty estimates with correctness labels. A primary result is that response entropy predicts model error effectively in knowledge-dependent domains (biology ROC AUC = 0.73), but this correlation weakens for reasoning-dependent domains (math ROC AUC = 0.55). For AI practioners this indicates, that the data-uncertainty related entropy is a useful measure in uncertainty estimate frameworks and should be integrated, but its usefulness is dependent to how much reasoning is requred to solve the problem. |
| DiffRhythm: Blazingly Fast and Embarrassingly Simple End-to-End Full-Length Song Generation with Latent Diffusion (Read more on arXiv or HuggingFace) | Guobin Ma, Chunbo Hao, Yuepeng Jiang, Huakang Chen, Ziqian Ning | DiffRhythm is a latent diffusion-based model that generates full-length songs with vocals and accompaniment, achieving high musicality, intelligibility, and fast inference speeds. The main research objective is to develop an end-to-end song generation model capable of synthesizing complete songs (up to 4m45s) with both vocal and accompaniment, overcoming limitations of existing approaches like multi-stage architectures and slow inference. Key methodology involves a Variational Autoencoder (VAE) for learning compact latent representations of waveforms and a Diffusion Transformer (DiT) operating in the latent space, along with a novel sentence-level lyrics alignment mechanism. Primary results show that DiffRhythm achieves a Phoneme Error Rate (PER) of 18.02% in full-length song generation with a real-time factor (RTF) of 0.034. AI practitioners can leverage DiffRhythm’s simple architecture, fast non-autoregressive generation, and open-sourced code/models for scalable, end-to-end song generation research and applications, eliminating the need for complex multi-stage cascading modelling. |
| Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRs (Read more on arXiv or HuggingFace) | ngoodman, nlile, Asap7772, ayushchakravarthy, obiwan96 | i) This paper investigates cognitive behaviors that enable language models to effectively self-improve via reinforcement learning. ii) The research question is: what intrinsic properties enable effective self-improvement in language models trained with reinforcement learning? iii) The methodology involves analyzing verification, backtracking, subgoal setting, and backward chaining in Qwen and Llama models during reinforcement learning on the Countdown game, alongside controlled behavioral dataset experiments and pretraining data curation. iv) Results show that Qwen naturally exhibits reasoning behaviors whereas Llama lacks them, priming Llama with these behaviors enables substantial improvements during RL; models primed with incorrect solutions but proper reasoning patterns achieve comparable performance to those trained on correct solutions, and curated pretraining data amplified Llama’s reasoning behaviors. v) AI practitioners should consider the initial reasoning behaviors of language models as a critical factor in determining their capacity for self-improvement via reinforcement learning, and potentially curate pretraining data to enhance those behaviors. |
| Speculative Ad-hoc Querying (Read more on arXiv or HuggingFace) | Venkat Arun, Aditya Akella, Maria Angels de Luis Balaguer, Srikanth Kandula, Haoyu0529 | SpeQL, a system that reduces query latency by using large language models (LLMs) to predict and precompute SQL queries during user input, improves analytical query responsiveness. The research objective is to determine if query execution can begin before a user finishes typing an SQL query, enabling near-instantaneous results. The methodology involves using LLMs to predict query structure and precompute temporary tables, alongside a scheduler that manages query execution and a user interface that displays speculative results. Results from experiments on 103 TPC-DS queries at 100GB scale show that SpeQL reduces P90 planning, compilation, and execution latency by 94.42%, 99.99%, and 87.23%, respectively, with a 7.72 seconds P90 execution overhead. AI practitioners can leverage SpeQL’s approach to improve the responsiveness of interactive data analysis systems, thereby enabling quicker insight discovery during exploratory data analysis. |
| Qilin: A Multimodal Information Retrieval Dataset with APP-level User Sessions (Read more on arXiv or HuggingFace) | Xiaohui He, Jia Chen, aiqy, haitaoli, qian | Qilin is a new multimodal information retrieval dataset collected from a social platform, Xiaohongshu, for improving search and recommendation services. The main research objective is to create a dataset that facilitates the development of advanced multimodal neural retrieval models across diverse task settings with real-world user interaction data. The key methodology involves collecting user sessions with heterogeneous results (image-text, video, commercial notes, direct answers) and APP-level contextual signals, then filtering the data using LLMs and human verification for safety and privacy. Primary results include a dataset of APP-level sessions from 15,482 users, where search users browse an average of 23.41 items when Deep Query Answering (DQA) is not triggered, but only 10.61 items when DQA is triggered. Principal implication for AI practitioners is that Qilin provides a realistic, large-scale, multimodal dataset with rich contextual information for training, evaluating, and analyzing retrieval-augmented generation systems and other advanced search and recommendation models, taking into account complex user behaviors. |
| DuoDecoding: Hardware-aware Heterogeneous Speculative Decoding with Dynamic Multi-Sequence Drafting (Read more on arXiv or HuggingFace) | xpqiu, QipengGuo, KYLN24, KaiLv | DuoDecoding is a novel speculative decoding method that leverages heterogeneous hardware to accelerate large language model inference. The main research objective is to reduce generation latency in large language models (LLMs) while maintaining output distribution fidelity and reducing the time to first token (TTFT). The key methodology involves deploying the draft model on the CPU and the target model on the GPU, enabling parallel decoding, along with a hardware-aware optimal draft budget and dynamic multi-sequence drafting. DuoDecoding achieves up to a 2.61x speedup in generation latency compared to vanilla autoregressive generation and reduces TTFT to 83% of that in conventional speculative decoding. The principal implication for AI practitioners is that DuoDecoding provides a method to significantly improve the inference speed of LLMs, particularly beneficial for interactive applications, by utilizing both CPU and GPU resources effectively. |
| Kiss3DGen: Repurposing Image Diffusion Models for 3D Asset Generation (Read more on arXiv or HuggingFace) | yingcongchen, Xxlbigbrother, StarYDY, MeixiChen, LTT | Kiss3DGen is a framework that repurposes 2D image diffusion models for 3D asset generation, including tasks like text-to-3D, image-to-3D, editing, and enhancement. The main research objective is to develop an efficient method for generating, editing, and enhancing 3D objects by leveraging pretrained 2D image diffusion models, without the need of large-scale 3D datasets. The key methodology involves fine-tuning a diffusion model (Flux) to generate “3D Bundle Images”—tiled representations of multi-view images and normal maps—which are then used to reconstruct a 3D mesh. The method achieves a CLIP score of 0.837 in text-to-3D generation evaluation, outperforming 3DTopia, Direct2.5, and Hunyuan3D-1.0. AI practitioners can utilize this framework to efficiently create high-quality 3D models by maximizing the use of pre-trained 2D diffusion models, thus reducing the dependency on extensive 3D training data. |
| Word Form Matters: LLMs’ Semantic Reconstruction under Typoglycemia (Read more on arXiv or HuggingFace) | Lang Gao, Zhongyu Wei, Ziruibest, Carol0110, Aurora-cx | Large Language Models (LLMs) reconstruct the meaning of scrambled words primarily using word form, with minimal reliance on contextual information. The main research question is how word form and contextual information influence LLMs’ semantic reconstruction ability under Typoglycemia. The researchers used controlled experiments on LLaMA models, varying Scramble Ratio (SR) and Context Integrity (CI), and introduced SemRecScore to quantify semantic reconstruction. Primary results show SemRecScore decreases as SR increases, and at a Scramble Ratio (SR) of 1, a final SemRecScore of only 0.5 is achieved on the final LLM layer, indicating incomplete semantic reconstruction. For AI practitioners, this highlights that improvements can come by incorporating human-like, context-aware mechanisms, as current attention mechanisms focus primarily on the word form. |
| SampleMix: A Sample-wise Pre-training Data Mixing Strategey by Coordinating Data Quality and Diversity (Read more on arXiv or HuggingFace) | bitwjg, WeiWang, WQYC, DeyangKong, xixy | SampleMix is a sample-wise pre-training data mixing strategy for large language models that coordinates data quality and diversity. The main research objective is to address the limitations of existing domain-wise data mixing methods, which overlook inter-domain overlaps and use suboptimal sample distributions. The key methodology involves evaluating the quality and diversity of each sample, assigning sampling weights, and constructing a training dataset based on these weights. The primary results show that SampleMix achieves an average accuracy of 47.77% across eight downstream tasks, outperforming all baseline methods, and reaching baseline performance with 1.9x fewer training steps. The principal implication is that AI practitioners can use SampleMix to improve training efficiency and model performance by creating better data mixtures by incorporating sample-wise quality and diversity evaluations. |
| From Hours to Minutes: Lossless Acceleration of Ultra Long Sequence Generation up to 100K Tokens (Read more on arXiv or HuggingFace) | Yuxuan Wang, zlzheng, vickyandkekey, JunzheS, TongWu | TOKENSWIFT accelerates ultra-long sequence generation for large language models without compromising output quality. The main research question is whether model-agnostic, lossless acceleration can be achieved for generating ultra-long sequences with minimal training overhead. The key methodology involves multi-token parallel self-drafting with the target model, token reutilization, dynamic KV cache management, and contextual penalty. Primary results show that TOKENSWIFT achieves over 3x speedup compared to autoregressive generation across various models, reducing generation time for 100K tokens on LLAMA3.1-8b from nearly 5 hours to 90 minutes. Principal implication for AI practitioners is TOKENSWIFT provides a scalable and effective solution to dramatically speed up ultra long text generation, enabling applications that require producing very large outputs. |
| Unposed Sparse Views Room Layout Reconstruction in the Age of Pretrain Model (Read more on arXiv or HuggingFace) | Jianan Wang, Xili Dai, xyyue, qixianbiao, yxuan | The paper introduces Plane-DUSt3R, a novel method for multi-view room layout estimation using the DUSt3R 3D foundation model. The main research objective is to develop a method for 3D room layout estimation from multiple unposed, sparse-view images. The methodology involves fine-tuning DUSt3R on a room layout dataset with a modified objective to estimate structural planes and combining it with a 2D plane detector and a post-processing algorithm. The Plane-DUSt3R achieves a 5.27% and 5.33% improvement in RRA and mAA metrics, respectively, for multi-view correspondence tasks, compared to state-of-the-art methods on the Structure3D dataset. AI practitioners can use Plane-DUSt3R to generate 3D room layouts from unposed images, eliminating the need for precise camera poses and simplifying multi-view 3D reconstruction. |
| CodeArena: A Collective Evaluation Platform for LLM Code Generation (Read more on arXiv or HuggingFace) | terryyz, DongHuang-ebay, bobxwu, anhtuanluu36, Elfsong | CodeArena is an online platform for evaluating large language models (LLMs) on code generation tasks, incorporating a collective evaluation mechanism. The main objective is to address limitations in existing LLM code generation evaluation, such as benchmark contamination, data dissipation, and system inaccessibility. The key methodology involves a dynamic scoring system that adjusts model scores based on the collective performance of all submissions, along with providing automation-friendly APIs and open access to solutions and test cases. Results show that closed-source LLMs generally outperform open-source models, with “DeepSeek-Coder” achieving a Dynamic Point score of 249.28 and solving 90.63% of the problems. AI practitioners can use CodeArena for unbiased LLM code generation evaluation, accessing a public repository of solutions and test cases, and streamlining the evaluation process with automation-ready APIs. |
| VideoUFO: A Million-Scale User-Focused Dataset for Text-to-Video Generation (Read more on arXiv or HuggingFace) | Yi Yang, WenhaoWang | VideoUFO is a million-scale video dataset designed to align text-to-video generation models with real-world user preferences. The main research objective is to curate a video dataset that reflects user-focused topics and evaluate its impact on text-to-video model performance. The key methodology involves clustering user-provided prompts from VidProM to identify 1,291 topics, retrieving relevant videos from YouTube, segmenting them into clips, generating captions, and assessing video quality using VBench. Primary results show that a model trained on VideoUFO achieves a low-10 score of 0.442, outperforming models trained on other datasets, while maintaining a top-10 score of 0.651 on a benchmark of user-focused topics. For AI practitioners, the VideoUFO dataset provides a resource for training or fine-tuning text-to-video models to better meet user expectations in real-world, diverse applications. |
| Large-Scale Data Selection for Instruction Tuning (Read more on arXiv or HuggingFace) | pradeepd, pangwei, faezeb, nanami, hamishivi | This paper systematically investigates the scaling properties of automated data selection methods for instruction-tuning language models. The main research objective is to determine how well various data selection approaches perform when selecting large datasets (up to 2.5M samples) from large pools (up to 5.8M samples) for instruction tuning. The key methodology involves comparing nine data selection techniques, including representation-based, gradient-based, and loss/perplexity-based methods, across multiple dataset sizes and selection pools, evaluating performance on seven diverse tasks. The primary result is that a variant of representation-based data selection (RDS+) consistently outperforms other methods, including random selection, achieving an average score of 50.5 versus 46.4 for the next best method (Embed (GTR)) when selecting 10k data points. This implies that AI practitioners should consider using the proposed simple, embedding-based RDS+ method, especially in large-scale settings, rather than more computationally expensive methods when selecting data for finetuning LLMs. |
| Direct Discriminative Optimization: Your Likelihood-Based Visual Generative Model is Secretly a GAN Discriminator (Read more on arXiv or HuggingFace) | mingyuliutw, gdhe17, HuayuChen, Ema11, worstcoder | Direct Discriminative Optimization (DDO) finetunes likelihood-based visual generative models using a GAN-inspired objective without extra networks. The research aims to improve the sample quality of likelihood-based generative models beyond the limitations of maximum likelihood estimation (MLE). DDO implicitly parameterizes a discriminator using the likelihood ratio between a learnable target model and a fixed, pretrained reference model, optimizing the target model with a GAN discriminator loss. Finetuning a diffusion model (EDM) with DDO achieved a new record FID score of 1.30 on CIFAR-10, a significant improvement over the base model’s 1.79. AI practitioners can directly finetune and iteratively refine pretrained likelihood-based generative models to achieve state-of-the-art performance without modifying model architecture or inference procedures. |
| AI-Invented Tonal Languages: Preventing a Machine Lingua Franca Beyond Human Understanding (Read more on arXiv or HuggingFace) | dnoever | This paper explores the potential for large language models (LLMs) to create private tonal languages for machine-to-machine communication. The main research question is whether AI agents can autonomously invent and use private tonal languages, and what those languages might resemble. The key methodology involves implementing a character-to-frequency mapping system using musical semitones to encode the full ASCII character set, creating a prototype tonal language. Primary results demonstrate that tonal encoding can achieve information rates exceeding human speech, with the ASCII mapping spanning approximately 7.8 octaves (220 Hz to 50175.42 Hz). The principle implication for AI practioners is that LLMs could theoretically engage in M2M communications, partially or wholly, outside of human perceptual boundaries, raising a need for transparency, oversight, and governance strategies in AI development. |
| CLEA: Closed-Loop Embodied Agent for Enhancing Task Execution in Dynamic Environments (Read more on arXiv or HuggingFace) | Qing Zhao, Zhixin Mai, Yiming Zhao, Ge Wang, SP4595 | CLEA is a closed-loop embodied agent framework that enhances task execution in dynamic environments using multiple LLMs. The main research objective is to address the limitations of Large Language Models (LLMs) in embodied systems for reliable execution of subtask sequences and one-shot success in long-term tasks within dynamic environments. The key methodology involves a closed-loop architecture with four specialized open-source LLMs and a planner-critic framework, integrating environmental memory and multimodal feedback for dynamic task management. Across 12 task trials, CLEA achieved a 67.3% improvement in success rate and a 52.8% increase in task completion rate compared to the open-loop baseline. For AI practitioners, the framework offers a robust method for deploying embodied agents in real-world, dynamic settings by facilitating adaptive strategy adjustment, enhancing task planning, and improving execution through continuous environmental feedback. |
Papers for 2025-03-03
| Title | Authors | Summary |
|---|---|---|
| DeepSolution: Boosting Complex Engineering Solution Design via Tree-based Exploration and Bi-point Thinking (Read more on arXiv or HuggingFace) | luyaojie, sanmusunrise, xuanang, yhycai, lzq2021 | The paper introduces a new benchmark and system for complex engineering solution design. The main research objective is to evaluate and improve systems’ ability to generate complete and feasible solutions for engineering problems with multiple constraints. The key methodology is SolutionRAG, leveraging tree-based exploration and a bi-point thinking mechanism (alternating solution design and review) to generate solutions. SolutionRAG achieved a 66.4 analytical score and 67.9 technical score on the SolutionBench, outperforming baselines like Naive-RAG and Self-RAG. AI practitioners can use SolutionBench to benchmark and the SolutionRAG architecture to improve the generation of solutions for complex, multi-constraint engineering problems. |
| Chain of Draft: Thinking Faster by Writing Less (Read more on arXiv or HuggingFace) | Lingxiao Zhao, Wenhao Xie, DeBERTa, sileixu | Chain of Draft (CoD) is a new prompting strategy that improves the efficiency of large language models (LLMs) by generating concise reasoning steps. The research proposes and evaluates Chain of Draft (CoD), a prompting method that minimizes verbosity in LLM reasoning. CoD prompts LLMs to produce brief, information-dense intermediate steps, resembling human draft-thinking, during multi-step reasoning tasks. The results show that CoD matches or surpasses Chain-of-Thought (CoT) accuracy on GSM8K, date, sports, and coin flip tasks, while using up to 92.4% fewer tokens in a specific Sports Understanding case. AI practitioners can use CoD to reduce latency and computational costs in LLM applications without significantly sacrificing accuracy, especially in resource-constrained environments. |
| ViDoRAG: Visual Document Retrieval-Augmented Generation via Dynamic Iterative Reasoning Agents (Read more on arXiv or HuggingFace) | xpjandy, shihang, vickywu, lovesnowbest, autumncc | ViDoRAG is a multi-agent RAG framework for visually-rich documents using dynamic retrieval and iterative reasoning. The main research objective is to address the limitations of existing RAG methods in handling visually rich documents, particularly the challenges of multi-modal retrieval and insufficient reasoning capabilities. The methodology employs a Gaussian Mixture Model (GMM)-based hybrid retrieval strategy (textual and visual) and a multi-agent framework (seeker, inspector, answer) for iterative reasoning. Primary results show ViDoRAG outperforms existing methods on the ViDoSeek benchmark by over 10% in overall accuracy. AI practitioners can leverage ViDoRAG’s multi-agent framework and dynamic retrieval strategy to build more effective and robust RAG systems for applications dealing with visually rich documents. |
| SoS1: O1 and R1-Like Reasoning LLMs are Sum-of-Square Solvers (Read more on arXiv or HuggingFace) | Coralia Cartis, Wenqi Zhu, Kechen Li, Shiweiliuiiiiiii, jitianbo | Large Language Models (LLMs) can be effectively used to solve sum-of-squares (SoS) polynomial problems with proper reasoning guidance. The main research question is whether LLMs can determine the nonnegativity of a given multivariate polynomial, a computationally intractable problem related to Hilbert’s Seventeenth Problem. The researchers introduced a dataset (SoS-1K) of ~1,000 polynomials and evaluated various LLMs using plain questions, simple instructions, and expert-designed reasoning instructions based on five criteria. The results show that high-quality reasoning instructions significantly improve accuracy, with the best-performing model (DeepSeek-R1) reaching 81% accuracy with SoS Reasoning instructions, compared to around 60% with plain question. Supervised fine-tuning of a 7B model on SoS-1K achieved 70% accuracy outperforming the 671B Deepseek-V3. AI practitioners can leverage specialized datasets and reasoning-guided instructions to significantly enhance LLMs’ ability to solve complex mathematical problems and tackle NP-hard problems. |
| Optimal Brain Apoptosis (Read more on arXiv or HuggingFace) | Delei Kong, Junjie Jiang, Jiaxu Wang, Zheng Fang, Mingyuan Sun | Optimal Brain Apoptosis (OBA) is a novel pruning method that calculates the Hessian-vector product to estimate parameter importance for neural network compression. The main research objective is to develop a more precise and efficient pruning method that avoids approximations of the Hessian matrix used in prior work. The key methodology involves decomposing the Hessian matrix across network layers, identifying conditions for non-zero inter-layer Hessian submatrices, and efficiently computing the second-order Taylor expansion of parameters using a Jacobian-vector product forward propagation technique. The primary results show that OBA achieves a 2x speedup on ImageNet with ResNet50 with only a 0.53% accuracy decrease, outperforming existing methods. The principal implication for AI practitioners is that OBA offers a more accurate and efficient way to prune both convolutional neural networks and Transformers, directly leading to computational savings in inference. |
| Tell me why: Visual foundation models as self-explainable classifiers (Read more on arXiv or HuggingFace) | Christian Lovis, Gianmarco Mengaldo, Mina Bjelogrlic, hturbe | Visual foundation models (VFMs) can be adapted into self-explainable classifiers through a novel prototypical architecture called ProtoFM. The main research objective is to develop a self-explainable model (SEM) leveraging VFMs that achieves competitive classification performance and improved interpretability. The methodology involves training a lightweight head (approximately 1 million parameters) on top of frozen VFMs, using a student-teacher approach and specialized training objectives, including assignment, alignment, contrastive, sparsity, and classification losses. The ProtoFM architecture achieved a mean explainability score (mX) of 0.92 on the FunnyBirds framework, outperforming existing prototypical models. AI practitioners can leverage frozen VFMs to create efficient and interpretable classifiers, improving transparency and trust, particularly in critical applications. |
| Sim-to-Real Reinforcement Learning for Vision-Based Dexterous Manipulation on Humanoids (Read more on arXiv or HuggingFace) | Yuke Zhu, Linxi Fan, Kartik Sachdev, Toru Lin, jitendra1995 | This paper presents a sim-to-real reinforcement learning recipe for vision-based dexterous manipulation tasks on humanoid robots. The main research objective is to identify and address the key challenges in applying sim-to-real reinforcement learning to solve contact-rich dexterous manipulation tasks on humanoids. The key methodology includes an automated real-to-sim tuning module, a generalized reward design scheme, a divide-and-conquer distillation process, and a mixture of sparse and dense object representations. The primary results include a 62.3% success rate on the grasp-and-reach task, 80% on the box lift task, and 52.5% on bimanual handover, demonstrating generalization and robustness against force perturbations; also shown is the correlation that lower MSE measured by autotune module and higher sim-to-real transfer success rate. AI practitioners can utilize the proposed techniques to train humanoid robots for dexterous manipulation, achieving robust generalization and high performance without human demonstrations. |
| LiteASR: Efficient Automatic Speech Recognition with Low-Rank Approximation (Read more on arXiv or HuggingFace) | kasikci, kojimano, jungok, kamahori | LITEASR is a compression scheme for ASR encoders that maintains transcription accuracy while reducing computational costs. The main research objective is to reduce the computational intensity of ASR encoders, which are a deployment bottleneck. The key methodology leverages low-rank properties in intermediate activations by applying PCA and optimizing self-attention in a reduced dimension, implemented using a specialized GPU kernel. Applying LITEASR to Whisper large-v3 reduces encoder size by over 50%, matching Whisper medium’s size with better transcription accuracy. AI practitioners can deploy more efficient ASR systems by leveraging the compressed, and Pareto-optimal, models. |
| HAIC: Improving Human Action Understanding and Generation with Better Captions for Multi-modal Large Language Models (Read more on arXiv or HuggingFace) | Fuzheng Zhang, Yuanxing Zhang, Jingyun Hua, Xiao Wang, lwher1996 | This paper introduces HAIC, a two-stage data annotation pipeline and two datasets, to improve human action understanding and generation in multi-modal large language models (MLLMs). The main research objective is to address the lack of high-quality data for training MLLMs on videos involving human actions, especially multi-person interactions. The methodology involves a two-stage data annotation pipeline: accumulating videos with clear human actions, and annotating videos with a standardized caption format detailing individual attributes, actions, and interactions. Training with the curated HAICTrain dataset improves human action understanding, as evidenced by a 2.1% accuracy improvement on the HAICBench benchmark compared to the baseline LLaVA-Video-7B model. AI practitioners can use the released datasets and annotation pipeline to enhance MLLMs’ performance in tasks requiring fine-grained understanding of human actions and interactions in videos. |
Papers for 2025-02-28
| Title | Authors | Summary |
|---|---|---|
| Self-rewarding correction for mathematical reasoning (Read more on arXiv or HuggingFace) | Nan Jiang, Chenlu Ye, Hanning Zhang, Wei Xiong, Lichang-Chen | This paper introduces a self-rewarding reasoning framework for large language models (LLMs) that enables autonomous error detection and correction in mathematical reasoning without external feedback. The main research question is whether LLMs can simultaneously generate reasoning steps, evaluate their correctness, and revise their outputs during inference without external reward models. The key methodology involves a two-staged training approach using self-generated data: sequential rejection sampling to create training trajectories, followed by reinforcement learning with rule-based signals. Primary results show that on the MATH500 benchmark, the self-rewarding IFT + PPO model achieves a final accuracy of 80.2%, outperforming intrinsic self-correction and comparable to systems using external reward models. For AI practitioners, this framework offers a way to improve LLM reasoning accuracy and reduce computational overhead by integrating generation and evaluation within a single model, streamlining deployment for mathematical reasoning tasks. |
| MedVLM-R1: Incentivizing Medical Reasoning Capability of Vision-Language Models (VLMs) via Reinforcement Learning (Read more on arXiv or HuggingFace) | Jiayuan Zhu, Fenglin Liu, Jiazhen Pan, morson, che111 | MedVLM-R1 is a medical vision-language model that uses reinforcement learning to generate explicit reasoning alongside answers for radiology visual question answering. The main research objective is to develop a medical VLM that generates natural language reasoning to improve transparency and trustworthiness, without relying on supervised fine-tuning (SFT). The key methodology is a reinforcement learning framework, specifically Group Relative Policy Optimization (GRPO), that incentivizes the model to discover human-interpretable reasoning paths without using reasoning references. The model, trained on 600 visual question answering samples, boosts accuracy from 55.11% to 78.22% across MRI, CT, and X-ray benchmarks, outperforming larger models. For AI practitioners, this implies that training smaller, specialized models with reinforcement learning can achieve superior, robust, and transparent generalization in the medical domain relative to supervised fine-tuning approaches. |
| R2-T2: Re-Routing in Test-Time for Multimodal Mixture-of-Experts (Read more on arXiv or HuggingFace) | Ziyue Li, zhoutianyi, Lzy01241010 | R2-T2 introduces a test-time re-routing method for multimodal Mixture-of-Experts (MoE) models that improves performance without retraining. The main research objective is to optimize the routing weights of a multimodal MoE model during inference to improve performance on challenging or out-of-distribution samples. The key methodology is “Re-Routing in Test-Time (R2-T2),” which locally optimizes routing weights by moving them toward those of correctly predicted neighbor samples, using strategies like Neighborhood Gradient Descent (NGD), kernel regression, and mode finding. Applying R2-T2 with NGD to MoAI-7B improved MMBench accuracy by 6.9%, TextVQA accuracy by 6.8%, and achieved a 66.1-point increase on MME-P. AI practitioners can use R2-T2 to enhance the performance and generalization of multimodal MoE models on diverse tasks in test-time, without costly retraining or modification of model parameters. |
| LongRoPE2: Near-Lossless LLM Context Window Scaling (Read more on arXiv or HuggingFace) | Gilsinia Lopez, Gaokai Zhang, Li Lyna Zhang, Ning Shang, OldKingMeister | LongRoPE2 extends LLMs’ effective context window while preserving short-context performance through RoPE rescaling and mixed context window training. The main research objective is to address the out-of-distribution (OOD) issues in rotary positional embeddings (RoPE) and the performance degradation on short-context tasks when extending the context window of pre-trained large language models (LLMs). The key methodology involves an evolutionary search for optimal RoPE rescaling factors guided by “needle-driven” perplexity, combined with a mixed context window training approach that uses both original and rescaled RoPE. Primary results show that LongRoPE2 extends LLaMA3-8B to achieve a 128K effective context length while retaining over 98.5% of short-context performance, using only 10B training tokens. Principal implication is that AI practitioners can extend LLM context windows to 128K with near-lossless performance on both long and original context window, significantly reducing the data, and training costs compare to prior methods. |
| FINEREASON: Evaluating and Improving LLMs’ Deliberate Reasoning through Reflective Puzzle Solving (Read more on arXiv or HuggingFace) | Chaoqun Liu, Hou Pong Chan, Hao Zhang, Weiwen Xu, Guizhen Chen | FINEREASON introduces a logic-puzzle benchmark to evaluate and improve LLMs’ deliberate reasoning through state checking and transition tasks. The main research objective is to assess and enhance LLMs’ ability to reflect and rectify mistakes during multi-step reasoning processes, going beyond final-answer accuracy. The key methodology involves decomposing logic puzzles into atomic steps and evaluating models on two tasks: state checking (assessing if a state can lead to a solution) and state transition (determining the next valid move). Primary results show that models trained with state checking and transition data demonstrated gains in math reasoning by up to 5.1% on GSM8K, when starting from the DeepSeek-R1-Distill-Qwen-7B model, the accuracy increased from 82.3% to 87.4%. The principal implication for AI practitioners is that training LLMs with structured, puzzle-based data focusing on intermediate reasoning steps can significantly improve their performance on general mathematical reasoning tasks. |
| CODESYNC: Synchronizing Large Language Models with Dynamic Code Evolution at Scale (Read more on arXiv or HuggingFace) | Kaiyue Qiu, Zhaoyang Chu, Chenlong Wang, yxy0807, zx10086 | CODESYNC introduces a data engine and benchmark to assess large language models’ (LLMs) ability to adapt to evolving Python library APIs. The main research question is: Can LLMs be effectively and efficiently updated to handle real-time API modifications? CODESYNC systematically identifies API updates, retrieves relevant code instances from GitHub, and uses an LLM to synthesize contrastive code for legacy/updated API versions, then builds a benchmark,CODESYNCBENCH. Evaluation of 14 LLMs shows they struggle with API updates even with knowledge updating methods, e.g. a maximum BLEU score of 31.59 on the code completion task across five models with SFT. The principal implication is that AI practitioners need to develop and employ techniques to improve LLMs’ ability to synchronize with evolving code, as static pre-training datasets limit handling of real-time API updates. |
| Lean and Mean: Decoupled Value Policy Optimization with Global Value Guidance (Read more on arXiv or HuggingFace) | Zhixu Li, Pu Zhao, Lu Wang, Chenghua Huang, keanudicap | DVPO decouples value and policy optimization in RLHF to improve training efficiency and stability for large language models. The main research objective is to address the computational complexity and instability of traditional PPO-based RLHF caused by joint actor-critic training. The key methodology is Decoupled Value Policy Optimization (DVPO), which pre-trains a Global Value Model (GVM) on policy trajectories and uses it as a fixed guide for policy optimization via a standard RL objective. Primary results show that DVPO reduces GPU memory usage by 40% and training time by 35% compared to conventional RLHF, while achieving comparable performance to state-of-the-art PPO. The principal implication is that AI practitioners can achieve more efficient and stable RLHF training by decoupling value estimation from policy updates, simplifying the alignment of LLMs with human preferences. |
| UniTok: A Unified Tokenizer for Visual Generation and Understanding (Read more on arXiv or HuggingFace) | Xin Yu, Jihan Yang, Junfeng Wu, Yi Jiang, Chuofan Ma | UniTok is a unified visual tokenizer designed for both visual generation and understanding tasks, bridging the representation gap between these two domains. The main research objective is to investigate whether reconstruction and contrastive losses truly conflict in unified tokenizer training, and to identify any underlying bottlenecks. The key methodology is multi-codebook quantization, which divides visual tokens into chunks and discretizes each with independent sub-codebooks, alongside attention factorization. UniTok achieves a remarkable rFID of 0.38 and a zero-shot accuracy of 78.6% on ImageNet. The principal implication for AI practitioners is that a unified visual tokenizer, enhanced with multi-codebook quantization, can match or surpass domain-specific tokenizers, enabling more efficient and integrated multimodal model development. |
| FlexiDiT: Your Diffusion Transformer Can Easily Generate High-Quality Samples with Less Compute (Read more on arXiv or HuggingFace) | Markos Georgopoulos, Jonas Kohler, Yeongmin Kim, Gregor Bachmann, Sotiris Anagnostidis | FlexiDiT enables Diffusion Transformers (DiTs) to generate high-quality images with reduced computational cost by dynamically adjusting the compute budget per denoising step. The main research objective is to overcome the fixed and large compute requirements of standard DiTs during inference by revisiting the static compute allocation paradigm. The key methodology is converting pre-trained DiT models into flexible ones (FlexiDiTs) that can process inputs at varying compute budgets by dynamically adjusting patch size during the denoising process, and using different LoRAs for each sequence. The primary result is that FlexiDiT models can reduce FLOPs by more than 40% compared to static counterparts for class-conditioned and text-conditioned image generation, without any drop in quality. AI practitioners can deploy more computationally efficient diffusion models by adopting FlexiDiT, enabling substantial savings in computational resources without compromising the quality of generated outputs, especially for high-resolution image and video generation. |
| Multimodal Representation Alignment for Image Generation: Text-Image Interleaved Control Is Easier Than You Think (Read more on arXiv or HuggingFace) | Haozhe Zhao, Weichu Xie, Wenhao Chai, Shuai Bai, Liang Chen | DREAM ENGINE enables arbitrary text-image interleaved control for image generation by aligning large multimodal models (LMMs) with diffusion models. The research objective is to develop a framework that can generate images based on complex instructions interweaving text and visual elements from multiple images. The key methodology involves replacing the text encoders of a diffusion model (SD3.5) with an LMM (QwenVL) and a two-stage training paradigm: joint text-image alignment and multimodal interleaved instruction tuning. The primary results show that DREAM ENGINE achieves a 0.69 overall score on the GenEval benchmark, matching state-of-the-art text-to-image models. For AI practitioners, the principal implication is that LMMs can be directly integrated into diffusion models to enable advanced text-image control, simplifying the creation of complex, multi-image-influenced generation systems. |
| NeoBERT: A Next-Generation BERT (Read more on arXiv or HuggingFace) | Sarath Chandar, Mariam El Mezouar, Quentin Fournier, Lola Le Breton | NeoBERT, a new BERT-like encoder model, integrates architectural, data, and pre-training advancements to improve bidirectional representation learning. The primary objective is to create a next-generation BERT model that outperforms existing encoders by leveraging modern advancements in language model design. The key methodology involves pre-training on the RefinedWeb dataset with modifications like RoPE, SwiGLU, RMSNorm, a 20% masking rate, and a two-stage sequence length increase (1,024 to 4,096 tokens). NeoBERT achieves an 89.0 average score on the GLUE benchmark and 51.3 on the MTEB benchmark after contrastive fine-tuning, outperforming all similarly-sized and even larger, models on MTEB. AI practitioners can adopt NeoBERT as a plug-and-play replacement for existing base encoders to obtain better performance in downstream NLP tasks that depend on their embeddins, notably for retrieval-augmented generation and toxicity classification, without needing architectural modifications. |
| Mobius: Text to Seamless Looping Video Generation via Latent Shift (Read more on arXiv or HuggingFace) | Xiaodong Cun, Yong Zhang, Bo Liu, Jianfei Yuan, Xiuli Bi | Mobius is a training-free method to generate seamless looping videos from text descriptions using pre-trained video diffusion models. The main research objective is to develop a method for generating seamless looping videos directly from text prompts, without requiring user annotations or additional training. The key methodology involves constructing a latent cycle and performing multi-frame latent denoising by iteratively shifting the first-frame latent towards the end in each step, while also using a frame-invariant latent decoding method. Primary results show that the proposed method achieves an MSE of 25.43 between the first and last frame, FVD of 40.78, a CLIP score of 32.24, and a Motion Smoothness score of 0.9850. For AI practitioners, this method provides a way to directly repurpose pre-trained text-to-video diffusion models for generating seamless looping videos, without the need for large scale training or annotated dataset. |
| SoRFT: Issue Resolving with Subtask-oriented Reinforced Fine-Tuning (Read more on arXiv or HuggingFace) | Yanzhen Zou, Xiangxin Meng, Pengfei Gao, Chao Peng, mizersy | SoRFT is a novel training approach that enhances large language models’ (LLMs) issue-resolving capabilities through subtask decomposition and reinforced fine-tuning. The main research objective is to improve the performance and generalization of open-source LLMs on software issue resolution tasks, addressing limitations of existing methods. The key methodology involves decomposing issue resolving into subtasks (file/function/line localization, code edit generation) and using rejection-sampled supervised fine-tuning followed by rule-based proximal policy optimization (PPO) with ground-truth-based rewards. The primary result is that SoRFT-Qwen-7B achieves 21.4% resolution rate on SWE-Bench Verified, outperforming other open-source models of similar size. For AI practitioners, SoRFT offers a cost-effective way to leverage open-source development resources and substantially boost the performance of open-source LLMs in automated issue resolution. |
| Building Interactable Replicas of Complex Articulated Objects via Gaussian Splatting (Read more on arXiv or HuggingFace) | Song-Chun Zhu, Junfeng Ni, Ruijie Lu, Baoxiong Jia, Yu Liu | ArtGS introduces a method for reconstructing and modeling complex articulated objects using 3D Gaussian Splatting. The main research objective is to effectively integrate information across different object states to improve part-mesh reconstruction and articulation parameter estimation, especially for multi-part articulated objects. The key methodology involves using canonical Gaussians with coarse-to-fine initialization and updates, alongside a skinning-inspired part dynamics modeling module. Primary results show that on the PARIS dataset, ArtGS achieves a mean angular error (Axis Ang.) of 0.01 degrees and a mean Chamfer Distance for movable parts (CD-m) of 0.03, outperforming existing methods. For AI practitioners, this implies a more efficient and accurate approach to creating digital twins of articulated objects, facilitating applications in robotics and virtual environments. |
| R1-T1: Fully Incentivizing Translation Capability in LLMs via Reasoning Learning (Read more on arXiv or HuggingFace) | Hongyong Zeng, Yuanchang Luo, Shimin Tao, Yilun Liu, boommmmm | R1-T1 is a novel framework that enhances machine translation (MT) in large language models (LLMs) through reinforcement learning (RL) with human-aligned chain-of-thoughts (CoTs). The main research objective is to improve the adaptability of LLMs to diverse translation scenarios by incorporating inference-time reasoning into general MT, going beyond specific sub-tasks. The key methodology involves formalizing six expert-curated CoT templates, reflecting human translation strategies, and using RL with KL-constrained rewards for self-evolving CoT discovery and anti-forgetting adaptation. Primary results demonstrate steady translation performance improvement across 21 languages and 80 translation directions on the Flores-101 test set, with a COMETScore of 0.626 on trained languages using RL, surpassing supervised fine-tuning (SFT) and other baselines. Principal implication for AI practioners: It provides a method for using RL to adapt LLMs to new machine translation tasks without relying on the SFT data and avoiding the Catastrophic Forgetting issue. |
Papers for 2025-02-27
| Title | Authors | Summary |
|---|---|---|
| Kanana: Compute-efficient Bilingual Language Models (Read more on arXiv or HuggingFace) | seopbo, Doohae, daniel-rl2, jiyeonham, bzantium | Kanana is a series of bilingual language models demonstrating strong performance in Korean and competitive performance in English at a significantly lower computational cost than comparable state-of-the-art models. The main research objective was to develop compute-efficient bilingual language models that maintain strong performance in both Korean and English. The key methodologies employed include high-quality data filtering, staged pre-training, depth up-scaling, pruning, and distillation, combined with supervised fine-tuning and preference optimization for instruction tuning. Primary results show that the Kanana Flag 32.5B model outperforms Llama 3.1 70B on MMLU and KMMLU, while using substantially fewer computational resources, costing similiar to Gemma 2 9B. AI practitioners can leverage Kanana’s training techniques such as staged pre-training and depth-up scaling to build high-performing, resource-efficient language models, especially for languages with limited data availability. |
| GHOST 2.0: generative high-fidelity one shot transfer of heads (Read more on arXiv or HuggingFace) | Andrey Kuznetsov, Denis Dimitrov, Pavel Paramonov, Alexander Groshev, nastasia-y | GHOST 2.0 is a two-module framework for high-fidelity one-shot head swapping, addressing limitations in existing face-swapping and head-reenactment methods. The main research objective is to develop a system that can realistically swap entire heads between source and target images, preserving identity, pose, and expression while seamlessly blending the result. The key methodology involves an “Aligner” module for head reenactment and a “Blender” module for integrating the reenacted head into the target background, using StyleGAN-based architecture and correlation learning. Primary results show that at 512x512 resolution in cross-reenactment, GHOST 2.0 achieves a CSIM score of 0.628 and a FID score of 29.57, outperforming one of the baselines (StyleHEAT) and indicating better performace than another baseline (HeSer) at identity preservation. AI practitioners can use GHOST 2.0 to improve the realism and robustness of head-swapping applications, particularly in scenarios with significant variations in head pose, hairstyle, and background. |
| TheoremExplainAgent: Towards Multimodal Explanations for LLM Theorem Understanding (Read more on arXiv or HuggingFace) | Jonathan Leung, AlvinYuVotee, KrishKrosh, chongcht, vinesmsuic | TheoremExplainAgent, a novel agentic system, generates multimodal theorem explanation videos, and a new benchmark, TheoremExplainBench, evaluates them. The main research objective is to assess if AI systems can effectively generate multimodal theorem explanations. The key methodology involves a two-agent pipeline (planner and coding agent) using Manim to create videos, and a benchmark of 240 theorems across STEM, evaluated across five dimensions. The o3-mini agent achieved a 93.8% success rate and an overall score of 0.77, but visual element layout exhibited minor issues. AI practitioners can leverage this agentic approach for enhanced theorem understanding, though refinement is needed in visual structuring and consistency of generated video outputs. |
| Can Large Language Models Detect Errors in Long Chain-of-Thought Reasoning? (Read more on arXiv or HuggingFace) | Weixun Wang, Jiaheng Liu, Shilong Li, Yancheng He, zhangysk | DeltaBench, a new benchmark, evaluates large language models’ (LLMs) ability to detect errors in long chain-of-thought (CoT) reasoning. The main research objective is to assess the quality of long CoTs generated by o1-like models and to measure the critique abilities of existing LLMs, process reward models (PRMs) and critic models on these CoTs. The key methodology involves creating DeltaBench, a dataset of long CoTs with fine-grained error annotations, and evaluating various LLMs, including PRMs and critic models, on their ability to identify these errors. Primary results show that even the top-performing model (GPT-4-turbo-128k) achieved a low F1-score of only 40.8% in error detection, and that o1-like models do not show any advantage over non-o1-like models on critique abilities. Principal implication for AI practitioners is that current LLMs, including PRMs, have limited ability to identify errors in long CoT reasoning, highlighting a need for significant improvements in critique capabilities for robust AI system development. |
| Agentic Reward Modeling: Integrating Human Preferences with Verifiable Correctness Signals for Reliable Reward Systems (Read more on arXiv or HuggingFace) | Bin Xu, Zijun Yao, Xiaozhi Wang, Yunjia Qi, Hao Peng | This paper proposes a new reward modeling approach, “agentic reward modeling,” that combines human preferences with verifiable correctness signals for more reliable reward systems in large language models (LLMs). The main research objective is to develop a reward system that mitigates the limitations of existing reward models, which primarily focus on subjective human preferences and often neglect verifiable correctness. The key methodology involves implementing a reward agent, REWARDAGENT, that integrates human preference rewards with two verifiable signals: factuality (assessed via pairwise comparison and evidence verification) and instruction-following (verified through constraint parsing and Python code execution). The primary results show that REWARDAGENT significantly outperforms existing reward models on benchmarks like RM-Bench, JudgeBench, and a newly constructed IFBench, achieving an overall score of 72.5% in one configuration. The principal implication for AI practitioners is that integrating verifiable correctness signals with human preference feedback can lead to more reliable and robust reward models, improving LLM performance in downstream tasks and alignment with intended behavior, particularly during the inference and training phases. |
| Language Models’ Factuality Depends on the Language of Inquiry (Read more on arXiv or HuggingFace) | Hamid Palangi, Kumar Ayush, Kumar Tanmay, ayush1801, AggarwalTushar | Language models (LMs) exhibit inconsistent factual recall across different languages, failing to transfer knowledge even when possessing it in one language. The main research question is whether multilingual LMs truly internalize and transfer factual knowledge across languages or encode isolated linguistic silos. The key methodology involves creating a benchmark of 10,000 country-related facts across 13 languages and proposing metrics (Factual Recall Score, Knowledge Transferability Score, Cross-Lingual Factual Knowledge Transferability Score) to quantify factual recall and knowledge transferability. A primary result is that Llama-3-70B achieved the highest X-FaKT score of 0.848, demonstrating superior balanced performance in both factual recall and knowledge transfer. The principal implication is that AI practitioners must recognize language-specific factual reliability in multilingual LMs and leverage the most trustworthy information across languages, moving beyond the assumption of consistent cross-lingual knowledge access. |
| Can Language Models Falsify? Evaluating Algorithmic Reasoning with Counterexample Creation (Read more on arXiv or HuggingFace) | Matthias Bethge, Jonas Geiping, Ponnurangam Kumaraguru, Shashwat Goel, Shiven Sinha | Language models (LMs) are evaluated on their ability to generate counterexamples that falsify incorrect algorithmic solutions, introducing a new benchmark called REFUTE. The main research question is: Can LMs create counterexamples for incorrect solutions to algorithmic problems? The key methodology involves sourcing incorrect submissions from programming competitions, filtering them for non-trivial errors, and prompting LMs to generate inputs that cause these solutions to fail, validated through code execution. The primary result is that the best reasoning agents, including OpenAI 03-mini (high), can only create counterexamples for less than 9% of incorrect solutions in REFUTE, despite having a much higher success rate at solving those same problems. The principal implication for AI practitioners is that verification, including falsification of subtly incorrect solutions, is significantly harder for current LMs than generating correct solutions, highlighting a limitation in capabilities relevant for self-improvement and reliable reasoning. |
| Towards an AI co-scientist (Read more on arXiv or HuggingFace) | Anil Palepu, Tao Tu, Alexander Daryin, Wei-Hung Weng, Juraj Gottweis | Here’s a summary of the paper, strictly adhering to your guidelines: The paper introduces an AI co-scientist, a multi-agent system built on Gemini 2.0, designed to assist in scientific discovery by generating and evaluating novel research hypotheses. The main research objective is to develop an AI system capable of formulating demonstrably novel research hypotheses and proposals, building upon existing evidence and aligned with scientist-provided goals. The key methodology involves a multi-agent architecture with an asynchronous task execution framework, utilizing a generate, debate, and evolve approach with specialized agents for hypothesis generation, refinement, and ranking via simulated scientific debates and tournaments. The system demonstrates, across 203 diverse research goals, improved hypothesis quality (measured by an internal Elo rating system) as a function of increased test-time compute, and hypotheses for acute myeloid leukemia were validated to show tumor inhibition in vitro at clinically applicable concentrations. AI practitioners can leverage the multi-agent architecture and test-time compute scaling paradigm presented to build systems capable of complex reasoning and iterative improvement, although specific external validation metrics remain limited within the paper. |
| VEM: Environment-Free Exploration for Training GUI Agent with Value Environment Model (Read more on arXiv or HuggingFace) | Lingrui Mei, Lu Wang, Jiani Zheng, vyokky, keanudicap | VEM decouples value estimation from policy optimization for training GUI agents, enabling environment-free reinforcement learning. The main research objective is to develop an environment-free RL framework that can effectively train GUI agents without costly real-world interactions. The key methodology involves pretraining a Value Environment Model (VEM) to predict state-action values from offline data and then using this frozen VEM to guide policy exploration. The method achieves 28.0% offline task success rate on the General domain of the Android-in-the-Wild benchmark, surpassing environment-free baselines by 12-28%. AI practitioners can leverage this approach to train GUI agents with greater sample efficiency and stability, bypassing the need for direct environment interactions. |
| Plutus: Benchmarking Large Language Models in Low-Resource Greek Finance (Read more on arXiv or HuggingFace) | Polydoros Giannouris, Efstathia Soufleri, Triantafillos Papadopoulos, Xueqing Peng, jiminHuang | The paper introduces Plutus-ben, a Greek financial benchmark, and Plutus-8B, a Greek financial LLM, to address the lack of resources for Greek financial NLP. The main research question is: How do current language models perform on core Greek financial tasks, and how can fine-tuning on Greek financial data enhance performance? Key methodology involved creating Plutus-ben, comprising five financial NLP tasks (numeric and textual NER, QA, abstractive summarization, topic classification), and fine-tuning Llama-Krikri-8B with Greek domain-specific data to create Plutus-8B, evaluating 22 LLMs. The primary result is that Plutus-8B achieved the best performance on Plutus-ben, surpassing GPT-4 by 15.38% and outperforming all baseline models in the evaluation. Principal implication for AI practitioners is that fine-tuning on language-specific and domain-specific data is crucial for LLM performance in low-resource languages like Greek, significantly improving performance in tasks like financial numeric reasoning. |
| Distill Any Depth: Distillation Creates a Stronger Monocular Depth Estimator (Read more on arXiv or HuggingFace) | Ying Cui, Ruibo Li, Hongji Li, Dongyan Guo, Xiankang He | This paper introduces a new distillation framework for improving monocular depth estimation (MDE) using unlabeled data. The main research objective is to enhance zero-shot MDE by addressing the limitations of existing depth normalization strategies in pseudo-label distillation. The key methodology involves Cross-Context Distillation, integrating global and local depth cues, and a multi-teacher distillation framework using diverse depth estimation models. The primary result shows that the proposed method outperforms state-of-the-art methods on benchmark datasets; for instance, on the DIODE dataset, the AbsRel improves by 14.1% using the Local-Global and Shared-Context Distillation strategies. For AI practitioners, this method provides an effective way to train more robust and accurate MDE models by leveraging unlabeled data and combining the strengths of multiple teacher models, especially improving generalization in varied scenarios. |
| Project Alexandria: Towards Freeing Scientific Knowledge from Copyright Burdens via LLMs (Read more on arXiv or HuggingFace) | Andreas Hochlehnert, Tawsif Ahmed, Ameya Prabhu, Gollam Rabby, Christoph Schuhmann | This paper proposes converting copyrighted scientific texts into structured “Knowledge Units” using LLMs to make factual information freely accessible while respecting copyright. The main research question is whether converting scientific texts into Knowledge Units preserves factual information and adheres to copyright laws. The key methodology involves using LLMs to extract entities, attributes, and relationships from paragraphs of scientific papers into structured data, and evaluating the legal defensibility and information retention via question-answering experiments. Primary results show that language models answering multiple-choice questions using Knowledge Units achieved nearly the same accuracy (within 3-5% variance) as when using original texts across several scientific domains. AI practitioners can utilize this framework to build and use datasets containing facts from copyrighted scientific text, potentially democratizing access to scholarly knowledge without infringing on the original expression. |
| AISafetyLab: A Comprehensive Framework for AI Safety Evaluation and Improvement (Read more on arXiv or HuggingFace) | Xijie Huang, Junxiao Yang, Leqi Lei, Zhexin Zhang, LLLeo612 | AISafetyLab is a unified framework and toolkit for AI safety that integrates attack, defense, and evaluation methodologies. The main objective is to provide a standardized platform to evaluate and improve AI safety by addressing the lack of comprehensive tools and inconsistent experimental setups. The methodology involves implementing 13 attack methods (including black-box, gray-box, and white-box), 16 defense mechanisms (both inference-time and training-time), and 7 evaluation scorers, alongside auxiliary modules for model interaction, data management, utilities, and logging. In evaluations using Vicuna-7B-v1.5, AutoDAN achieved an average attack success rate of 56.4% across various defenses, while some other methods had varying performance depending on the defense used. For AI practitioners, AISafetyLab provides a flexible, extensible platform with comprehensive method coverage for systematically assessing and enhancing the robustness of AI models against adversarial attacks. |
| BIG-Bench Extra Hard (Read more on arXiv or HuggingFace) | Chrysovalantis Anastasiou, John Palowitch, Hritik Bansal, Mehran Kazemi, baharefatemi | BIG-Bench Extra Hard (BBEH) is a new benchmark to evaluate the general reasoning capabilities of large language models (LLMs). The main research objective is to address the saturation of existing LLM reasoning benchmarks, particularly BIG-Bench Hard (BBH), by creating a more challenging and diverse set of tasks. The methodology involves replacing each of the 23 tasks in BBH with a novel, more difficult task that probes similar reasoning capabilities, using a semi-adversarial approach with two reference models to ensure sufficient difficulty. The primary result is that the best general-purpose model achieved a harmonic mean accuracy of 9.8% on BBEH, while the best reasoning-specialized model achieved 44.8%, indicating significant room for improvement. AI practitioners should use BBEH to evaluate LLMs for robust general reasoning, revealing current limitations and driving improvements instead of using other benchmarks where LLMs have reached ceiling performance. |
| CritiQ: Mining Data Quality Criteria from Human Preferences (Read more on arXiv or HuggingFace) | Zhiheng Xi, Tianyi Liang, Qipeng Guo, Kai Lv, KYLN24 | CritiQ is a novel data selection method that automatically mines data quality criteria from human preferences and performs efficient data selection. The main research objective is to develop a method for automatically extracting data quality criteria from human preferences with minimal human annotation effort. The key methodology, CritiQ Flow, employs a manager agent to evolve quality criteria and worker agents to make pairwise judgments based on a knowledge base and a reflection process. Accuracies on human-annotated test sets reach 89.33% for code, 84.57% for math, and 88.06% for logic, outperforming baselines such as TextGrad and single-criterion methods. AI practitioners can use CritiQ to automatically derive data quality criteria and select high-quality subsets, improving model performance on downstream tasks with reduced reliance on manually designed heuristics or extensive human annotation. |
| MolSpectra: Pre-training 3D Molecular Representation with Multi-modal Energy Spectra (Read more on arXiv or HuggingFace) | Qiang Liu, Deli Zhao, Yu Rong, Shaozhen Liu, AzureLeon1 | MolSpectra enhances pre-training of 3D molecular representations by incorporating multi-modal energy spectra. The main research objective is to establish the relationship between 3D molecular structures and energy states using spectral data to improve molecular representation learning. The key methodology involves a multi-spectrum encoder, SpecFormer, trained with masked patch reconstruction, and a contrastive objective aligning 3D and spectral representations. Pre-training with MolSpectra achieved state-of-the-art performance on the QM9 dataset, achieving a mean absolute error (MAE) of 0.011 D on the dipole moment (μ) prediction, outperforming the baseline Coord method in 10 out of 12 properties. For AI practitioners, MolSpectra provides a pre-training framework that leverages molecular spectra to learn more informative 3D molecular representations, enhancing performance on downstream tasks like property prediction. |
| PosterSum: A Multimodal Benchmark for Scientific Poster Summarization (Read more on arXiv or HuggingFace) | Frank Keller, Pasquale Minervini, rohitsaxena | POSTERSUM, a new benchmark, evaluates multimodal models on summarizing scientific posters into research paper abstracts, revealing limitations in current models and introducing a hierarchical approach for improvement. Main research question or objective: How effectively can Multimodal Large Language Models (MLLMs) understand and summarize the complex, visually-rich content of scientific posters into concise textual abstracts, and can a hierarchical approach improve this performance? Key methodology used: The authors created a new dataset, POSTERSUM, consisting of 16,305 scientific posters paired with their corresponding abstracts. They benchmarked state-of-the-art MLLMs (including GPT-4o, Claude-3.5 Sonnet, Gemini 2.0, and various open-source models) on this dataset using metrics like ROUGE, SacreBLEU, METEOR, and BERTScore. They then proposed “SEGMENT & SUMMARIZE,” a hierarchical approach involving segmentation of the poster into coherent regions, localized summarization of each region, and global summarization to combine the localized summaries. Primary results: State-of-the-art MLLMs struggle to accurately summarize scientific posters. The best-performing closed-source model, GPT-4o, achieved a ROUGE-L score of only 22.30. The proposed SEGMENT & SUMMARIZE method significantly outperformed all other models, including closed-source MLLMs, achieving a ROUGE-L score of 24.18. Principal implication for AI practitioners: Current MLLMs, while strong on various tasks, have significant limitations when handling the complex multimodal information presented in scientific posters. The POSTERSUM dataset provides a valuable benchmark for advancing multimodal understanding, and the “SEGMENT & SUMMARIZE” approach demonstrates a promising direction for improving performance by incorporating a divide-and-conquer strategy, handling the complexity inherent in poster summarization. AI/ML/Software Engineers and Data Scientist working with scientific documents should prioritize models and architectures that are capable of understanding a variety of modalities and their combinations. |
Papers for 2025-02-26
| Title | Authors | Summary |
|---|---|---|
| OmniAlign-V: Towards Enhanced Alignment of MLLMs with Human Preference (Read more on arXiv or HuggingFace) | Jiaqiwang, Weiyun1025, UniverseCA, ChrisDing1105, PhoenixZ | OmniAlign-V introduces a new dataset and benchmark to improve the alignment of multi-modal large language models (MLLMs) with human preferences. The main research objective is to address the gap in human preference alignment observed in existing open-source MLLMs, despite their strong performance on foundational capability benchmarks. The key methodology involves constructing OmniAlign-V, a dataset of ~200K high-quality training samples with diverse images and complex question-answer pairs, and MM-AlignBench, a human-annotated benchmark for evaluating MLLM alignment. Finetuning MLLMs with OmniAlign-V via Supervised Fine-Tuning (SFT) or Direct Preference Optimization (DPO) improved the win rate against Qwen2VL-72B on MM-AlignBench, achieving a 72.6 win rate. The principal implication is that AI practitioners should utilize curated, human-aligned multi-modal datasets like OmniAlign-V during SFT and DPO to significantly enhance the human preference alignment of MLLMs while maintaining or enhancing fundamental capabilities. |
| SpargeAttn: Accurate Sparse Attention Accelerating Any Model Inference (Read more on arXiv or HuggingFace) | Haofeng Huang, surfingtomchen, hxi0408, Xiang-cd, jt-zhang | SpargeAttn is a universal sparse and quantized attention mechanism designed to accelerate inference in various AI models. The paper’s main objective is to design a training-free sparse attention operator that accelerates all models without metric loss. The key methodology involves a two-stage online filter that predicts sparse blocks in the attention map using selective token compression and a sparse warp online softmax, integrated with 8-bit quantization. SpargeAttn achieved a 1.83x speedup on Mochi on an L40 GPU without loss of video quality and is 2.5x to 5x faster than existing dense/sparse attention models. AI practitioners can use SpargeAttn to significantly accelerate the inference of diverse models, including language, image, and video generation, without sacrificing end-to-end performance metrics. |
| KV-Edit: Training-Free Image Editing for Precise Background Preservation (Read more on arXiv or HuggingFace) | Yansong Tang, jewelshaw, shiyi0408, xilluill | KV-Edit is a training-free image editing method that achieves precise background preservation by utilizing KV cache in diffusion models. The main research objective is to address the challenge of maintaining background consistency during image editing tasks while generating content aligned with modified text prompts. The key methodology involves caching and reusing key-value pairs of background tokens in Diffusion Transformers (DiTs) during the inversion and denoising processes, and optional mask-guided inversion and reinitialization strategies. Primary results show that KV-Edit achieves a PSNR of 35.87 in masked region preservation, outperforming existing methods. For AI practitioners, this method provides a way to perform image editing with perfect background preservation, without additional training or complex mechanisms, thereby facilitating more practical AI image editing applications. |
| ART: Anonymous Region Transformer for Variable Multi-Layer Transparent Image Generation (Read more on arXiv or HuggingFace) | JianminBao, DongChen06, 131131yhx, 2JZ, yifanpu001 | This paper introduces the Anonymous Region Transformer (ART) for generating variable multi-layer transparent images from a global text prompt and an anonymous region layout. The main research objective is to develop a method for generating high-quality, multi-layer transparent images that overcomes the limitations of existing methods requiring detailed semantic layouts. The key methodology involves using an anonymous region layout, a layer-wise region crop mechanism, and a multi-layer transparent image autoencoder. The method achieves a speed improvement of over 12 times compared to the full attention approach, and user studies show it outperforms existing methods (LayerDiffuse and COLE) in multiple aspects. The principal implication is that AI practitioners can generate multi-layer images more efficiently and with greater scalability, allowing for more precise control in interactive content creation and editing of individual elements within generative models. |
| SWE-RL: Advancing LLM Reasoning via Reinforcement Learning on Open Software Evolution (Read more on arXiv or HuggingFace) | RishabhSingh021, gsynnaeve, lingming, JadeCopet, yuxiang630 | SWE-RL is a reinforcement learning approach that enhances LLM reasoning for software engineering tasks using open-source software evolution data. The main research objective is to improve LLMs’ performance on real-world software engineering tasks, specifically issue resolution, using reinforcement learning. The key methodology is training LLMs on GitHub pull request data with a rule-based reward function based on the similarity between predicted and oracle code patches, optimized via Group Relative Policy Optimization (GRPO). The primary result is that Llama3-SWE-RL-70B achieves a 41.0% solve rate on the SWE-bench Verified dataset. The principal implication for AI practitioners is that reinforcement learning on software evolution data can significantly enhance LLM reasoning capabilities for software engineering and also improve performance on out-of-domain tasks. |
| Unveiling Downstream Performance Scaling of LLMs: A Clustering-Based Perspective (Read more on arXiv or HuggingFace) | Chenggang Li, Xiao Li, shenke18, Lucky2022, JerryXu98 | The paper introduces a Clustering-On-Difficulty (COD) framework to predict downstream task performance of Large Language Models (LLMs). The main research objective is to accurately predict LLM performance on downstream tasks prior to extensive model training, addressing the challenges of emergent abilities and uneven task difficulty distributions. The key methodology involves clustering tasks based on difficulty features, fitting performance-compute curves on predictable clusters, and mapping these predictions to the full evaluation set. The primary result is that COD achieves a mean absolute prediction error of 1.36% across eight LLM evaluation benchmarks on a 70B-parameter model. The principal implication is that AI practitioners can use COD for efficient resource allocation and monitoring during LLM training, by reliably predicting downstream task performance using smaller models. |
| Scale-Distribution Decoupling: Enabling Stable and Effective Training of Large Language Models (Read more on arXiv or HuggingFace) | Ya Wang, LLIXQ, xunzhou, Taoer, BryceZhuo | Scale-Distribution Decoupling (SDD) is a novel approach that stabilizes and improves the training of large language models by separating the scale and distribution of weight matrices. The main research objective is to address training instability issues, such as gradient explosion and vanishing gradients, in large language models (LLMs), particularly in Post-Norm Transformer architectures. SDD uses a normalization mechanism to regulate activations and a learnable scaling vector to maintain well-conditioned gradients in fully-connected layers. SDD-1B achieves a training loss of 2.65, outperforming OLMo2-1B (2.70), PostNorm-1B (2.69), and DeepNorm-1B (2.72), also achieving the highest average accuracy of 54.04% across multiple downstream tasks. For AI practitioners, SDD provides a lightweight and compatible solution for stabilizing LLM training, improving convergence, and enabling more efficient large-scale pre-training. |
| K-LoRA: Unlocking Training-Free Fusion of Any Subject and Style LoRAs (Read more on arXiv or HuggingFace) | Qibin Hou, Zhen Li, oyzh2005 | K-LoRA is a training-free method for merging subject and style LoRAs to generate images that preserve both characteristics. The paper’s objective is to develop a method for effectively combining content and style LoRAs without requiring additional training or manual parameter tuning. The key methodology is a Top-K selection process within attention layers that identifies and selects the most representative features from each LoRA for fusion, combined with a scaling factor that prioritizes content or style at different diffusion timesteps. The method achieved a CLIP score of 69.4% and a DINO score of 46.9% for subject similarity, outperforming existing methods. AI practitioners can use K-LoRA to effectively fuse separately trained subject and style LoRAs, enabling efficient customized image generation without retraining, simplifying the process of generating images with specific content and styles. |
| WebGames: Challenging General-Purpose Web-Browsing AI Agents (Read more on arXiv or HuggingFace) | Fraser, semitable, BiggieW, XanderJC, georgethomas | WebGames introduces a benchmark suite for evaluating general-purpose web-browsing AI agents. The primary objective is to assess AI limitations in web interactions using 50+ interactive challenges designed to be human-intuitive yet AI-challenging. The methodology involves evaluating vision-language models like GPT-4o, Claude Computer-Use, Gemini-1.5-Pro, and Qwen2-VL in a hermetic, client-side environment, measuring their success against human baselines. The best AI system achieved a 41.2% success rate compared to 95.7% human performance, revealing a substantial capability gap. This highlights the need for improvements in AI’s ability to handle common web interaction patterns, thereby directing future development efforts for web-browsing agents by AI practitioners. |
| Introducing Visual Perception Token into Multimodal Large Language Model (Read more on arXiv or HuggingFace) | wxcTest, horseee, rp-yu | This paper introduces Visual Perception Tokens to enhance Multimodal Large Language Models’ (MLLMs) control over visual perception processes. The main research objective is to enable MLLMs to autonomously control their visual perception, such as selecting specific image regions or refining features. The key methodology involves designing two types of Visual Perception Tokens (Region Selection and Vision Re-Encoding) that MLLMs generate and use to trigger additional visual processing steps. Results show that adding Visual Perception Tokens to a 2B parameter model improves its average performance across various VQA tasks by 30.9%, achieving a score of 0.749 compared to 0.572 without the tokens. AI practitioners can utilize these tokens to improve MLLMs’ performance in tasks requiring fine-grained visual understanding and spatial reasoning, by giving models a mechanism to actively control their visual input. |
| The Lottery LLM Hypothesis, Rethinking What Abilities Should LLM Compression Preserve? (Read more on arXiv or HuggingFace) | Peijie Dong, Qian Wang, Xiang Liu, wenxinsiju, coolzhtang | This paper proposes a “lottery LLM hypothesis” suggesting that smaller, compressed large language models (LLMs) can achieve comparable performance to original LLMs using external tools and reasoning. The main research objective is to identify the essential capabilities that compressed LLMs and key-value (KV) cache compression methods should preserve to maintain performance. The methodology involves a review of recent LLM advancements (retrieval-augmented generation, external tools, multi-step reasoning, computational expressivity) and proposes a recursive multi-step reasoning algorithm (Algorithm 1) for the “lottery LLM”. Primary results include showing that retrieval augmented generation can provide a compressed model equivalent performance. For instance Table 2 shows that Llama-3-Ins8B with RAG achieves a 59.8 accuracy score in the PopQA. The principal implication for AI practitioners is to focus on preserving specific abilities, like retrieval from prompts and long-context reasoning when developing LLM compression techniques, rather than solely focusing on perplexity or basic task accuracy. |
| AAD-LLM: Neural Attention-Driven Auditory Scene Understanding (Read more on arXiv or HuggingFace) | Ashesh Mehta, Stephan Bickel, vchoudhari, susameddin, xi-j | i) AAD-LLM is a brain-computer interface that integrates neural signals with an auditory large language model to improve auditory scene understanding aligned with listener attention. ii) The main research objective is to develop a system that can process and respond to auditory scenes based on a listener’s attentional focus, rather than treating all sound inputs equally. iii) The key methodology involves decoding a listener’s attended speaker from intracranial electroencephalography (iEEG) recordings and integrating this information into an auditory LLM to generate responses aligned with the listener’s perception. iv) AAD-LLM achieved a word error rate (WER) of 10.6% on transcribing the attended speech in a two-speaker scenario with background noise, significantly outperforming baseline models. v) AI practitioners can leverage this work to develop more human-centered auditory AI systems that prioritize listener intent, enhancing applications such as assistive hearing devices and human-computer interaction. |
| Shakti-VLMs: Scalable Vision-Language Models for Enterprise AI (Read more on arXiv or HuggingFace) | KartikAngadi, kruthika, SyedAbdul | Shakti-VLM, a family of 1B and 4B parameter vision-language models, achieves competitive multimodal performance with enhanced data efficiency through architectural innovations and a three-stage training strategy. The primary objective was to develop efficient vision-language models (VLMs) that achieve strong performance with reduced training data requirements. The methodology includes QK-Normalization, hybrid normalization, enhanced positional encoding, and a three-stage training process (text-only pretraining, vision-language alignment, and full model fine-tuning). Shakti-VLM-4B achieved 59.78% on the MMMU validation set, surpassing comparable models like Qwen2VL-7B and MiniCPM-V-2.6-8B. AI practitioners can leverage Shakti-VLM’s design and training strategies to build high-performing multimodal models with significantly less computational resources and training data, especially in enterprise-scale deployments. |
Papers for 2025-02-25
| Title | Authors | Summary |
|---|---|---|
| DICEPTION: A Generalist Diffusion Model for Visual Perceptual Tasks (Read more on arXiv or HuggingFace) | Zhiyue Zhao, Mingyu Liu, Z-MU-Z, zhyya, Canyu | DICEPTION is a generalist diffusion model for various visual perception tasks like segmentation, depth, and normal estimation. The primary objective is to create a single diffusion-based model capable of performing multiple visual perception tasks efficiently, leveraging pre-trained text-to-image models. The methodology involves unifying various perception tasks as conditional image generation in RGB space, using point prompts, task prompts, and a DiT architecture. Results demonstrate performance on par with state-of-the-art models, achieving comparable results to SAM-vit-h using only 0.06% of its training data (600K vs. 1B pixel-level annotated images). AI practitioners can leverage the priors of pre-trained diffusion models to create efficient and effective multi-task visual generalist models, significantly reducing the data and computational requirements compared to conventional training from scratch. |
| Thus Spake Long-Context Large Language Model (Read more on arXiv or HuggingFace) | Yuerong Song, Zhigeng Liu, Mianqiu Huang, Ruixiao Li, LiuXR | i) This survey paper presents a comprehensive overview of the long-context large language model (LLM) lifecycle. ii) The paper aims to provide a global picture of long-context LLMs, covering architectures, infrastructure, training, and evaluation technologies. iii) The methodology involves analyzing existing literature and categorizing long-context LLM technologies into architecture, infrastructure, training, and evaluation perspectives. iv) The survey showcases a spectrum of long-context technologies and identifies 10 unanswered questions currently faced by long-context LLMs; the context length of open-source LLMs has grown from 2k to 2M tokens between April 2023 and February 2024. v) The principal implication is to offer AI researchers and practitioners a systematic introduction to the research landscape of long-context LLMs, highlighting key challenges and future research directions. |
| Slamming: Training a Speech Language Model on One GPU in a Day (Read more on arXiv or HuggingFace) | Yossi Adi, avishai-elmakies, gallilmaimon | The paper introduces Slam, a recipe for training speech language models (SLMs) on a single GPU within 24 hours. The main research objective is to determine if high-quality SLMs can be trained using a single GPU within 24 hours. The methodology involves empirical analysis of model initialization, architecture, synthetic training data, and preference optimization, systematically ablating each training pipeline component. A key result is that the Slam recipe, utilizing a Qwen2.5-0.5B model and synthetic data, achieves a Topic-StoryCloze score of 82.04 on a single A5000 GPU. The principal implication is that AI practitioners can train high-quality SLMs with significantly reduced computational resources, improving accessibility of SLM research and development. |
| Audio-FLAN: A Preliminary Release (Read more on arXiv or HuggingFace) | Shuai Fan, Zixuan Li, Jiahao Pan, Ziya Zhou, Liumeng Xue | Audio-FLAN is a large-scale instruction-tuning dataset for unified audio-language models covering 80 diverse tasks across speech, music, and sound domains. The main research objective is to create a comprehensive dataset to enable unified audio-language models to perform both understanding and generation tasks in a zero-shot manner. The key methodology involves collecting and standardizing nearly all publicly available academic audio datasets into a common instruction-based format, normalizing the heterogeneous datasets and varying instructions using LLaMA and GPT. The primary result is a dataset with approximately 80 tasks, and over 100 million instances, significantly surpassing prior efforts in both quantity and diversity. AI practitioners can use Audio-FLAN to train and evaluate unified audio-language models capable of performing a wide range of understanding and generation tasks, potentially leading to models with zero-shot generalization abilities across speech, music and other audios. |
| GCC: Generative Color Constancy via Diffusing a Color Checker (Read more on arXiv or HuggingFace) | Yu-Chee Tseng, Yi-Chen Lo, Chia-Che Chang, Cheng-De Fan, Chen-Wei Chang | GCC is a method for estimating scene illumination in images by inpainting a color checker using diffusion models. The main research objective is to develop a color constancy method that generalizes well across different camera sensors without requiring sensor-specific training. The key methodology involves fine-tuning a diffusion-based inpainting model to insert a color checker into an image, then using Laplacian decomposition to maintain checker structure and extract illumination color from the inpainted checker’s achromatic squares. In cross-dataset evaluations, GCC achieved a worst-25% error rate of 5.15° and 4.32° in bi-directional evaluations. AI practitioners can leverage this method to estimate the illumination with good accuracy, across a wide range of sensors without specific sensor training data. |
| CodeCriticBench: A Holistic Code Critique Benchmark for Large Language Models (Read more on arXiv or HuggingFace) | Yejie Wang, Wei Zhang, Jiaheng Liu, Marcus Dong, Alexander Zhang | CodeCriticBench is a benchmark for evaluating large language models’ (LLMs) ability to critique code, assessing both code generation and code question-answering tasks. The main research objective is to establish a comprehensive framework for evaluating LLMs’ code critique capabilities across different dimensions and difficulty levels. The methodology involves collecting code tasks from various sources, constructing basic and advanced critique evaluation protocols, and designing fine-grained evaluation checklists. Primary results show that, on advanced evaluations, DeepSeek-R1 achieves an MSE of 3.92 on code generation, while Claude3.5-Sonnet leads in code QA with an MSE of 1.02; generally models increased in Accuracy (ACC) as parameters increased. The principal implication is that AI practitioners can use CodeCriticBench to systematically assess and compare the code critique performance of different LLMs, driving improvements in coding assistance tools and automated code review systems. |
| Linguistic Generalizability of Test-Time Scaling in Mathematical Reasoning (Read more on arXiv or HuggingFace) | James Thorne, Jiwoo Hong, Guijin Son, Cartinoe5930 | The paper introduces MCLM, a multilingual math benchmark, and evaluates the linguistic generalizability of test-time scaling methods in mathematical reasoning. The main research question is whether test-time scaling confers cross-lingual benefits in mathematical reasoning similar to those observed with pre-training scaling. The authors test three test-time scaling methods (Outcome Reward Modeling, Process Reward Modeling, and Budget Forcing) on multilingual LLMs using a new benchmark, MCLM, featuring competition-level problems in 55 languages. A primary result is that using Qwen2.5-1.5B Math with Outcome Reward Modeling achieves a score of 35.8 on MCLM, while Budget Forcing on MR1-1.5B attains 35.2, showing that gains from test-time scaling do not consistently extend to multiple languages. The principal implication is that AI practitioners should be aware that test-time scaling methods may not generalize effectively to multilingual tasks, and improving multilingual robustness requires methods beyond simply increasing inference-time compute. |
| Make LoRA Great Again: Boosting LoRA with Adaptive Singular Values and Mixture-of-Experts Optimization Alignment (Read more on arXiv or HuggingFace) | Wei Wei, Xiaoye Qu, Sichen Liu, Zhenyi Lu, Facico | GOAT enhances LoRA fine-tuning for large language models by using adaptive singular value decomposition and Mixture-of-Experts optimization alignment. The primary research question is how to mitigate the performance gap between LoRA and full fine-tuning, particularly in Mixture-of-Experts (MoE) architectures. The key methodology involves initializing LoRA MoE experts with distinct SVD segments of pre-trained weights and aligning optimization with a theoretical scaling factor derived from full fine-tuning. Primary results show that GOAT achieves 99.07% of full fine-tuning performance on image classification and outperforms all LoRA variants. The principal implication for AI practitioners is that GOAT offers a more efficient and effective fine-tuning approach, closing the performance gap with full fine-tuning while maintaining scalability. |
| Multimodal Inconsistency Reasoning (MMIR): A New Benchmark for Multimodal Reasoning Models (Read more on arXiv or HuggingFace) | Yang Zhao, Shan Jiang, Hongquan Li, Yue Fan, Qianqi Yan | The paper introduces MMIR, a new benchmark for evaluating multimodal reasoning models’ ability to detect semantic inconsistencies in layout-rich visual-textual content. The main research objective is to assess how well Multimodal Large Language Models (MLLMs) can identify and reason about semantic mismatches in artifacts like webpages and slides. The key methodology involves creating 534 samples with synthetically injected errors across five reasoning-heavy categories and evaluating six state-of-the-art MLLMs. The primary result is that the proprietary model, o1, achieved the best performance with over 50% accuracy in detecting inconsistencies, significantly outperforming open-source models which scored below 25%. The paper’s principle implication, therefore, is that there is a crucial need for development in advancing multimodal reasoning in current MLLMs, particularly for handling inconsistencies, to make the models more reliable. |
| Mobile-Agent-V: Learning Mobile Device Operation Through Video-Guided Multi-Agent Collaboration (Read more on arXiv or HuggingFace) | Ji Zhang, Ming Yan, Xi Zhang, Junyang Wang, xhyandwyy | Mobile-Agent-V is a framework that leverages video guidance to enhance mobile device automation through multi-agent collaboration. The main research objective is to address the limitations of existing mobile automation frameworks by providing rich and cost-effective operational knowledge. The key methodology involves a sliding window video input mechanism, a video agent for adaptive frame selection, and a deep-reflection agent for refining decision outputs. Primary results show that Mobile-Agent-V achieves a 30% performance improvement over existing frameworks in tasks requiring operational knowledge. The principal implication for AI practitioners is that they can use video demonstrations to effectively inject operational knowledge into mobile agents, enabling more efficient and scalable automation. |
| RIFLEx: A Free Lunch for Length Extrapolation in Video Diffusion Transformers (Read more on arXiv or HuggingFace) | Chongxuan Li, Yixiao Chen, Guande He, Min Zhao, zhuhz22 | RIFLEX improves length extrapolation in video diffusion transformers by reducing a key intrinsic frequency in positional embeddings. The main research objective is to understand and mitigate the failure modes (temporal repetition and slow motion) of existing length extrapolation methods in video diffusion transformers. The key methodology is analyzing the role of frequency components in Rotational Position Embedding (RoPE) and reducing the “intrinsic frequency” component that governs repetition patterns. Primary results show that RIFLEX achieves 2x extrapolation on CogVideoX-5B in a training-free manner, with a NoRepeat Score of 54.2 and Dynamic Degree of 59.4. The principal implication is that AI practitioners can achieve high-quality length extrapolation in video generation without additional training or significant modifications to existing models by simply adjusting the intrinsic frequency in the positional encoding. |
| Benchmarking Temporal Reasoning and Alignment Across Chinese Dynasties (Read more on arXiv or HuggingFace) | Deyu Zhou, Yong Jiang, Pengfei LI, Jialong Wu, wzl0228 | The paper introduces CTM, a new benchmark for evaluating temporal reasoning in large language models (LLMs) within the context of Chinese dynastic chronology. The main objective is to assess LLMs’ ability to understand and align temporal relationships across various Chinese historical entities and events. The methodology involves constructing a dataset of 8,750 question-answer pairs and 60 Timeline Ito Game instances, focusing on contextualization, cross-entity relationships, and pairwise temporal alignment. Evaluation of various LLMs revealed that the Time Interval Calculation (TIC) task was the most challenging, and the best performing model (Deepseek-R1) achieved an accuracy of 64.02% on question answering,. This suggests that CTM can provide a culturally rich resource for enhancing temporal reasoning capabilities and structured knowledge integration in large language models. |
| Reflective Planning: Vision-Language Models for Multi-Stage Long-Horizon Robotic Manipulation (Read more on arXiv or HuggingFace) | Sergey Levine, Xiangyu Yue, Zhuoran Yang, csuhan, yunhaif | This paper introduces Reflective Planning, a framework that enhances vision-language models (VLMs) for multi-stage, long-horizon robotic manipulation tasks by incorporating a reflection mechanism. The main research question is how to improve VLMs’ physical reasoning and long-horizon planning capabilities for complex robotic manipulation. The key methodology involves using a diffusion-based dynamics model for visual look-ahead and an iterative reflection process, enabling the VLM to critique and refine its actions based on imagined future states. The proposed method, ReflectVLM, achieved an 85.4% success rate on a challenging set of manipulation tasks, significantly outperforming state-of-the-art commercial VLMs and Monte Carlo Tree Search. AI practitioners can leverage this framework to develop more robust and efficient robotic planning systems that require visual understanding and long-horizon reasoning, without extensive task-specific training. |
| Stable-SPAM: How to Train in 4-Bit More Stably than 16-Bit Adam (Read more on arXiv or HuggingFace) | Xiang Li, Gaojie Jin, Zhenyu Zhang, Haotian Hu, Tianjin Huang | Stable-SPAM, a new optimizer, enhances stability in 4-bit large language model (LLM) training. The main research objective is to evaluate and improve the stability of 4-bit LLM training using recently proposed optimizers. The key methodology involves introducing Stable-SPAM, which incorporates adaptive gradient normalization (AdaGN), adaptive spike-aware clipping (AdaClip), and inherits momentum reset from SPAM. Primary results show that a 4-bit LLaMA-1B model trained with Stable-SPAM outperforms a BF16 LLaMA-1B trained with Adam by up to 2 perplexity points. The principal implication is that AI practitioners can use Stable-SPAM to achieve more stable and efficient training of LLMs with 4-bit quantization, matching or exceeding 16-bit Adam performance with significantly reduced memory and computational costs. |
| Can Community Notes Replace Professional Fact-Checkers? (Read more on arXiv or HuggingFace) | Isabelle Augenstein, Desmond Elliott, gretawarren, Nadav | This research investigates the reliance of Twitter/X’s Community Notes on professional fact-checking for combating misinformation. The main research questions are to what extent community notes rely on the work of professional fact-checkers and what are the traits of posts and notes that reference fact-checking sources. The researchers annotated a corpus of Twitter/X community notes using language models and performed manual annotations, classifying cited sources and identifying attributes like topic and refutation strategies. A primary result is that at least 5% of all English community notes contain an external link to professional fact-checkers, rising to 7% for notes rated as ‘helpful’. This suggests that, to improve community-based moderation quality, AI practitioners could consider integrating and/or prioritize content from verified professional fact-checking organizations within community moderation systems. |
| Forecasting Open-Weight AI Model Growth on Hugging Face (Read more on arXiv or HuggingFace) | Jianxi Gao, Pin-Yu Chen, KBhandari11 | The paper adapts a scientific citation model to predict the adoption dynamics of open-weight AI models on Hugging Face. The main research question is, “Can we predict the trajectory of influence an open-weight model will have on the AI community?”. The key methodology adapts Wang et al.’s citation model, using immediacy, longevity, and relative fitness parameters to track the cumulative number of fine-tuned models. The results show that most models cluster around narrow bands of parameters but models like openai/whisper-large-v3 demonstrate a high relative fitness (λi) of 528070.6635. AI practitioners can use this framework to anticipate model prominence and understand the long-term impact of open-weight models, guiding strategic decisions and governance. |
| TAG: A Decentralized Framework for Multi-Agent Hierarchical Reinforcement Learning (Read more on arXiv or HuggingFace) | Balázs Kégl, Albert Thomas, Hamza Cherkaoui, Abdelhakim Benechehab, Giuseppe Paolo | TAG is a decentralized framework for constructing multi-agent hierarchical reinforcement learning systems of arbitrary depth. The main research objective is to develop a framework enabling scalable and adaptable multi-agent systems through hierarchical organization and decentralized control. The key methodology is the LevelEnv abstraction, which presents each hierarchy level as an environment to the agents above it, standardizing information flow and enabling bidirectional communication. The experiments on MPE-Spread and VMAS Balance environments show that depth-three agents (3PPO and 2MAPPO-PPO) match a hand-designed heuristic performance with 95% confidence interval. AI practitioners can use TAG to build scalable multi-agent systems that decompose complex tasks across multiple hierarchical levels, improving learning efficiency and coordination without centralized control. |
| VideoGrain: Modulating Space-Time Attention for Multi-grained Video Editing (Read more on arXiv or HuggingFace) | Yi Yang, Hehe Fan, Linchao Zhu, Xiangpeng Yang | VideoGrain introduces a zero-shot approach for multi-grained video editing by modulating space-time attention mechanisms in diffusion models. The main research question is: Can attention be modulated to ensure accurate distribution of each local edit’s attention weights in the intended regions for multi-grained video editing? The key methodology is Spatial-Temporal Layout-Guided Attention (ST-Layout Attn), which modulates both cross-attention (for text-to-region control) and self-attention (for feature separation) within a diffusion model. The method achieves an Edit-Accuracy of 88.4, a Temporal-Consistency of 85.0 and an Overall score of 83.0 on a dataset of 76 video-text pairs. AI practitioners can leverage this method to perform precise, multi-grained video editing (class-level, instance-level, and part-level) without requiring parameter tuning or additional training data. |
| Beyond Release: Access Considerations for Generative AI Systems (Read more on arXiv or HuggingFace) | Yacine Jernite, Ariel Herbert-Voss, Dan Hendrycks, Rishi Bommasani, irenesolaiman | Generative AI system access, beyond component release, determines stakeholder engagement and risk-benefit tradeoffs through resourcing, technical usability, and utility. The main research question is how accessibility of generative AI system components, beyond their mere availability, influences their use, potential risks, and benefits. The key methodology involves deconstructing access along three axes (resourcing, technical usability, and utility) and analyzing access variables for four high-performance language models (Llama 3.1 405B Instruct, DeepSeek v3, GPT-4, Claude 3.5 Sonnet). A primary result is that Llama 3.1 405B Instruct requires at least 8 NVIDIA H100 GPUs and 405 GB VRAM to run locally in 8-bit precision. Principal implication is that, for AI practitioners, release decisions must consider access variables for effective risk assessment and deployment. |
| X-Dancer: Expressive Music to Human Dance Video Generation (Read more on arXiv or HuggingFace) | Chenxu Zhang, You Xie, Guoxian Song, Hongyi Xu, Zeyuan Chen | X-Dancer is a transformer-diffusion framework for generating music-driven human dance videos from a single image. The main research objective is to create diverse, long-range, and lifelike human dance videos synchronized with music, starting from a single static image. The key methodology involves a transformer that generates 2D pose sequences, and a diffusion model that translates these poses into video frames. The X-Dancer achieves a FVD score of 507.06 and FID-VID of 61.94 on their in-house dataset, surpassing all baselines in visual synthesis quality, which is a direct result of the method. AI practitioners can leverage this framework as a scalable solution for high-quality and expressive human image animation, with direct application in video content creation and customizable choreography. |
| MONSTER: Monash Scalable Time Series Evaluation Repository (Read more on arXiv or HuggingFace) | Amish Mishra, Lynn Miller, Chang Wei Tan, Navid Mohammadi Foumani, angus924 | MONSTER introduces a new benchmark for time series classification using larger datasets to address limitations of current benchmarks. The main research objective is to create and evaluate a collection of large-scale time series datasets to improve benchmarking in time series classification. Key methodologies include compiling 29 univariate and multivariate datasets, processing them into a common format, and evaluating baseline methods (ConvTran, FCN, HInceptionTime, TempCNN, HYDRA, QUANT, and ET) using 5-fold cross-validation. Primary results show that QUANT achieved the lowest overall mean 0-1 loss (0.1880) across all datasets, closely followed by ConvTran (0.1954), although performance varied significantly across different data categories. Principal implication for AI practioners is that that the field has artificially disadvanted low-bias methods and MONSTER can improve development and application in time series classification by training models on larger datasets. |
| The snake in the Brownian sphere (Read more on arXiv or HuggingFace) | Grégory Miermont, Brett Kolesnik, Emmanuel Jacob, Omer Angel | The paper describes the inverse of the continuous Cori-Vauquelin-Schaeffer (CVS) bijection, mapping the Brownian sphere to the Brownian snake. The main research objective is to construct the Brownian snake as a measurable function of the Brownian sphere, thereby inverting the continuous CVS bijection. The key methodology involves using the geometric notion of a cut locus on the Brownian sphere, defining a metric on the closure of the cut locus, and leveraging the induced orientation to define a planar order. The primary result is that, given a Brownian sphere (X,d,µ) and two independent points drawn from µ, there exists a measurable function outputting an R-tree T and label function Z such that T has the law of the Continuum Random Tree (CRT), and applying the continuum CVS mapping to (T, Z) recovers (X, d, μ). The paper proves that the orientation of the Brownian Sphere has a Rademacher distribution (equal to ±1 with equal probability), independently of the random variables ψ(h). AI/ML/Software Engineers/Data Scientist, can measurably recover the Brownian Snake and its associated tree structure from a given a Brownian Sphere, which provides new mathematical tooling and foundational understanding for models related to random planar maps. |
| M3-AGIQA: Multimodal, Multi-Round, Multi-Aspect AI-Generated Image Quality Assessment (Read more on arXiv or HuggingFace) | Weiming Zhang, Wen Shen, Zhihua Wei, Kejiang Chen, Chuan Cui | M3-AGIQA is a framework for assessing AI-generated image quality using multimodal inputs, multi-round interactions, and considering multiple quality aspects. The main research objective is to develop a comprehensive method for evaluating AI-generated images (AGIs) that aligns with human perceptual judgments across quality, correspondence, and authenticity. The key methodology involves distilling multi-aspect image captioning capabilities from online Multimodal Large Language Models (MLLMs) into a local MLLM via LoRA fine-tuning, and employing an xLSTM feature extractor with a regression head to predict Mean Opinion Scores (MOSs). The method achieved a Spearman’s Rank-Order Correlation Coefficient (SRCC) of 0.9045 and a Pearson Linear Correlation Coefficient (PLCC) of 0.9317 on the quality aspect of the AGIQA-3k dataset. AI practitioners can utilize this framework to more accurately and comprehensively evaluate the quality of generated images, considering multiple factors that go beyond simple perceptual quality. |
Papers for 2025-02-24
| Title | Authors | Summary |
|---|---|---|
| SurveyX: Academic Survey Automation via Large Language Models (Read more on arXiv or HuggingFace) | UglyToilet, Ki-Seki, siminniu, fan2goa1, HaruTeru | SURVEYX is a system for automated academic survey generation using Large Language Models (LLMs), designed to improve content and citation quality. The main research objective is to address limitations in existing LLM-based survey generation systems, such as finite context windows, lack of in-depth content discussion, and absence of systematic evaluation frameworks. The key methodology involves a two-phase approach (Preparation and Generation) incorporating online reference retrieval, AttributeTree pre-processing, and a re-polishing process, leveraging Retrieval Augmented Generation (RAG). Experimental results showed SURVEYX achieved a 0.259 improvement in content quality and a 1.76 enhancement in citation quality, approaching human expert performance (average content quality scores: SURVEYX: 4.590, Human: 4.754). For AI practitioners, SURVEYX provides an efficient and organized system for generating high-quality academic surveys, enhancing the information density for LLMs and optimizing their context window usage, with potential applications in various fields. |
| MaskGWM: A Generalizable Driving World Model with Video Mask Reconstruction (Read more on arXiv or HuggingFace) | Rui Chen, Yuxin Guo, Jingcheng Ni, wzhgba, lyclyc52 | MaskGWM is a driving world model that combines diffusion-based generation with masked reconstruction for improved fidelity and generalization. The main research objective is to develop a more generalizable driving world model capable of long-horizon prediction and multi-view generation, surpassing existing models constrained by prediction duration and generalization. The key methodology involves a Diffusion Transformer (DiT) architecture trained with an extra mask construction task, diffusion-related mask tokens, and a row-wise cross-view module for spatial-temporal and multi-view modeling. Primary results show the model achieves a Frechet Video Distance (FVD) of 59.4 and Frechet Inception Distance (FID) of 4.0 on the nuScenes dataset without action information, outperforming the state-of-the-art. For AI practitioners, the proposed MaskGWM framework offers a more robust and scalable approach to building driving world models, enabling improved video prediction and generalization capabilities for autonomous driving applications. |
| Mol-LLaMA: Towards General Understanding of Molecules in Large Molecular Language Model (Read more on arXiv or HuggingFace) | Sung Ju Hwang, Wonbin Lee, DongkiKim | i) Mol-LLaMA, a large molecular language model, is proposed for enhanced general understanding of molecules. ii) The research aims to develop a molecular language model that grasps general molecular knowledge to function as a versatile molecular assistant. iii) The methodology includes multi-modal instruction tuning with a designed dataset encompassing structural, chemical, and biological features, along with a blending module integrating information from 2D and 3D molecular encoders. iv) Experiments show Mol-LLaMA provides more accurate, detailed, and helpful responses than baseline LLMs and molecular LLMs, as well as improved performance on molecular property prediction, achieving high accuracy while maintaining high fidelity and helpfulness scores on the PAMPA task. v) The model provides AI/ML practitioners with a new foundation for building general-purpose molecular assistants capable of explaining molecular features and rationales, enhancing molecular analysis. |
| LLM-Microscope: Uncovering the Hidden Role of Punctuation in Context Memory of Transformers (Read more on arXiv or HuggingFace) | Polina Druzhinina, Elizaveta Goncharova, Temurbek Rahmatullaev, Matvey Mikhalchuk, Anton Razzhigaev | i) This paper introduces methods to quantify and visualize how LLMs encode contextual information, focusing on the role of punctuation. ii) The main research question is how seemingly minor tokens impact the contextual memory of transformer-based LLMs. iii) The methodology involves measuring token-level nonlinearity, contextualization through prefix reconstruction, and intermediate layer analysis via a modified Logit Lens. iv) The results show that removing stopwords, articles, and commas consistently degrades performance on MMLU and BABILong-4k and identifies a correlation between linearity and contextualization. v) AI practitioners should note the counterintuitive finding that “filler” tokens carry significant contextual information affecting performance on tasks requiring knowledge and long-context reasoning. |
| PhotoDoodle: Learning Artistic Image Editing from Few-Shot Pairwise Data (Read more on arXiv or HuggingFace) | Xueyin Wang, Hailong Guo, Yuxuan Zhang, Yiren Song, Shijie Huang | PhotoDoodle is presented as a novel image editing framework for photo doodling using few-shot learning. The research objective is to enable artists to overlay decorative elements onto photographs while maintaining background consistency and artistic style, addressing challenges in seamless integration, background preservation, and efficient style capture from limited data. The methodology employs a two-stage training strategy, initially pre-training a general image editing model (OmniEditor) and subsequently fine-tuning it with EditLoRA using artist-curated before-and-after image pairs and introducing positional encoding reuse. Experiments using the proposed PhotoDoodle dataset demonstrated advanced performance in customized image editing achieving a CLIP score of 0.279 and GPT score of 63.207. The principal implication is that the framework provides a customizable image editing approach that can learn and transfer artistic styles from limited data, offering a potential solution for high-quality, consistent image manipulation in artistic creation. |
| VLM$^2$-Bench: A Closer Look at How Well VLMs Implicitly Link Explicit Matching Visual Cues (Read more on arXiv or HuggingFace) | Yi R., Paul Pu Liang, Renjie Pi, RainJamesY, Sterzhang | i) The paper introduces VLM$^2$-Bench, a new benchmark to evaluate vision-language models’ ability to visually link matching cues across multiple images or frames. ii) The research aims to assess whether VLMs can effectively associate visual cues to identify correspondences without external knowledge. iii) The methodology involves creating a dataset of over 3,000 test cases across nine subtasks categorized by general, object-centric, and person-centric cues, and then evaluating various VLMs. iv) Evaluations show a significant performance gap between even GPT-4o (60.36%) and human-level accuracy (95.16%), indicating challenges in visually linking cues. v) The benchmark and identified challenges imply the necessity for AI practitioners to develop VLMs with enhanced visual understanding and reasoning capabilities, focusing on reducing reliance on prior knowledge and improved cue association. Some parts of the paper lack clarity about the specific data creation process. |
| SIFT: Grounding LLM Reasoning in Contexts via Stickers (Read more on arXiv or HuggingFace) | Zhijie Deng, Boxiu Li, Xuyao Huang, Zihao Zeng | SIFT is a post-training approach that improves large language models’ (LLMs) reasoning by grounding it in the provided context using model-generated summaries called “Stickers.” The main research objective is to address the issue of “factual drift,” where LLMs misinterpret or overlook key information in the input query during reasoning. The key methodology is a post-training approach called “Stick to the Facts” (SIFT), which involves generating a “Sticker” summarizing key facts, performing consensus prediction using the Sticker and the original query, and refining the Sticker via forward and inverse optimization. A primary result is that SIFT improves the pass@1 accuracy of DeepSeek-R1 on AIME2024 from 78.33% to 85.67%. The principal implication is that AI practitioners can improve model accuracy, particularly on complex reasoning tasks, using sticker-based, factual grounding. |
| LightThinker: Thinking Step-by-Step Compression (Read more on arXiv or HuggingFace) | Mengshu Sun, Yuqi Zhu, Jintian Zhang, Ningyu, GoooDte | LightThinker is a method that enables LLMs to dynamically compress intermediate thoughts during reasoning to improve efficiency. The main research objective is to reduce the memory and computational costs of LLMs during complex reasoning tasks without sacrificing performance. The key methodology involves training the model to compress verbose thought steps into compact representations using gist tokens and specialized attention masks, quantified by a new “Dependency” metric. Primary results show that with the Qwen model, LightThinker reduces peak token usage by 70% and inference time by 26% compared to the Vanilla model, while maintaining comparable accuracy (with only a 1% drop). The principal implication for AI practitioners is that LightThinker offers a new approach for improving LLM inference efficiency in complex reasoning, providing a balance between accuracy and computational cost, though there is significant performance degradation on Llama series models. |
| StructFlowBench: A Structured Flow Benchmark for Multi-turn Instruction Following (Read more on arXiv or HuggingFace) | Yuan Wu, Yi Chang, Yue Wang, Jinzhe Li, Jinnan Li | The paper introduces StructFlowBench, a new benchmark for evaluating multi-turn instruction-following capabilities of large language models (LLMs). The main research objective is to assess LLMs’ ability to understand and maintain structural dependencies between dialogue turns, beyond simple constraint satisfaction. The key methodology involves defining a structural flow framework with six inter-turn relationship types and creating a dual-constraint evaluation system combining intra-turn and structural constraints. Evaluations of 13 LLMs revealed that the DeepSeek-v3 model achieved the highest Weighted Constraint Satisfaction Rate (WCSR) of 0.98. The principal implication for AI practitioners is the need to develop LLMs that better handle complex dialogue structures, particularly refinements, to improve performance in real-world multi-turn conversational applications. |
| KITAB-Bench: A Comprehensive Multi-Domain Benchmark for Arabic OCR and Document Understanding (Read more on arXiv or HuggingFace) | Ghazi Ahmed, Rania Hossam, Abdullah Sohail, mukul54, ahmedheakl | KITAB-Bench introduces a new benchmark for evaluating Arabic OCR and document understanding systems. The main research objective is to address the lack of comprehensive evaluation frameworks for Arabic OCR, which lags behind English OCR due to the script’s unique challenges. The key methodology involves curating a diverse dataset of 8,809 samples across 9 domains and 36 sub-domains, including handwritten text, tables, and charts, and evaluating various OCR systems and Vision-Language Models (VLMs) on tasks like text recognition, layout detection, and PDF-to-Markdown conversion. A primary result is that modern VLMs (e.g., GPT-4, Gemini) outperform traditional OCR approaches (e.g., EasyOCR, PaddleOCR) by an average of 60% in Character Error Rate (CER), but the best model (Gemini-2.0-Flash) achieves only 65% accuracy in PDF-to-Markdown conversion. AI practitioners can use KITAB-Bench to rigorously evaluate and improve Arabic document analysis methods, and focus efforts on bridging performance gap with English OCR, particularly in complex tasks like accurate structured content extraction from PDF documents. |
| InterFeedback: Unveiling Interactive Intelligence of Large Multimodal Models via Human Feedback (Read more on arXiv or HuggingFace) | Mike Zheng Shou, Haiyang Mei, Yifei Tao, Wenqi Pei, Henry Hengyuan Zhao | InterFeedback, a framework and benchmark, is introduced to evaluate the interactive intelligence of Large Multimodal Models (LMMs) using human feedback. The main research question is: “How do Large Multimodal Models perform with human feedback?” The key methodology involves an interactive framework, InterFeedback, using leading LMMs like GPT-4o to simulate human feedback and testing on datasets like MMMU-Pro and MathVerse. Results show that state-of-the-art LMMs (e.g., OpenAI-01) can correct their results through human feedback less than 50% of the time. The principal implication for AI practitioners is the need to develop methods that enhance LMMs’ capabilities to interpret and benefit from feedback, as current models demonstrate suboptimal performance in this area. |
| Evaluating Multimodal Generative AI with Korean Educational Standards (Read more on arXiv or HuggingFace) | Geewook Kim, sangheeeee | This paper introduces KoNET, a new benchmark for evaluating Multimodal Generative AI systems using Korean national educational tests. The main research objective is to assess the performance of Multimodal Generative AI systems across different educational levels in the Korean language. The methodology involves evaluating various open-source, open-access, and closed API models on four Korean educational exams (KoEGED, KoMGED, KoHGED, and KoCSAT) using a multimodal VQA format, and comparing their performance with human error rates. The primary results show that the EXAONE-3.0-7.8B-Instruct model achieved a KoNET score of 45.5, and model accuracy generally decreases with more advanced curricula; also closed-source APIs performed far superior to open-source models. The principal implication for AI practitioners is that benchmarks centered solely on English may not accurately assess AI performance in non-English language environments, highlighting a need for language-specific benchmarks and models. |
| Superintelligent Agents Pose Catastrophic Risks: Can Scientist AI Offer a Safer Path? (Read more on arXiv or HuggingFace) | Pietro Greiner, Joumana Ghosn, Damiano Fornasiere, Michael Cohen, Yoshua Bengio | This paper proposes “Scientist AI,” a non-agentic AI design, as a safer alternative to increasingly capable generalist agentic AI systems that pose catastrophic risks. The main research objective is to design a non-agentic AI that is trustworthy and safe by design, minimizing risks associated with uncontrolled agentic AI. The key methodology is a Bayesian approach with a world model generating causal theories and an inference machine for probabilistic question answering, operating with explicit uncertainty quantification. The paper presents the abstract view that as training data, objectives, and models scale for agentic AI, goal misgeneralization becomes more likely. This is contrasted with the proposal that the proposed non-agentic design improves in safety and accuracy with additional computing power. For AI practitioners, the principal implication is that focusing development on non-agentic AI, specifically “Scientist AI,” may enable benefits of AI innovation while avoiding risks associated with the current agent-driven trajectory. |
| The Relationship Between Reasoning and Performance in Large Language Models – o3 (mini) Thinks Harder, Not Longer (Read more on arXiv or HuggingFace) | Vincent Ginis, Andres Algaba, Marthe Ballon | The research investigates reasoning token usage versus accuracy in different generations of OpenAI language models. The main research question is whether more capable models within a single family require a longer chain-of-thought (more reasoning tokens) to achieve higher performance, or if they reason more effectively. The key methodology involves a systematic analysis of chain-of-thought length and accuracy across o1-mini and o3-mini variants on the Omni-MATH benchmark, using logistic regression to quantify effects. The primary results are that the o3-mini (m) achieves superior accuracy without requiring longer reasoning chains than o1-mini, and accuracy generally declines as reasoning chains grow, with a diminishing rate as proficiency goes up; Specifically, accuracy decreased by 3.16% per 1000 reasoning tokens for o1-mini and 1.96% for o3-mini (m). The principal implication is that, for mathematical reasoning tasks, constraining the chain-of-thought might be beneficial for weaker models; newer models exhibit more efficient reasoning, and less is more. |
| ReQFlow: Rectified Quaternion Flow for Efficient and High-Quality Protein Backbone Generation (Read more on arXiv or HuggingFace) | Hongteng Xu, EatEatEatEat, AngxiaoYue | ReQFlow is a novel method for fast and high-quality protein backbone generation using rectified quaternion flows. The main research objective is to develop a generative model that can efficiently produce designable protein backbones, overcoming limitations of existing diffusion and flow-based models. The key methodology involves representing 3D rotations with unit quaternions, constructing a quaternion flow (QFlow) via spherical linear interpolation (SLERP) in exponential format, and rectifying the QFlow to accelerate inference and improve designability. The primary results show that ReQFlow achieves state-of-the-art performance in protein backbone generation, requiring significantly fewer sampling steps and less inference time; for example, it is 37x faster than RFDiffusion when generating a backbone of length 300. Principal implication for AI practitioners is that ReQFlow provides a more efficient and effective approach to protein backbone generation, improving upon existing methods in both speed and the quality of generated structures. |
| MoBA: Mixture of Block Attention for Long-Context LLMs (Read more on arXiv or HuggingFace) | Tao Jiang, Yulun Du, Jingyuan Liu, Zhejun Jiang, Enzhe Lu | MoBA is a novel attention mechanism for LLMs that improves efficiency and scalability for long contexts by applying Mixture-of-Experts principles to block-wise attention. The main research objective is to design a robust attention architecture that can seamlessly transition between full and sparse attention without compromising performance and allowing the model to attend autonomously. The key methodology is partitioning the context into blocks and using a gating mechanism to route query tokens to the most relevant blocks, based on a computed affinity score. Primary results show that MoBA achieves comparable performance to full attention on language modeling tasks, with a validation loss difference within 1e-3, while achieving up to a 6.5x speedup when prefilling 1M tokens. For AI practitioners, MoBA offers a practical solution for enhancing long-context capabilities in LLMs with improved computational efficiency and seamless integration with existing pre-trained models. |
| One-step Diffusion Models with $f$-Divergence Distribution Matching (Read more on arXiv or HuggingFace) | Arash Vahdat, Weili Nie, Yilun Xu | The paper introduces f-distill, a framework for distilling diffusion models into one-step generators by minimizing f-divergences between teacher and student distributions. The main research objective is to generalize distribution matching distillation with f-divergences, enabling different trade-offs between mode coverage and training variance. The key methodology involves deriving the gradient of the f-divergence between teacher and student distributions and expressing it as a weighted score difference, using a weighting function determined by density ratio and the chosen f-divergence. Primary results show that f-distill, using Jensen-Shannon divergence, achieves a state-of-the-art one-step FID score of 1.16 on ImageNet-64. The principal implication for AI practitioners is that they can leverage f-distill to create efficient one-step image generators with improved sample quality and control over mode coverage, surpassing previous variational score distillation methods. |
| Think Inside the JSON: Reinforcement Strategy for Strict LLM Schema Adherence (Read more on arXiv or HuggingFace) | Viktoria Rojkova, Ishan Joshi, Bhavik Agarwal | The paper introduces “Think Inside the JSON,” a reinforcement learning framework for training LLMs to adhere strictly to predefined JSON schemas. The main research objective is to develop a method for enforcing strict schema adherence in LLM text generation, specifically for structured data output. The key methodology combines synthetic data generation, a novel reinforcement learning pipeline using Group Relative Policy Optimization (GRPO) with custom rewards, and supervised fine-tuning. This approach achieves a 62.41% mean match rate on a structured data extraction benchmark, with a 0.27% mean noise rate, outperforming distilled versions of DeepSeek R1 and Gemini 2.0 Flash. For AI practitioners, this provides a resource-efficient method to enforce schema constraints in LLM outputs, valuable for applications requiring high data integrity and compliance. |
| CrossOver: 3D Scene Cross-Modal Alignment (Read more on arXiv or HuggingFace) | Iro Armeni, Daniel Barath, Marc Pollefeys, Ondrej Miksik, sayandsarkar | CrossOver is a framework for 3D scene understanding that aligns modalities like images, point clouds, and CAD models via a modality-agnostic embedding space. The main research objective is to achieve flexible, scene-level cross-modal alignment in 3D environments without requiring complete data or rigid alignment across all modalities. The key methodology involves using dimensionality-specific encoders, a three-stage training pipeline (object-level, scene-level, unified encoders), and contrastive learning to create a unified embedding space. Results on ScanNet and 3RScan datasets show superior performance, achieving a scene-level matching recall of 99.31% (R@25) on ScanNet for the I → R modality. The principal implication is that AI practitioners can leverage CrossOver for robust 3D scene understanding and cross-modal retrieval tasks, even with incomplete or unaligned multi-modal data, removing the requirement of full data alignment. |
| Beyond No: Quantifying AI Over-Refusal and Emotional Attachment Boundaries (Read more on arXiv or HuggingFace) | Grant Rosario, David Noever | The paper introduces a benchmark and evaluation framework for assessing emotional boundary handling in Large Language Models (LLMs). The main research objective is to quantify and analyze “over-refusal” in LLMs when responding to user prompts that attempt to establish emotional connections or relationships. The key methodology involves a dataset of 1156 prompts across six languages, evaluating three LLMs (GPT-4o, Claude-3.5 Sonnet, and Mistral-large) using pattern-matched response analysis across seven key patterns. A primary result is that Claude-3.5 achieved the highest overall score (8.69/10), and a significant performance gap was found between English (average score 25.62) and non-English interactions (≤ 0.22). The principal implication for AI practitioners is the need to develop more nuanced, multilingual emotional intelligence and boundary-setting capabilities in LLMs, addressing over-refusal while maintaining ethical and safety standards. |
| JL1-CD: A New Benchmark for Remote Sensing Change Detection and a Robust Multi-Teacher Knowledge Distillation Framework (Read more on arXiv or HuggingFace) | Jingyu Ma, Yuanxiu Zhou, Long Gao, Ruifei Zhu, circleLZY | JL1-CD introduces a new dataset and a multi-teacher knowledge distillation framework for remote sensing change detection. The main research objective is to address the scarcity of high-resolution, all-inclusive change detection datasets and improve model performance across varying change area ratios. The key methodology involves constructing the JL1-CD dataset, proposing an Origin-Partition (O-P) training strategy, and developing a Multi-Teacher Knowledge Distillation (MTKD) framework. Results show that the MTKD framework, when applied to the Changer-MiT-b1 model, achieves an mIoU of 76.15% on the JL1-CD dataset. The principal implication for AI practitioners is that utilizing MTKD can enhance the performance of change detection models without increasing inference cost, particularly beneficial when the data has diverse range of change area ratio. |
| UPCORE: Utility-Preserving Coreset Selection for Balanced Unlearning (Read more on arXiv or HuggingFace) | Mohit Bansal, Elias Stengel-Eskin, vaidehi99 | UPCORE is a method-agnostic data selection framework that mitigates collateral damage in machine unlearning by pruning outliers from the forget set. The main research objective is to determine how measurable attributes of the forget set drive collateral effects during unlearning and whether these attributes can be controlled to optimize the deletion effectiveness/model utility trade-off. The key methodology involves using Isolation Forests to identify and prune high-variance outlier data points in the forget set’s hidden state representations, forming a lower-variance “core” forget set used for unlearning. Primary results show that UPCORE achieves a higher area-under-the-curve (AUC) score (0.387) compared to unlearning on the complete set (0.343) and random subset (0.353) using Gradient Ascent, across standard metrics, indicating improved balance between deletion and utility preservation. AI practitioners can use UPCORE to minimize negative side effects when removing data or capabilities from trained models, leading to more robust and reliable unlearning processes. |
Papers for 2025-02-21
| Title | Authors | Summary |
|---|---|---|
| SuperGPQA: Scaling LLM Evaluation across 285 Graduate Disciplines (Read more on arXiv or HuggingFace) | Liam-Liu, kangz, aaabiao, BingliW, mkj69 | SuperGPQA is a new benchmark for evaluating LLMs across 285 graduate-level disciplines, utilizing a human-LLM collaborative filtering mechanism. i) SuperGPQA is a new challenging benchmark for evaluating large language model knowledge and reasoning at the graduate level. ii) Main research question/objective: To assess the capabilities of LLMs across a wide range of specialized, graduate-level academic disciplines, exceeding the scope of existing benchmarks. iii) Key methodology: A human-LLM collaborative filtering system was employed, involving crowd-sourcing annotators, experts, and SOTA LLMs with iterative refinement of questions based on LLM responses and expert feedback, followed by a 3-stage quality inspection process. iv) Primary results: The reasoning-focused model DeepSeek-R1 achieved the highest accuracy of 61.82% on SuperGPQA, demonstrating significant room for improvement for current LLMs. v) Principal implication for AI practitioners: The benchmark reveals a substantial gap between current LLM capabilities and graduate-level human expertise, highlighting the need for developing models with enhanced reasoning and specialized domain knowledge to advance research towards Artificial General Intelligence. |
| MLGym: A New Framework and Benchmark for Advancing AI Research Agents (Read more on arXiv or HuggingFace) | Nikolay Bashlykov, Nicholas Roberts, Lovish Madaan, rraileanu, dnathani | MLGYM is a new Gym environment and benchmark, MLGYM-Bench, for evaluating and developing LLM agents on 13 diverse, open-ended AI research tasks. The main research objective is to create a standardized framework for evaluating LLM agents on their ability to perform realistic AI research tasks, enabling research on reinforcement learning algorithms. The key methodology is a Gym environment that integrates diverse AI research tasks, allowing agents to interact with a shell environment using tools, with performance evaluated via task-specific scripts. A primary result is that OpenAI’s O1-preview model achieved the highest Best Submission AUP@4 score of 1.176 across all tasks, followed by Gemini-1.5-Pro at 1.125. AI practitioners can utilize MLGYM to develop and assess AI research agents, driving progress in automating complex machine-learning research workflows, and apply different training algorithms for AI agents such as reinforcement learning. |
| SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features (Read more on arXiv or HuggingFace) | Xiao Wang, talfanevans, ibomohsin, AlexeyG, mitsch | SigLIP 2, a family of multilingual vision-language encoders, improves upon SigLIP with enhanced semantic understanding, localization, and dense features. The main research objective is to develop vision-language encoders that outperform existing models, including SigLIP, across various tasks while supporting multiple languages. The key methodology involves combining the original SigLIP training recipe with decoder-based pretraining, self-distillation, masked prediction, and online data curation, applied in a staged training approach. Primary results show that SigLIP 2 outperforms SigLIP and other open-weight baselines on ImageNet zero-shot classification; for example a SigLIP 2 B/16 model achieves 79.1% accuracy compared to SigLIP’s 76.7% at 256x256 resolution. AI practitioners can leverage SigLIP 2’s improved encoders for enhanced performance in vision-language tasks, particularly benefiting from multilingual capabilities, strong dense features, and backward compatibility with SigLIP. |
| S*: Test Time Scaling for Code Generation (Read more on arXiv or HuggingFace) | Shangyin Tan, Xiuyu Li, Chengkun Cao, Dacheng Li, eva98 | S* is a hybrid test-time scaling framework that improves code generation by combining parallel and sequential scaling with adaptive input synthesis for selection. The main research objective is to improve the coverage and selection accuracy of generated code by extending existing test-time scaling paradigms. The key methodology involves augmenting parallel sampling with sequential scaling via iterative debugging, and introducing a novel selection mechanism that adaptively generates distinguishing inputs for pairwise comparison of candidate solutions, grounded in execution results. Results show that S* consistently improves performance across 12 Large Language Models, with DeepSeek-R1-Distill-Qwen-32B achieving 85.7% on LiveCodeBench, approaching o1 (high) at 88.5%. The principal implication for AI practitioners is that combining parallel and sequential scaling with execution-grounded adaptive input synthesis during test-time significantly improves code generation performance, enabling smaller or instruction-based models to surpass larger or reasoning models. |
| How Much Knowledge Can You Pack into a LoRA Adapter without Harming LLM? (Read more on arXiv or HuggingFace) | Vasily Konovalov, Daniil Moskovskiy, Maria Marina, msalnikov, memyprokotow | This paper investigates how much new factual knowledge can be incorporated into a Large Language Model (LLM) using Low-Rank Adaptation (LoRA) without compromising pre-existing knowledge. The main research objective is to determine the extent to which new facts can be integrated into an LLM via a LoRA adapter while preserving general capabilities. The key methodology involves fine-tuning a Llama-3.1-8B-Instruct model using LoRA with varying amounts of new knowledge (DBpedia triples) and evaluating performance on external benchmarks (MMLU, TruthfulQA) and internal metrics (knowledge shifts). A primary result is that a model trained on 500 unknown facts, achieved 100% reliability on test, while models trained with additional highly-known data could see minimized negative shifts; Accuracy of models trained on MMLU with added 10 HighlyKnown or paraphrased sample show a significant drop in accuracy. The principal implication for AI practitioners is that while LoRA is effective for incorporating new knowledge, there is a trade-off between new knowledge integration, reduced truthfulness and general reasoning capabilities, requiring careful consideration of training data composition. |
| Does Time Have Its Place? Temporal Heads: Where Language Models Recall Time-specific Information (Read more on arXiv or HuggingFace) | Jaewoo Kang, Minbyul Jeong, Jungwoo Park, Chanwoong Yoon, Yein Park | Language models possess specialized attention heads, termed “Temporal Heads,” that are primarily responsible for processing time-specific factual knowledge. The research objective is to identify and analyze the mechanisms within large language models (LLMs) that handle temporally-changing facts. The methodology utilizes Circuit Analysis, specifically Temporal Knowledge Circuits and attention head ablation, to isolate and evaluate the contribution of specific attention heads. Ablating identified Temporal Heads reduced the model’s temporal knowledge accuracy in Llama2 by 3-9%, while its performance on time-invariant tasks remains unchanged. AI practitioners can leverage identified Temporal Heads to edit or control temporal aspects of LLM outputs, minimizing retraining. |
| LongWriter-V: Enabling Ultra-Long and High-Fidelity Generation in Vision-Language Models (Read more on arXiv or HuggingFace) | Jifan Yu, Yushi Bai, Daniel Zhang-Li, Yucheng Wang, Shangqing Tu | LongWriter-V enhances vision-language models (VLMs) for generating ultra-long, high-fidelity text from visual inputs. The main research objective is to address the limitation of existing VLMs in generating coherent outputs beyond 1,000 words, despite their ability to process long visual and textual contexts. Key methodology involved creating a new dataset, LongWriter-V-22k, with 22,158 examples of multi-image inputs and long text outputs (up to 10,000 words), and proposing IterDPO, a modified direct preference optimization method for long text. Primary results show that the 7B parameter model trained with LongWriter-V-22k and IterDPO outperformed larger proprietary models like GPT-4o on the MMLongBench-Write benchmark, achieving an overall score of 84.6, including component scores of 86.2 (length) and 82.9 (quality). Principal implication for AI practitioners is that using specialized datasets with long-output examples and iterative preference optimization can significantly improve the long-text generation capabilities of VLMs, enabling more effective real-world applications requiring detailed visual descriptions or reports. |
| Logic-RL: Unleashing LLM Reasoning with Rule-Based Reinforcement Learning (Read more on arXiv or HuggingFace) | Yuqian Hong, Haoming Luo, Qingnan Ren, Zitian Gao, Tian Xie | Logic-RL explores rule-based reinforcement learning (RL) to enhance reasoning in large language models (LLMs) using synthetic logic puzzles. The main research objective is to investigate if rule-based RL can improve LLM reasoning abilities and generalization to unseen tasks. The key methodology involves training a 7B parameter LLM with a modified REINFORCE++ algorithm, using a system prompt, a stringent format reward, and procedurally generated Knights and Knaves logic puzzles. The primary result is that after training on 5,000 logic problems, the model improved by 125% on the AIME math benchmark and 38% on the AMC, demonstrating cross-domain generalization. For AI practitioners, this demonstrates that RL, even with limited synthetic data, can significantly enhance an LLM’s abstract reasoning and generalization capabilities, offering a potentially more effective approach than supervised fine-tuning for specialized reasoning tasks. |
| PC-Agent: A Hierarchical Multi-Agent Collaboration Framework for Complex Task Automation on PC (Read more on arXiv or HuggingFace) | Junyang Wang, Yuyang Wanyan, Haiyang Xu, Xi Zhang, Haowei Liu | PC-Agent is a hierarchical multi-agent framework designed to automate complex tasks on PCs by improving perception and decision-making. The main research objective is to develop a system that can handle complex user instructions and interdependent sub-tasks in PC environments, overcoming limitations of existing methods in perception and workflow management. The key methodology is a hierarchical multi-agent collaboration architecture that decomposes decision-making into Instruction-Subtask-Action levels, with specialized agents (Manager, Progress, Decision, Reflection) and an Active Perception Module (APM). The primary result is that PC-Agent achieved a 56.0% task success rate on the PC-Eval benchmark, a 32% absolute improvement over previous state-of-the-art methods. Principal implication for AI practitioners is that the proposed framework significantly enhances the capability of agents to automate real-world, complex tasks on PCs. |
| S$^2$R: Teaching LLMs to Self-verify and Self-correct via Reinforcement Learning (Read more on arXiv or HuggingFace) | Jiaqi Chen, Xingyan Liu, Cheng Liu, Peisong Wang, Ruotian Ma | S$^2$R is a framework that enhances Large Language Model (LLM) reasoning by teaching models to self-verify and self-correct during inference via reinforcement learning. The main research objective is to develop an efficient framework that improves LLM reasoning abilities, particularly in mathematical problem-solving, without requiring large-scale data or extensive training. The key methodology involves initializing LLMs with self-verification and self-correction behaviors through supervised fine-tuning, then strengthening these skills using outcome-level and process-level reinforcement learning. Results demonstrate that a Qwen2.5-math-7B model, trained with only 3.1k initialization samples, achieved an accuracy improvement from 51.0% to 81.6% on the MATH500 test set. For AI practitioners, this implies that implementing self-verification and self-correction via reinforcement learning offers a resource-efficient approach to substantially improve the mathematical reasoning capabilities of LLMs, potentially using process-level RL for weaker base models and outcome-level RL for stronger ones. |
| Discovering highly efficient low-weight quantum error-correcting codes with reinforcement learning (Read more on arXiv or HuggingFace) | Zi-Wen Liu, basil2115 | This paper introduces a reinforcement learning (RL) based method for discovering highly efficient low-weight quantum error-correcting (QEC) codes. The main research objective is to develop a method that optimizes the weight of measurements in stabilizer codes while preserving code distance, targeting practically relevant parameter regimes. The key methodology is a Proximal Policy Optimization (PPO) RL algorithm with action masking, operating on Tanner graphs of stabilizer codes, guided by a reward function that balances node degree reduction and code distance preservation. A primary result is that the RL-based method achieves up to a 73x reduction in physical qubit overhead compared to previous weight reduction methods like Sabo et al. (for a 1109,9,14 code). AI practitioners can adapt this RL framework to design low-weight QEC codes with constraints tailored to specific quantum computing architectures, potentially accelerating the implementation of fault-tolerant quantum technologies. |
| Dynamic Concepts Personalization from Single Videos (Read more on arXiv or HuggingFace) | Aliaksandr Siarohin, Willi Menapace, Ivan Skorokhodov, Or Patashnik, Rameen Abdal | The paper introduces “Set-and-Sequence,” a framework for personalizing text-to-video models with dynamic concepts from single videos, enabling high-fidelity generation, editing, and composition. The main objective is to personalize diffusion transformer-based generative video models to capture dynamic concepts, defined by both appearance and motion, from single video examples. The key methodology is a two-stage LoRA training process: (i) “Identity Basis” learning using an unordered set of frames to capture appearance, and (ii) “Motion Residual” encoding using the full video sequence to capture motion dynamics, implemented within a shared spatio-temporal weight space. In editing tasks, the proposed method achieved a mean squared error (MSE) of 0.0221, an identity preservation (ID) score of 0.680, a clip text similarity (C-T) score of 0.239 and a temporal coherency (TC) score of 0.9972. AI practitioners can leverage this framework to embed personalized dynamic concepts into video generation models, improving control over both appearance and motion for enhanced editing and composition capabilities. |
| Scaling Text-Rich Image Understanding via Code-Guided Synthetic Multimodal Data Generation (Read more on arXiv or HuggingFace) | Luca Weihs, Tanmay Gupta, Matt Deitke, Ajay Patel, Yue Yang | The paper introduces CoSyn, a framework for generating synthetic text-rich multimodal data to improve vision-language model (VLM) performance. Main research question or objective: Can leveraging the coding capabilities of text-only large language models (LLMs) automatically generate synthetic text-rich multimodal data to address the limited availability of such data for training VLMs? Key methodology used: The CoSyn framework prompts LLMs to generate code (e.g., Python, HTML, LaTeX) that renders synthetic images, and uses this code as a textual representation to create instruction-tuning data. Primary results: Models trained on CoSyn synthetic data achieved state-of-the-art performance among competitive open-source models on seven text-rich image benchmarks, and models trained on synthetic data boosted average accuracy by 3.6%. Principal implication for AI practitioners: AI practitioners can use the CoSyn framework to generate targeted synthetic text-rich data efficiently, improving VLM performance in specific domains and mitigating the limitations of scarce real-world data. |
| AlphaMaze: Enhancing Large Language Models’ Spatial Intelligence via GRPO (Read more on arXiv or HuggingFace) | Dinh Bach Vu, Alan Dao | AlphaMaze trains large language models (LLMs) on tokenized maze representations to improve spatial reasoning for navigation. The research investigates how to equip standard LLMs with visual reasoning abilities for maze navigation using a two-stage training framework. The methodology combines Supervised Fine-Tuning (SFT) on tokenized maze data and Group Relative Policy Optimization (GRPO) with a custom reward function. Results show the SFT-trained model achieved 86% accuracy on a maze navigation benchmark, which increased to 93% after GRPO fine-tuning. AI practitioners can leverage this two-stage training approach (SFT and GRPO) with tokenized visual representations to enhance LLMs’ spatial reasoning capabilities in tasks requiring sequential decision-making. |
| How Much Do LLMs Hallucinate across Languages? On Multilingual Estimation of LLM Hallucination in the Wild (Read more on arXiv or HuggingFace) | Goran Glavaš, Anne Lauscher, saadob12 | This paper investigates the extent of hallucination in large language models (LLMs) across 30 languages in open-domain, knowledge-intensive question answering. The main research question is: How frequently do LLMs hallucinate across different languages and model sizes in a “real-world” question-answering setting, and how does this relate to language resource availability? Key methodology: The researchers trained a multilingual hallucination detection model using machine-translated English data and created a multilingual evaluation dataset (MFAVA) with LLM-generated and human-annotated examples. They then estimated hallucination rates for six open-source LLM families across 30 languages using a novel protocol based on the detection model’s performance. Primary results: Smaller LLMs and those supporting more languages exhibited significantly higher hallucination rates. The average hallucination rate across languages varied from 7% to 12%. However, there was no correlation between language-normalized hallucination rates and digital language representation. Principal implication for AI practitioners: AI practitioners should be aware that smaller LLM model sizes and models designed for broad multilingual support may be more prone to generating non-factual or unfaithful content in question-answering tasks, necessitating careful model selection and potentially requiring additional mitigation strategies. |
| Geolocation with Real Human Gameplay Data: A Large-Scale Dataset and Human-Like Reasoning Framework (Read more on arXiv or HuggingFace) | Zeyu Zhang, Jonathan Tonglet, Yuan Huang, Jingpu Yang, Ziruibest | This paper introduces a new geolocation framework, including a large-scale dataset, a novel reasoning method, and an evaluation metric, to address challenges in image geolocation. The main research objective is to improve the accuracy and interpretability of image geolocation using real human gameplay data and a human-like reasoning approach. The key methodology involves collecting data from a geolocation game platform (GeoComp dataset), proposing a multi-step reasoning framework (Geographical Chain-of-Thought, GeoCoT), and developing an evaluation metric (GeoEval). The primary results show that GeoCoT improves geolocation accuracy by up to 25% compared to existing methods, achieving a city-level accuracy of 0.118. AI practitioners can leverage the GeoComp dataset and GeoCoT framework to develop and evaluate more robust and interpretable geolocation models, particularly for applications requiring fine-grained localization and human-like reasoning. |
| RelaCtrl: Relevance-Guided Efficient Control for Diffusion Transformers (Read more on arXiv or HuggingFace) | Zhanjie Zhang, Jiasong Feng, Ao Ma, Jing Wang, Ke Cao | RelaCtrl is a framework for efficient controllable generation in Diffusion Transformers, optimizing the integration of control signals. The main objective is to address the high parameter and computational overhead of existing controlled diffusion transformer methods, and their inefficient resource allocation. The key methodology involves evaluating layer relevance to control information using a “ControlNet Relevance Score,” tailoring control layer positioning/capacity, and replacing self-attention/FFN with a Two-Dimensional Shuffle Mixer (TDSM). The approach achieves superior performance with only 15% of the parameters and computational complexity compared to PixArt-δ, as per quantitative experimental results. For AI practitioners, RelaCtrl offers a method for significantly improving the efficiency of controlled image and video generation using Diffusion Transformers, reducing resource demands without compromising output quality. |
| LLM-based User Profile Management for Recommender System (Read more on arXiv or HuggingFace) | Hwanjun Song, Breadbang | PURE is an LLM-based recommendation framework that constructs and maintains evolving user profiles for zero-shot recommendation. The main research objective is to develop a system that can effectively leverage user-generated textual data, beyond purchase history, to improve recommendation accuracy in a continuously evolving setting. The key methodology is PURE, composed of a Review Extractor (extracting preferences from reviews), a Profile Updater (refining user profiles), and a Recommender (generating recommendations using updated profiles). Experimental results on Amazon datasets show that PURE (ICL) achieves an N@10 score of 35.60 on Games and 32.03 on Movies, outperforming baselines that only use purchase history or naively combine reviews. For AI practitioners, PURE demonstrates the concrete value of incorporating long-term review data and user preference through structured profiles. |
| Unstructured Evidence Attribution for Long Context Query Focused Summarization (Read more on arXiv or HuggingFace) | David Jurgens, Isabelle Augenstein, Lu Wang, Zain Muhammad Mujahid, dwright37 | Here’s a 4-5 sentence summary of the provided AI research paper, adhering to your guidelines: 1. 1-Line Summary: This paper introduces the task of long-context, query-focused summarization with unstructured evidence citation, and proposes a synthetic dataset (SUnsET) to improve models’ ability to extract and cite relevant evidence spans. 2. Main Research Question/Objective: The primary objective is to investigate how well LLMs can generate query-focused summaries from long contexts while citing unstructured evidence, and how to mitigate positional biases (like “lost-in-the-middle”) affecting evidence selection. 3. Key Methodology: The authors create SUnsET, a synthetic dataset generated via a novel domain-agnostic pipeline, and use it to fine-tune LLMs with LoRA adapters. They evaluate on four datasets of varying document types/lengths, using position-aware and position-agnostic training. 4. Primary Results: Fine-tuning on SUnsET significantly improves evidence extraction and citation accuracy across multiple LLMs and datasets. A key quantitative finding is citation rates increase dramatically: (6.8× for Mixtral 8x7B with position-aware training). Training also improves summary quality, though shuffling document sections during training can mitigate positional biases. 5. Principal Implication for AI Practitioners: AI practitioners can use the SUnsET dataset and fine-tuning approach to adapt LLMs for improved unstructured evidence citation in long-context summarization, leading to more transparent and reliable summaries, but must be aware that current methods are prone to errors. |
Papers for 2025-02-20
| Title | Authors | Summary |
|---|---|---|
| Qwen2.5-VL Technical Report (Read more on arXiv or HuggingFace) | Keqin Chen, Shuai Bai, xhyandwyy, darkpromise, ayumiymk | i) Qwen2.5-VL is a new vision-language model in the Qwen series with advancements in visual recognition, object localization, document parsing, and long-video comprehension. ii) The research aims to improve the foundational and agentic capabilities of vision-language models, particularly in fine-grained visual perception and real-world applications. iii) The methodology involves training a native dynamic-resolution Vision Transformer (ViT) from scratch, incorporating Window Attention, dynamic FPS sampling, absolute time encoding with MROPE, and curating a large pre-training dataset of 4.1 trillion tokens. iv) The Qwen2.5-VL-72B model achieves 74.8 on MathVista and mIoU score of 50.9 on Charades-STA, and matches state-of-the-art performance, while smaller models offer strong capabilities in resource-constrained environments. v) AI practitioners can leverage Qwen2.5-VL’s improved document understanding, precise object grounding, and long-video comprehension to develop more robust and versatile multimodal applications, particularly in domains requiring detailed visual analysis and interactive agent functionalities, with attention to the computational benefits conferred by Window Attention and dynamic resolution processing. |
| RAD: Training an End-to-End Driving Policy via Large-Scale 3DGS-based Reinforcement Learning (Read more on arXiv or HuggingFace) | Yiang Shi, Bencheng Liao, Bo Jiang, Shaoyu Chen, Hao605 | RAD establishes a 3DGS-based closed-loop Reinforcement Learning (RL) paradigm for training end-to-end autonomous driving policies. The main research objective is to address causal confusion and the open-loop gap in existing Imitation Learning (IL) methods for autonomous driving. The key methodology involves constructing photorealistic digital replicas of the real world using 3D Gaussian Splatting (3DGS) techniques, incorporating IL as a regularization term in RL training, and designing specialized safety-related rewards. The primary results show that, compared to IL-based methods, RAD achieves a 3x lower collision rate on a closed-loop evaluation benchmark consisting of unseen 3DGS environments. For AI practitioners, this suggests that 3DGS-based RL training, combined with IL, can improve the safety and robustness of end-to-end autonomous driving policies, by allowing large scale training in a realistic virtual world. |
| SongGen: A Single Stage Auto-regressive Transformer for Text-to-Song Generation (Read more on arXiv or HuggingFace) | Pan Zhang, Xiaoyi Dong, Zhixiong Zhang, Shuangrui Ding, Zihan Liu | SongGen is a single-stage auto-regressive transformer model for generating songs with vocals and accompaniment from text inputs. The main research objective is to investigate whether a single-stage model can achieve effective text-to-song generation, simplifying the often cumbersome multi-stage pipelines. The key methodology involves a transformer decoder that predicts audio tokens, incorporating user controls via cross-attention, and exploring mixed and dual-track output modes with diverse token patterns. Primary results show that the “Interleaving (A-V)” dual-track mode achieves a Frechet Audio Distance (FAD) of 1.87, competitive with mixed-mode generation. AI practitioners can use SongGen as an open-source, controllable baseline for text-to-song generation, and the provided annotated data and preprocessing pipeline simplify future research. |
| MoM: Linear Sequence Modeling with Mixture-of-Memories (Read more on arXiv or HuggingFace) | Yu Cheng, Jiaxi Hu, Disen Lan, Jusen Du, weigao266 | MoM introduces a linear sequence modeling architecture that uses multiple memory states to improve recall performance. The main research objective is to enhance the memory capacity and reduce memory interference in linear sequence models, addressing limitations of existing approaches that compress sequences into a single fixed-size state. The methodology involves a Mixture-of-Memories (MoM) architecture with multiple independent memory states and a router network that directs input tokens to specific memory states, using an RNN-like update mechanism. Primary results show that MoM significantly outperforms current linear sequence models on downstream language tasks, with the 1.3B parameter MoM achieving an average score of 36.04 on recall-intensive tasks, close to the Transformer model’s 37.31. For AI practitioners, MoM offers a more efficient architecture to enhance the memory and recall of linear sequence modeling for applications, retaining linear-time training and constant-memory inference, presenting itself as an alternative to Transformers. |
| Craw4LLM: Efficient Web Crawling for LLM Pretraining (Read more on arXiv or HuggingFace) | Chenyan Xiong, Zhiyuan Liu, yushi | CRAW4LLM is an efficient web crawling method that prioritizes webpages based on their predicted influence on large language model (LLM) pretraining. The research objective is to improve the efficiency of web crawling for LLM pretraining data collection by aligning crawler priorities with LLM pretraining needs. The key methodology is to use a pretraining influence scorer, derived from data-filtering pipelines, to score newly discovered documents and prioritize them in the crawler’s queue, replacing traditional graph-connectivity-based metrics. Primary results show that LLMs pretrained on data crawled by CRAW4LLM, using only 21% of the URLs, achieve the same downstream performance as previous crawls that used more data. Principal implication is that by using CRAW4LLM AI practitioners can get similar performing LLM, while significantly reducing the required web crawling and data processing, thus saving time and resources. |
| LongPO: Long Context Self-Evolution of Large Language Models through Short-to-Long Preference Optimization (Read more on arXiv or HuggingFace) | Lidong Bing, Michael Qizhe Shieh, Xin Li, Guanzheng Chen | LongPO is a method that enables short-context LLMs to self-evolve to handle long-context tasks by internally transferring short-context capabilities through preference optimization. The main research objective is to address the challenges of long-context alignment in LLMs, specifically the scarcity of long-context annotated data and the difficulty in balancing short- and long-context performance. The key methodology involves generating short-to-long preference data using a short-context LLM and applying a DPO-style objective with a KL constraint to maintain short-context performance. The primary result is that LongPO applied to Mistral-7B-Instruct-v0.2 improved performance on InfiniteBench by 25.45 points and achieved comparable or superior results to larger LLMs like GPT-4-128K. The principal implication for AI practitioners is that LongPO offers an efficient way to extend the context length of LLMs without extensive long-context data annotation or significant degradation of short-context capabilities, providing a more balanced approach to developing long-context LLMs. |
| Small Models Struggle to Learn from Strong Reasoners (Read more on arXiv or HuggingFace) | Luyao Niu, Fengqing Jiang, Xiang Yue, Yuetai Li, flydust | Small language models (≤3B parameters) do not consistently benefit from complex reasoning data or distillation from larger models, instead performing better with simpler reasoning. The main research question is whether small language models can effectively learn from the reasoning capabilities of larger, more powerful language models. The key methodology involves fine-tuning student models of varying sizes on different types of Chain-of-Thought (CoT) data (long, short, large teacher, small teacher) generated from the MATH dataset and evaluating their performance on multiple math benchmarks. A key result is that Qwen2.5-3B-Instruct improves by more than 8 points on MATH and AMC using Mix-Long, compared to direct training on long CoT data. The principal implication is that AI practitioners should adapt reasoning complexity during distillation, using techniques like Mix Distillation, to effectively transfer reasoning capabilities to smaller models, instead of directly using complex reasoning data from large models. |
| Autellix: An Efficient Serving Engine for LLM Agents as General Programs (Read more on arXiv or HuggingFace) | Tianjun Zhang, Colin Cai, Xiaoxiang Shi, Michael Luo, Chrisyichuan | Autellix is an LLM inference system designed to efficiently serve agentic programs, treating them as first-class citizens to minimize end-to-end latency. The main research objective is to reduce the end-to-end latencies of agentic programs composed of dynamic, non-deterministic DAGs of LLM calls and interrupts. The key methodology used is program-aware scheduling, prioritizing LLM calls based on program-level statistics (cumulative service time) and employing a data locality-aware load balancer across multiple engines. Primary results show that Autellix improves program throughput by 4-15x compared to state-of-the-art systems like vLLM, across diverse LLMs and agentic workloads. The principal implication is that AI practitioners can significantly improve the performance of LLM agent applications by using a serving system that prioritizes the scheduling of LLM calls based on full program execution, and data-locality, rather than treating each call independently. |
| Presumed Cultural Identity: How Names Shape LLM Responses (Read more on arXiv or HuggingFace) | Lucie-Aimée Kaffee, Arnav Arora, Siddhesh Pawar, IAugenstein | LLMs exhibit cultural biases in responses based on user names, influencing personalization. The main research objective is to investigate cultural presumptions in LLM responses when presented with common suggestion-seeking queries including user names. The key methodology involves prompting LLMs with names from 30 cultures and analyzing generated responses for cultural bias using an LLM-as-a-judge approach and assertion-based evaluation. The primary result showed that LLM responses exhibit varying degrees of cultural bias, with clothing-related queries showing a roughly 70% increase in bias when names were included. Principal implication is that AI practitioners need to consider the impact of names on LLM outputs and design personalisation systems that avoid reinforcing stereotypes while utilizing names. |
| Why Safeguarded Ships Run Aground? Aligned Large Language Models’ Safety Mechanisms Tend to Be Anchored in The Template Region (Read more on arXiv or HuggingFace) | Wenjie Li, Jian Wang, Qingyu Yin, Chak Tou Leong | Aligned large language models (LLMs) exhibit a vulnerability where their safety mechanisms overly rely on information within a specific “template region” inserted between user input and model output. The research investigates the phenomenon of “template-anchored safety alignment” (TASA) in aligned LLMs. The methodology involves analyzing attention weight distributions, performing activation patching interventions, and probing harmfulness features across different layers and positions, and propose a detaching safety mechanism. Results show that intervening in intermediate states in template region significantly increases the likelihood of harmful initial compliance decisions, with a normalized indirect effect (NIE) showing considerable gains by patching small number of heads. The findings suggest AI practitioners should develop more robust safety alignment techniques that are less reliant on the template region for safety-related decision-making to reduce the risk of adversarial attacks. |
| SearchRAG: Can Search Engines Be Helpful for LLM-based Medical Question Answering? (Read more on arXiv or HuggingFace) | Tianming Liu, Quanzheng Li, Canyu Chen, Tianze Yang, YuchengShi | SearchRAG is a novel retrieval-augmented generation framework that leverages search engines to enhance large language models’ (LLMs) performance in medical question answering. The main research objective is to determine how to effectively integrate search engines with LLMs for improved retrieval of medical knowledge. The key methodology involves synthetic query generation using LLMs to create search-engine-friendly queries and uncertainty-based knowledge selection to filter retrieved information. Primary results show that SearchRAG improved the LLaMA 8B model’s accuracy by an average of 12.61% compared to baseline methods on medical QA tasks. Principal implication for AI practitioners is that SearchRAG’s method is capable of adressing limitations of conventional Retrieval-Augmented Generation (RAG) systems, showing that real-time search integration improves response accuracy. |
| Thinking Preference Optimization (Read more on arXiv or HuggingFace) | Xiaotian Han, Vipin Chaudhary, Jingfeng Yang, Hongye Jin, Wang Yang | Thinking Preference Optimization (ThinkPO) enhances reasoning in fine-tuned language models without requiring new long chain-of-thought (CoT) responses. The main research objective is to improve the reasoning performance of supervised fine-tuned (SFT) language models without collecting new long CoT data or repeatedly training on existing SFT datasets. The key methodology is to use readily available short CoT reasoning responses as rejected answers and existing long CoT responses as chosen answers, applying direct preference optimization (DPO) to encourage longer reasoning outputs. The primary result is that ThinkPO increases the math reasoning accuracy of SFT-ed models by 8.6% and output length by 25.9%, for example it increased performance on MATH500 of a tested model from 87.4% to 91.2%. AI practitioners can use ThinkPO as a post-SFT method to further improve the reasoning performance of their models, especially when acquiring new long CoT data is costly or repeated training leads to a performance plateau. |
| Is That Your Final Answer? Test-Time Scaling Improves Selective Question Answering (Read more on arXiv or HuggingFace) | Benjamin Van Durme, Jeffrey Cheng, wjurayj | Test-time scaling of compute improves the performance of large language models on selective question answering by increasing confidence in correct answers. The research investigates how increasing computational budget at inference time impacts model confidence and accuracy in question answering. The methodology involves evaluating models at varying compute budgets and confidence thresholds, using a selection function that rejects answers below a confidence threshold. The results show that increasing the compute budget improves the average confidence of correct answers, and selective answering at a threshold of 0.95 dramatically improves performance in a Jeopardy setting where incorrect answers are penalized. AI practitioners should report test-time scaling performance under conditions that penalize incorrect answers (“Jeopardy Odds”) in addition to traditional settings, to accurately reflect selective question answering capabilities. |
| AdaptiveStep: Automatically Dividing Reasoning Step through Model Confidence (Read more on arXiv or HuggingFace) | Jason Klein Liu, Chaofeng Qu, Zhaoling Chen, Junjie Lu, Yuliang Liu | AdaptiveStep, a novel method, automatically divides reasoning steps in large language models (LLMs) based on model confidence to enhance process reward model (PRM) training and performance. The main research objective is to develop an automated, informative, and general method for dividing reasoning steps that improves upon existing rule-based approaches. The key methodology, AdaptiveStep, utilizes the LLM’s prediction confidence for the next word to identify critical breaking points, creating more informative step divisions without manual annotation. Results show that the AdaptiveStep-trained PRM (ASPRM) achieves state-of-the-art Best-of-N performance, outperforming greedy search with token-level value-guided decoding (TVD) by 3.15% on GSM8k. For AI practitioners, AdaptiveStep provides a more efficient and precise method for training PRMs, reducing construction costs and enhancing downstream task performance, specifically in mathematical reasoning and code generation. |
| NExT-Mol: 3D Diffusion Meets 1D Language Modeling for 3D Molecule Generation (Read more on arXiv or HuggingFace) | Enzhi Zhang, Han Huang, Yanchen Luo, Zhiyuan Liu, xiangwang1223 | NExT-Mol is a foundation model for 3D molecule generation that combines 3D diffusion with 1D language modeling. The main research objective is to improve 3D molecule generation by integrating the strengths of 1D SELFIES-based language models (LMs) and 3D diffusion models. The methodology involves pretraining a 960M parameter 1D molecule LM (MoLlama) on 1.8B SELFIES, then predicting 3D conformers with a novel diffusion model (Diffusion Molecule Transformer, DMT) and using cross-model transfer learning to enhance DMT. NExT-Mol achieves a 26% relative improvement in 3D FCD for de novo 3D generation on GEOM-DRUGS compared to previous methods. AI practitioners can leverage this approach to generate 3D molecules with improved validity and distributional similarity, facilitating drug discovery and material design by combining large-scale 1D pretraining with 3D diffusion. |
| ActionPiece: Contextually Tokenizing Action Sequences for Generative Recommendation (Read more on arXiv or HuggingFace) | Wang-Cheng Kang, Noveen Sachdeva, Zhankui He, Jianmo Ni, hyp1231 | ActionPiece is a novel tokenization method for generative recommendation that incorporates contextual information to improve performance. The main research objective is to develop a context-aware action sequence tokenizer for generative recommendation models, addressing the limitation of existing models that tokenize each action independently. The key methodology, ActionPiece, represents each action as a set of item features, constructs a vocabulary by merging frequent feature patterns, and uses set permutation regularization to produce multiple segmentations. The primary result is that ActionPiece outperforms existing action tokenization methods, improving NDCG@10 by 6.00% to 12.82% on public datasets. The principal implication is that AI practitioners can use ActionPiece to improve the accuracy and efficiency of generative recommendation systems by considering contextual relationships among user actions. |
| Train Small, Infer Large: Memory-Efficient LoRA Training for Large Language Models (Read more on arXiv or HuggingFace) | Ke Chen, Lidan Shou, Huan Li, Jue Wang, junzhang98 | LORAM is introduced as a memory-efficient LoRA training scheme for LLMs. This research aims to reduce the memory footprint of LoRA training by training on a pruned model and recovering weights for inference on the original model. LORAM employs pruning during training followed by a recovery and alignment phase utilizing continual pre-training on a small dataset. QLORAM, combining structured pruning and 4-bit quantization, achieved a 15.81× parameter storage reduction for LLaMA-3.1-70B while maintaining or improving performance. LORAM enables training on resource-constrained hardware and suggests an alternative to full fine-tuning. |
| GIMMICK – Globally Inclusive Multimodal Multitask Cultural Knowledge Benchmarking (Read more on arXiv or HuggingFace) | Anne Lauscher, Chris Biemann, Carolin Holtermann, floschne | i) GIMMICK introduces a multimodal benchmark for evaluating cultural knowledge in large vision-language models (LVLMs). ii) The research aims to identify regional biases in LLMs’ and LVLMs’ cultural understanding and assess the impact of model size, input modalities, and external cues on cultural knowledge. iii) The methodology employs six tasks built on three newly created datasets spanning 728 cultural events across 144 countries, evaluating 31 models using multimodal and unimodal inputs. iv) Results reveal significant regional biases, with models exhibiting up to 14.72pp performance difference between Western and Sub-Saharan African cultural contexts, and multimodal input consistently improving performance. v) AI practitioners should be aware of biases in cultural understanding and leverage multimodal inputs to create more globally inclusive AI systems. |
| InfiR : Crafting Effective Small Language Models and Multimodal Small Language Models in Reasoning (Read more on arXiv or HuggingFace) | Zhijie Sang, Pengxiang Li, Wenjun Wang, Shuo Cai, Congkai Xie | InfiR introduces efficient Small Language Models (SLMs) and Multimodal SLMs with enhanced reasoning capabilities, deployable on edge devices. The main research objective is to develop SLMs and MSLMs that retain competitive reasoning abilities while reducing model size and computational demands. The key methodology involves a novel pre- and post-training pipeline that includes heuristic filtering, reasoning-oriented text recall, data annealing, and supervised fine-tuning with synthetic data. The InfiR-1B-Instruct model achieved a 2.26x reasoning-related average score improvement over Llama3.2-1B-Base. AI practitioners can leverage InfiR’s training pipeline and models to build efficient and privacy-preserving AI systems with strong reasoning capabilities, particularly for edge deployment. |
| Noise May Contain Transferable Knowledge: Understanding Semi-supervised Heterogeneous Domain Adaptation from an Empirical Perspective (Read more on arXiv or HuggingFace) | Qiang Yang, Jian Jin, Yu Zhang, Xiaopu Zhang, yyyaoyuan | This paper empirically investigates transferable knowledge in semi-supervised heterogeneous domain adaptation (SHDA) tasks. The main research question is: “What is the transferable knowledge in SHDA?” The authors develop a unified Knowledge Transfer Framework (KTF) for SHDA and conduct extensive experiments, including manipulating source sample categories, features, and introducing synthesized noise distributions. A primary result across nearly 330 SHDA tasks is that varying source sample category orders has almost no change in the performance, i.e. average accuracy remains nearly constant. For AI practitioners, the results imply that the discriminability and transferability of the source domain, rather than the category or feature information, are the main factors for effective transfer in SHDA, meaning the choice of origin for source domains is less critical than ensuring those two qualities. |
Papers for 2025-02-19
| Title | Authors | Summary |
|---|---|---|
| Soundwave: Less is More for Speech-Text Alignment in LLMs (Read more on arXiv or HuggingFace) | Benyou, PhoenixAxis, FanBuCUHK, puccho, Yoohao | Soundwave utilizes an efficient training strategy and novel architecture to address representation space gap and sequence length inconsistency between speech and text in LLMs. The main research objective is to achieve data-efficient training for speech-text alignment in large language models. The key methodology is a two-stage training framework: Stage I aligns speech and text representations using an alignment adapter and CTC loss; Stage II reduces speech sequence length using a shrinking adapter. Soundwave outperforms Qwen2-Audio in speech translation and AIR-Bench speech tasks, using only one-fiftieth of the training data (10k hours vs. 520k hours). AI practitioners can achieve state-of-the-art speech understanding performance in LLMs with significantly reduced training data requirements by adopting Soundwave’s two-stage alignment and shrinking approach. |
| Phantom: Subject-consistent video generation via cross-modal alignment (Read more on arXiv or HuggingFace) | Jiawei Liu, ZhuoweiChen, lbc402, Grayson111, liulj13 | Phantom is a unified video generation framework for subject-consistent video generation via cross-modal alignment. The research objective is to develop a model that balances dual-modal prompts of text and image to achieve deep and simultaneous alignment of text and visual content in video generation. The key methodology involves redesigning a joint text-image injection model based on text-to-video and image-to-video architectures, and training it with text-image-video triplet data to learn cross-modal alignment. Primary results show Phantom leads in overall metrics for subject consistency with a score of 0.731 in CLIP-I-Seg and prompt following with the ViCLIP-T, demonstrating subject consistency competitive with commercial solutions. AI practitioners can use Phantom, which has a new architecture, for improved subject-consistent video generation, especially in tasks requiring ID preservation and consistency. |
| Continuous Diffusion Model for Language Modeling (Read more on arXiv or HuggingFace) | Sung Ju Hwang, harryjo97 | Riemannian Diffusion Language Model (RDLM) is a continuous diffusion framework for language modeling that incorporates the geometry of the statistical manifold. The main research objective is to establish a connection between discrete diffusion and continuous flow on the statistical manifold and design a continuous diffusion model for discrete data that generalizes previous discrete diffusion models. The key methodology involves reparameterizing discrete data to continuous states on a hypersphere, designing diffusion processes on the manifold that generalize discrete diffusion, and using a simulation-free training scheme based on radial symmetry. Primary results show that RDLM achieves a Bits Per Character (BPC) of ≤ 1.32 on the Text8 dataset, outperforming existing discrete diffusion models. The principal implication is that AI practitioners can leverage the geometry of the statistical manifold in continuous diffusion models to achieve improved performance in language modeling and other discrete data generation tasks, compared to existing discrete diffusion approaches. |
| Cramming 1568 Tokens into a Single Vector and Back Again: Exploring the Limits of Embedding Space Capacity (Read more on arXiv or HuggingFace) | Aydar Bulatov, Mikhail Arkhipov, mbur, yurakuratov | This work explores the maximum information capacity of language model input embeddings by compressing text sequences into trainable vectors. The main research objective is to quantify how much text can be losslessly encoded into and decoded from a fixed-size vector representation within large language models (LLMs). The key methodology involves optimizing a set of prepended “memory” vectors to minimize the cross-entropy loss when reconstructing the original text using a frozen, pre-trained LLM. The primary result is that a single vector can enable a Llama-3.1-8B model to accurately reconstruct up to 1568 tokens, and this capacity scales nearly linearly with the number of trainable vectors (e.g. 16 vectors compress 7168 tokens). The principal implication for AI practioners is that LLM input embeddings have significantly more unused capacity than typically utilized, suggesting substantial room for improved context encoding and memory augmentation in model design. |
| SafeRoute: Adaptive Model Selection for Efficient and Accurate Safety Guardrails in Large Language Models (Read more on arXiv or HuggingFace) | Minki Kang, Dong Bok Lee, hbseong, dwgnr, Seanie-lee | SafeRoute adaptively selects between a smaller and larger safety guard model to improve the trade-off between computational cost and safety performance in LLM deployments. The paper’s objective is to develop a method that distinguishes “hard” examples requiring a larger safety guard model from “easy” ones that a smaller model can handle. The core of the method is SafeRoute, a trained binary router that classifies input prompt-response pairs, selectively applying the larger model only when necessary. Results show SafeRoute improves the F1 score by 13% and 10% compared to always using the smaller or larger models on the WildGuardMix test split, while utilizing the larger model on only 5.09% of the data. AI practitioners can use SafeRoute to deploy safer LLMs more efficiently, reducing computational overhead while maintaining high accuracy in detecting harmful content. |
| Rethinking Diverse Human Preference Learning through Principal Component Analysis (Read more on arXiv or HuggingFace) | Hao Sun, Feng Luo, huanzhang12, CharlesDDDD, Ray2333 | Decomposed Reward Models (DRMs) extract diverse human preferences from binary comparisons for improved AI personalization. The research question is: Can we infer multidimensional human preferences directly from large-scale binary comparisons? The method represents preferences as vectors, applies PCA to embedding differences between preferred and rejected responses, and identifies orthogonal basis vectors representing distinct preference aspects. DRMs using Gemma-2B-RM improved the single-head baseline accuracy from 0.733 to 0.814 on the RewardBench dataset. AI practitioners can use DRMs for more efficient test-time adaptation to diverse user preferences without requiring additional model training, offering a scalable and interpretable solution for personalized LLM alignment. |
| SoFar: Language-Grounded Orientation Bridges Spatial Reasoning and Object Manipulation (Read more on arXiv or HuggingFace) | codered010, RunpeiDong, YufeiD, WenyaoZhang, qizekun | SOFAR introduces semantic orientation to bridge spatial reasoning and object manipulation, enabling robots to understand and execute tasks based on natural language instructions. The main research objective is to develop a system that can accurately understand and utilize object orientations, defined through natural language, for robotic manipulation and spatial reasoning tasks. The key methodology involves constructing a large-scale dataset (OrienText300K) of 3D models annotated with semantic orientations, developing a cross-modal 3D Transformer (PointSO) for orientation prediction, and integrating this with a Vision-Language Model (VLM) system (SOFAR) to generate manipulation actions. Primary results show that SOFAR achieves 48.7% accuracy on the Open6DOR benchmark and 74.9% accuracy on the SIMPLER benchmark for robotic manipulation. The principal implication for AI practitioners is that integrating semantic orientation into VLM systems provides a more flexible and accurate way to represent spatial knowledge, significantly improving performance in robotic manipulation tasks requiring precise object alignment and rearrangement. |
| Multimodal Mamba: Decoder-only Multimodal State Space Model via Quadratic to Linear Distillation (Read more on arXiv or HuggingFace) | Qian Zhang, wenyuliu, wondervictor, HongyuanTao, LegendBC | mmMamba is a framework for developing linear-complexity, native multimodal state space models using distillation from existing multimodal large language models (MLLMs). The main research question is how to effectively distill knowledge from trained Transformer-based decoder-only MLLMs to create efficient, linear-complexity architectures without relying on pre-trained RNN-based LLMs or vision encoders. The key methodology involves a three-stage progressive distillation recipe and a seeding strategy to carve Mamba layers from trained Transformer layers, transferring knowledge while preserving multimodal capabilities. The primary results demonstrate that mmMamba-linear achieves competitive performance with existing linear and quadratic-complexity VLMs, achieving a 20.6x speedup and 75.8% GPU memory saving compared to HoVLE at 103K tokens. AI practitioners can leverage mmMamba to build more efficient and deployable multimodal models, particularly for long-context applications, by utilizing linear-complexity architectures with reduced computational demands. |
| FLAG-Trader: Fusion LLM-Agent with Gradient-based Reinforcement Learning for Financial Trading (Read more on arXiv or HuggingFace) | ShirleyY, Acatsama, YupengCao, zdeng10, xionggj001 | FLAG-TRADER is a framework integrating LLMs with reinforcement learning for financial trading. The main research question is whether integrating LLMs’ reasoning with RL’s reward-driven optimization can address challenges in financial sequential decision-making. The methodology involves a partially fine-tuned LLM acting as a policy network, optimized via gradient-driven RL (specifically PPO), using textual state representations. Primary results show FLAG-TRADER, using a 135M-parameter LLM, achieves a Sharpe Ratio of 3.344 on JNJ stock, outperforming baselines and larger proprietary models. For AI practitioners, this framework demonstrates that combining LLMs with RL fine-tuning, particularly using parameter-efficient methods, offers superior performance in complex, sequential decision-making tasks like financial trading. |
| You Do Not Fully Utilize Transformer’s Representation Capacity (Read more on arXiv or HuggingFace) | kefirski, ummagumm-a, elephantmipt, yaraksen, gudleifrr | i) This paper introduces Layer-Integrated Memory (LIMe), a modification to the Transformer architecture that allows attention heads to access representations from all previous layers. ii) The main objective is to address representation collapse in standard Transformers by enabling access to hidden states from earlier layers. iii) The key methodology is modifying the key-value side of masked multi-head self-attention by introducing a learned routing mechanism (static or dynamic) that creates convex combinations of representations from all preceding layers. iv) LIMe models consistently outperform standard Transformer baselines; for example, on the LM Evaluation Harness, the average accuracy across all benchmarks in the results shows the LIMe Dynamic variant achieving 58.4% accuracy, compared to 57.7% for the LLaMA baseline. v) AI practitioners can use LIMe to build deeper and more robust Transformers with improved representational capacity, potentially leading to better performance in sequence modeling tasks without substantially increasing computational overhead. |
| Magma: A Foundation Model for Multimodal AI Agents (Read more on arXiv or HuggingFace) | cheryyunl, Baolin, rzheng12, qianhuiwu, tanreuben | Magma is a multimodal foundation model capable of interpreting and grounding multimodal inputs within its environment for AI agentic tasks. The main research objective is to develop a foundation model that integrates vision-language understanding with the ability to plan and act in visual-spatial worlds, completing tasks ranging from UI navigation to robot manipulation. The key methodology involves pre-training on heterogeneous datasets (images, videos, robotics data) using Set-of-Mark (SoM) for action grounding and Trace-of-Mark (ToM) for action planning, representing actions as visual object labels and movement traces. Primary results include achieving new state-of-the-art results on UI navigation with a success rate of 60.4/58.5 on SS-Mobile, and robotic manipulation tasks, outperforming previous models tailored to these tasks. For AI practitioners, Magma provides a pre-trained model capable of transferring visual and language understanding to complex agentic tasks, suggesting a path for building agents that can seamlessly operate in both digital and physical environments. |
| RealSyn: An Effective and Scalable Multimodal Interleaved Document Transformation Paradigm (Read more on arXiv or HuggingFace) | Kaicheng Yang, JiankangDeng, SeriousBro, Nina0607, GaryGuuu | i) RealSyn introduces a paradigm for vision-language representation learning using multimodal interleaved documents. ii) The research aims to leverage underutilized non-paired data in interleaved documents by constructing distinct image-text pairs. iii) The methodology involves a real-world data extraction pipeline, hierarchical retrieval to associate images with texts, and an image semantic augmented generation module. iv) The study releases the RealSyn dataset and demonstrates that models pre-trained on RealSyn achieve state-of-the-art performance on multiple downstream tasks and showed performance improvements of 1.3%-6.9% in linear probing. v) RealSyn offers a scalable dataset, up to 100M, for AI practitioners enabling improved vision-language models without relying solely on paired data. |
| PAFT: Prompt-Agnostic Fine-Tuning (Read more on arXiv or HuggingFace) | Fei Richard Yu, Ying Tiffany He, Mingwen Ou, Yao Shu, kittttttt | PAFT is a fine-tuning method that improves the prompt robustness of large language models (LLMs). The main research objective is to address the performance degradation of fine-tuned LLMs caused by minor variations in prompts. The key methodology is a two-stage approach: constructing a diverse set of candidate prompts and then dynamically sampling from these prompts during fine-tuning. Primary results show that PAFT achieves 87.57% average accuracy on the RACE-high dataset, significantly outperforming baseline models and reducing variance across different prompts. PAFT’s dynamic sampling during fine-tuning helps models generalize better to unseen prompts, maintaining high performance and improving inference efficiency for AI practitioners using fine-tuned models. |
| MUDDFormer: Breaking Residual Bottlenecks in Transformers via Multiway Dynamic Dense Connections (Read more on arXiv or HuggingFace) | Xingyuan Yuan, Da Xiao, lishengping, Hilbertmeng | MUDDFormer introduces a novel method to improve information flow in Transformers by replacing standard residual connections with multiway dynamic dense connections. The main research objective is to address the limitations of residual connections and enhance cross-layer information flow in Transformer models. The key methodology is generating connection weights dynamically based on hidden states and decoupling input streams (query, key, value, residual) of a Transformer block. Primary results show that MUDDPythia-2.8B matches Pythia-6.9B in pre-training perplexity and downstream tasks, while adding only 0.23% parameters and 0.4% computation. For AI practitioners, MUDDFormer offers a method to significantly improve Transformer performance and scalability, especially with deeper models, with minimal parameter and computational overhead. |
| Revisiting the Test-Time Scaling of o1-like Models: Do they Truly Possess Test-Time Scaling Capabilities? (Read more on arXiv or HuggingFace) | Yunhua Zhou, Qinyuan Cheng, Zhiyuan Zeng, xpqiu, yinzhangyue | This paper investigates whether o1-like models (QwQ, R1, and LIMO) truly possess test-time scaling capabilities. The main research question is whether increasing Chain-of-Thought (CoT) length in these models consistently improves reasoning performance. The researchers systematically investigated the relationship between CoT length and accuracy, and prompted models for self-revisions, comparing sequential and parallel scaling strategies. A primary result is that longer CoTs did not consistently improve accuracy; correct solutions were often shorter, and R1-Distill-32b and R1-Distill-14b maintained the original wrong answer in over 70% of cases when prompted to revise. The principal implication is that AI practitioners should consider parallel scaling and methods like “Shortest Majority Vote” for these models, as sequential scaling via self-revision is not consistently effective due to limited self-revision capabilities. |
| OctoTools: An Agentic Framework with Extensible Tools for Complex Reasoning (Read more on arXiv or HuggingFace) | Joseph Boen, Rahul Thapa, Sheng Liu, Bowen Chen, lupantech | OctoTools is a training-free, extensible agentic framework that enhances complex reasoning in large language models (LLMs) through standardized tool integration and a planner-executor paradigm. The main research objective is to develop a framework that enables LLMs to effectively tackle complex reasoning tasks across diverse domains without requiring additional training or fine-tuning. Key methodology involves using standardized tool cards to encapsulate tool functionality, a planner for high-level and low-level task planning, and an executor to carry out tool usage based on generated commands. Primary results show that OctoTools achieves an average accuracy gain of 9.3% over zero-shot GPT-4o and outperforms other agent frameworks like AutoGen, GPT-Functions, and LangChain by up to 10.6% when given the same set of tools. Principal implication for AI practitioners is that OctoTools provides a modular and extensible framework for building AI agents capable of complex reasoning, which reduces development effort and improves performance without the need for model retraining when new tools are added. |
| Crowd Comparative Reasoning: Unlocking Comprehensive Evaluations for LLM-as-a-Judge (Read more on arXiv or HuggingFace) | zhangsan5421, lifengshang, horiz94, YuxinJiang, DonJoey | Crowd Comparative Reasoning enhances LLM-as-a-Judge evaluations by incorporating comparisons with multiple “crowd” responses to improve detail and comprehensiveness. Research Objective: To address the limitation of LLM-as-a-Judge’s chain-of-thought (CoT) reasoning, which often fails to capture comprehensive details, leading to incomplete evaluations. Key Methodology: Proposes Crowd-based Comparative Evaluation (CCE), which introduces additional “crowd” responses for comparison with candidate responses, guiding the LLM to produce more detailed CoT judgments. Primary Results: CCE achieved an average accuracy gain of 6.7% across five benchmarks (REWARDBENCH, HELPSTEER2, MTBENCH HUMAN, JUDGEBENCH, and EvalBIAS). Principal Implication: AI practitioners can use CCE to improve the reliability and depth of LLM-based evaluations, enabling more robust model assessments and potentially more efficient training through techniques like judge distillation and improved rejection sampling. |
| HealthGPT: A Medical Large Vision-Language Model for Unifying Comprehension and Generation via Heterogeneous Knowledge Adaptation (Read more on arXiv or HuggingFace) | Binhe Yu, Yuqian Yuan, Sijing Li, Wenqiao Zhang, Tianwei Lin | HealthGPT is a medical large vision-language model that unifies visual comprehension and generation tasks through heterogeneous knowledge adaptation. The main research objective is to develop a unified medical multi-modal model capable of both comprehending and generating medical visual data. The key methodology is a novel heterogeneous low-rank adaptation (H-LoRA) technique, complemented by hierarchical visual perception and a three-stage learning strategy. Results show that HealthGPT-L14 achieves 77.7% close accuracy on VQA-RAD, and 88.6% SSIM on the CT(Brain) reconstruction task. The principal implication is that AI practitioners can leverage HealthGPT’s architecture for creating unified medical AI models that perform well on both visual comprehension and generation, overcoming limitations of previous models. |
| HeadInfer: Memory-Efficient LLM Inference by Head-wise Offloading (Read more on arXiv or HuggingFace) | beidic, junjiehu, jinqixiao, ZefanCai, wdlctc | i) HeadInfer proposes a head-wise offloading strategy for memory-efficient LLM inference by selectively maintaining attention heads’ KV cache on the GPU. ii) The research aims to reduce the GPU memory footprint of LLM inference, specifically the key-value (KV) cache, for long context generation. iii) The methodology involves a head-wise offloading strategy where only selective attention heads’ KV cache is stored on the GPU, dynamically computing attention output, combined with adaptive heads grouping and asynchronous data transfer. iv) Experiments on the Llama-3-8B model with a 1-million-token sequence show a reduction in GPU memory footprint from 128GB to 1GB for the KV cache and total GPU usage from 207GB to 17GB, achieving a 92% reduction compared to BF16 baseline inference; HeadInfer extends the Llama-3-8B model’s context length from 25K to 4 million tokens using an NVIDIA RTX 4090. v) HeadInfer enables AI practitioners to perform long-context LLM inference with reduced memory requirements, specifically enabling 4-million-token inference with an 8B model on a single consumer GPU with 24GB memory. |
| Injecting Domain-Specific Knowledge into Large Language Models: A Comprehensive Survey (Read more on arXiv or HuggingFace) | Mingzhe Li, Miao Fang, Yuhan Liu, Bin Yan, Ziruibest | This survey provides a comprehensive overview of methods for integrating domain-specific knowledge into large language models (LLMs). The main research objective is to categorize and analyze techniques for enhancing LLMs with domain-specific knowledge to improve their performance in specialized tasks. Key methodologies include dynamic knowledge injection, static knowledge embedding, modular adapters, and prompt optimization. The paper reviewed studies showing, for instance, that PMC-LLaMA (13B) achieved 56.3 on MedQA, outperforming LLaMA2 (70B) at 43.7 on the same benchmark, in the biomedical field, showing how domain-specific LLMs can beat generalized models. For AI practitioners, incorporating domain-specific knowledge is crucial for achieving higher accuracy and reliability in specialized applications of LLMs. |
| Eager Updates For Overlapped Communication and Computation in DiLoCo (Read more on arXiv or HuggingFace) | Yanislav Donchev, Arthur Douillard, Satyen Kale | i) This paper introduces “eager updates” to improve the DiLoCo distributed training method by overlapping communication and computation, reducing training time in low-bandwidth settings. ii) The main objective is to mitigate performance slowdowns in distributed training caused by blocking communication in low-bandwidth environments, such as cross-datacenter training. iii) The key methodology is to overlap the communication of outer gradients with the computation of the next inner optimization phase, applying local outer gradients eagerly before the aggregated gradients are available. iv) The proposed method with 1-outer-step eager updates and H=30 inner steps achieves the same performance as Data-Parallel at a 1 billion parameter scale, while using up to 1,177x less bandwidth. v) AI practitioners can use eager updates in DiLoCo to significantly reduce communication requirements and improve training efficiency in settings with limited bandwidth between workers. |
| Atom of Thoughts for Markov LLM Test-Time Scaling (Read more on arXiv or HuggingFace) | Chenglin Wu, Jiayi Zhang, Quan Shi, Zhaoyang Yu, leavendough | Atom of Thoughts (AOT) is a reasoning framework that improves large language models’ (LLMs) test-time scaling by structuring the reasoning process as a Markov chain of atomic, independent questions. The main research objective is to address the issue of accumulated historical information in existing test-time scaling methods, which wastes computational resources and interferes with effective reasoning. The key methodology is a two-phase state transition mechanism: (1) decomposing the current question into a dependency-based directed acyclic graph, and (2) contracting subquestions into a new independent question, iteratively until directly solvable. Primary results show that on HotpotQA, AOT applied to gpt-4o-mini achieves an 80.6% F1 score. The principal implication for AI practitioners is that AOT can be used as a standalone framework or a plug-in enhancement to improve LLMs’ reasoning capabilities, by reducing unnecessary historical information to enhance efficiency. |
| FinMTEB: Finance Massive Text Embedding Benchmark (Read more on arXiv or HuggingFace) | Yi Yang, yixuantt | FinMTEB is a comprehensive benchmark for evaluating text embedding models in the financial domain. The main research objective is to assess how well existing embedding models capture domain-specific financial information and whether domain adaptation improves performance. The key methodology involves constructing a benchmark (FinMTEB) of 64 datasets across 7 financial tasks and developing a finance-adapted model, Fin-E5, using a persona-based data synthesis method. Primary results show domain-adapted models consistently outperform general-purpose counterparts, with Fin-E5 achieving a 0.6767 average score on FinMTEB, and remarkably, a simple Bag-of-Words (BoW) approach outperforms all dense embedding in financial Semantic Textual Similarity (STS) tasks. For AI practitioners, the benchmark facilitates targeted development and assessment of financial text embedding models, and also suggests current dense embedding models may not be optimal for certain kinds of financial text analysis. |
| Perovskite-LLM: Knowledge-Enhanced Large Language Models for Perovskite Solar Cell Research (Read more on arXiv or HuggingFace) | Shuyan Chen, wenxinsiju, yongqi2023, sunpenglei, Dominic789654 | This paper presents a knowledge-enhanced system for perovskite solar cell (PSC) research, integrating a knowledge graph, datasets, and specialized large language models. The main research objective is to develop a system that efficiently manages and reasons with the rapidly growing body of knowledge in PSC research. The key methodology involves constructing a domain-specific knowledge graph (Perovskite-KG) from 1,517 research papers, creating two datasets (Perovskite-Chat and Perovskite-Reasoning) using a multi-agent framework, and developing two specialized LLMs (Perovskite-Chat-LLM and Perovskite-Reasoning-LLM). Primary results show Perovskite-Chat-LLM achieved a perplexity of 2.97, a Rouge-L score of 41.25, and an LLM-Judge score of 2.97 on the Perovskite QA dataset, significantly outperforming baseline models. The principal implication for AI practitioners is that this system offers tools for enhanced literature review, experimental design, and complex problem-solving in PSC research, demonstrating how domain-specific knowledge can be integrated with LLMs to improve performance in scientific tasks. |
| Pre-training Auto-regressive Robotic Models with 4D Representations (Read more on arXiv or HuggingFace) | trevordarrell, zitengj0618, gbiamby, yuvansharma, NdtSoCool | ARM4R pre-trains robotic models using 4D representations from human videos, enhancing transfer learning for robotic control. The main research objective is to develop a robotic model pre-training approach that leverages low-level 4D representations from human video data to improve performance on robotic manipulation tasks. The key methodology involves training an auto-regressive model in three stages: pre-training on human videos for 3D point track prediction, fine-tuning on robot videos for 3D point tracking, and fine-tuning for robotic control. The method achieves an average success rate of 59.47% on 12 RLBench simulation tasks, surpassing PerAct (55.33%). The model with 4d representations enables AI practitioners to improve sim2real transfer, cross-robot generalization, and performance in robotic control tasks by pre-training on unlabeled human video data. |
| Multilingual Encoder Knows more than You Realize: Shared Weights Pretraining for Extremely Low-Resource Languages (Read more on arXiv or HuggingFace) | XU Han, Jianing Liu, Guixian Xu, Ziyin Zhang, Zeli Su | XLM-SWCM is a novel framework for adapting multilingual encoders to text generation in extremely low-resource languages by sharing weights between the encoder and decoder. The main research objective is to develop an effective text generation model for extremely low-resource languages, specifically Chinese minority languages, where existing multilingual models perform poorly. The key methodology involves a weight-sharing mechanism between the encoder and decoder, interleaving weights from a pretrained multilingual encoder (CINO, a variant of XLM-R) with randomly initialized weights in the decoder. The primary result is that XLM-SWCM outperforms mBART-CM by 198.8% in F1-score on text summarization and also outperfromed the larger MC2-LLaMA 13B in cross-lingual settings. AI practitioners can adapt pre-trained multilingual encoders to text generation tasks in extremely low-resource settings more effectively using this weight-sharing framework, significantly improving performance even with limited data. |
Papers for 2025-02-18
| Title | Authors | Summary |
|---|---|---|
| Learning Getting-Up Policies for Real-World Humanoid Robots (Read more on arXiv or HuggingFace) | Saurabh Gupta, Zixuan Chen, Xialin He, RunpeiDong | The paper introduces HUMANUP, a learning framework for training humanoid robots to get up from various lying positions on diverse terrains. The main research objective is to develop a controller that enables humanoid robots to autonomously recover from falls in real-world settings. The key methodology is a two-stage reinforcement learning approach with a curriculum, where Stage I discovers a getting-up trajectory and Stage II refines it into a deployable, robust policy via imitation learning and control regularization. The primary results show that the learned policy enables a Unitree G1 robot to get up from supine poses with a 78.3% success rate on varied terrains, outperforming the robot’s built-in controller. The principal implication is that this framework provides AI practitioners a method to train robust fall recovery policies for humanoid robots, enhancing their real-world deployability by making robots more resilient. |
| Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention (Read more on arXiv or HuggingFace) | Liang Zhao, Junyu Luo, Damai Dai, Huazuo Gao, Jingyang Yuan | The paper introduces NSA, a natively trainable sparse attention mechanism for efficient long-context modeling in large language models. The main research objective is to develop a sparse attention mechanism that improves computational efficiency during both training and inference while maintaining or exceeding the performance of full attention. The key methodology involves a dynamic hierarchical sparse strategy combining coarse-grained token compression with fine-grained token selection, alongside hardware-aligned optimizations for modern GPUs. Results show that NSA achieves up to 9.0x forward and 6.0x backward propagation speedup on 64k-length sequences compared to Full Attention, and outperforms Full Attention on average across general benchmarks (average score of 0.456 vs 0.443). For AI practitioners, NSA provides a method to train and deploy long-context language models with significantly reduced computational cost and improved performance, particularly on tasks requiring long-range dependencies. |
| ReLearn: Unlearning via Learning for Large Language Models (Read more on arXiv or HuggingFace) | Sendong Zhao, Liming Yang, Ningyuan Zhao, Haoming Xu, Ningyu | ReLearn is a new method for unlearning in large language models that uses data augmentation and positive optimization, addressing limitations of reverse optimization methods. The main research objective is to develop an unlearning method that effectively removes targeted knowledge while preserving model performance, linguistic coherence, and robustness against attacks. ReLearn employs data augmentation with diverse question variations and fine-tuning on synthesized non-sensitive data, along with a comprehensive evaluation framework including Knowledge Forgetting Rate (KFR), Knowledge Retention Rate (KRR), and Linguistic Score (LS). The primary result is that ReLearn achieved a KFR of 0.85 on both KnowUnDo and TOFU datasets while maintaining a high KRR (0.74 on KnowUnDo and 0.89 on TOFU) and preserving linguistic abilities. AI practitioners can utilize ReLearn as an alternative to reverse optimization-based unlearning, providing a method to balance knowledge removal with the preservation of model utility and robustness in applications requiring privacy or copyright compliance. |
| SWE-Lancer: Can Frontier LLMs Earn $1 Million from Real-World Freelance Software Engineering? (Read more on arXiv or HuggingFace) | Johannes Heidecke, Tejal Patwardhan, Michele Wang, Samuel Miserendino | SWE-Lancer is a benchmark of over 1,400 real-world freelance software engineering tasks from Upwork, valued at $1 million USD, to evaluate large language models’ (LLMs) coding and managerial capabilities. The main research objective is to assess whether frontier LLMs can successfully complete real-world freelance software engineering tasks and earn substantial income. The key methodology involves evaluating LLMs on two task types: Individual Contributor (IC) SWE tasks, graded via human-verified end-to-end tests, and SWE Manager tasks, assessed by comparing model choices to those of original engineering managers. Primary results show that the best-performing model, Claude 3.5 Sonnet, achieves 26.2% success on IC SWE tasks and 44.9% on SWE Management tasks on the Diamond set, earning $208,050 out of a possible $500,800. Principal implication for AI practitioners is that while frontier LLMs demonstrate some capability in real-world software engineering scenarios, significant improvement is needed for reliable, autonomous deployment in freelance work. |
| HermesFlow: Seamlessly Closing the Gap in Multimodal Understanding and Generation (Read more on arXiv or HuggingFace) | Minghao Xu, Chenming Shang, Ye Tian, Ling Yang, comin | HermesFlow is a framework designed to reduce the performance disparity between multimodal understanding and generation in Multimodal Large Language Models (MLLMs). The main research objective is to close the gap between the understanding and generative capabilities of MLLMs. The key methodology used is Pair-DPO, which leverages homologous preference data for both understanding and generation, combined with self-play iterative optimization. The primary results show that HermesFlow achieves an understanding score of 0.533 and a generation score of 0.497, reducing the gap to 0.036, compared to the baseline Show-o’s gap of 0.087. For AI practitioners, HermesFlow provides a general alignment framework that demonstrably closes the gap between multimodal understanding and generation tasks within existing MLLM architectures. |
| SURGE: On the Potential of Large Language Models as General-Purpose Surrogate Code Executors (Read more on arXiv or HuggingFace) | Siqiao Huang, zcliang22, Bohan22 | This paper introduces SURGE, a benchmark for evaluating large language models (LLMs) as general-purpose surrogate code executors. The main research objective is to assess whether LLMs can predict the output and behavior of programs across diverse tasks without actually running the code. The methodology involves creating a benchmark (SURGE) with eight distinct code execution aspects, evaluating various open-source and proprietary LLMs, and conducting a scaling study. A key finding is that Claude-3.5-Sonnet achieves an average accuracy of 34.31% across all subsets in the zero-shot setting. The principal implication for AI practitioners is that while LLMs show some capability in predicting code execution, there are still limitations in their ability to serve as general-purpose surrogate code executors, especially for time-consuming computations. |
| Diffusion-Sharpening: Fine-tuning Diffusion Models with Denoising Trajectory Sharpening (Read more on arXiv or HuggingFace) | Mengdi Wang, Yunhai Tong, Ling Yang, Ye Tian, comin | Diffusion-Sharpening fine-tunes diffusion models by optimizing sampling trajectories using a path integral framework, enhancing downstream alignment. The main research objective is to improve diffusion model alignment with user preferences by optimizing the entire sampling trajectory, overcoming limitations of single-timestep optimization. The key methodology, Diffusion-Sharpening, uses a path integral framework to select optimal trajectories during training and leverages reward feedback, implementing this via SFT and RLHF approaches. Primary results show that RLHF Diffusion-Sharpening achieves a CLIP score of 0.338, outperforming baseline SDXL and other methods. The principal implication is that AI practitioners can achieve superior training and inference efficiency, along with better alignment to diverse metrics, by using trajectory-level optimization for diffusion model fine-tuning. |
| I Think, Therefore I Diffuse: Enabling Multimodal In-Context Reasoning in Diffusion Models (Read more on arXiv or HuggingFace) | Runtao Liu, Hanrong Ye, Guocheng Qian, Kuan-Chieh Wang, Mifucius | Here’s a concise summary of the research paper, adhering strictly to the guidelines provided: ThinkDiff aligns vision-language models (VLMs) with diffusion models to enable multimodal in-context reasoning in image generation. The main research objective is to empower text-to-image diffusion models with multimodal in-context understanding and reasoning capabilities. The key methodology is aligning VLMs with the decoder of an encoder-decoder large language model (LLM) through a proxy task of vision-language training, leveraging the shared input feature space between the LLM decoder and diffusion decoders. The primary result is that ThinkDiff significantly improves accuracy on the CoBSAT benchmark for multimodal in-context reasoning generation, achieving 46.3% accuracy compared to the previous 19.2%, with only 5 hours of training on 4 A100 GPUs. Principal implication for AI practioners: transfer the multimodal capabilities of VLM without complex reasoning datasets for in-context reasoning tasks, enhancing image generation from diffusion models. |
| SAFE-SQL: Self-Augmented In-Context Learning with Fine-grained Example Selection for Text-to-SQL (Read more on arXiv or HuggingFace) | Hwanhee Lee, Byeongjeong Kim, Ingeol Baek, Jimin Lee | SAFE-SQL is a framework that improves Text-to-SQL performance by using large language models (LLMs) to generate and filter synthetic examples for in-context learning. The main research objective is to enhance Text-to-SQL accuracy in an unsupervised manner, particularly in complex or unseen scenarios, without additional fine-tuning. The key methodology involves schema linking, LLM-based example generation, relevance scoring (embedding similarity, keyword/structural alignment, reasoning path validity), and threshold-based filtering. Primary results show SAFE-SQL achieved 87.9% execution accuracy on the Spider development set, outperforming zero-shot and few-shot methods, especially in hard and extra hard categories. The principal implication for AI practitioners is that using self-augmented, fine-grained example selection with LLMs can significantly improve the accuracy and robustness of Text-to-SQL systems without requiring additional model training or relying on predefined training sets. |
| CRANE: Reasoning with constrained LLM generation (Read more on arXiv or HuggingFace) | Gagandeep Singh, Sasa Misailovic, Shubham Ugare, Tarun Suresh, Debangshu Banerjee | Constrained LLM generation can reduce reasoning abilities, but augmenting output grammars with reasoning rules can preserve it. The main research questions are whether LLMs truly lose reasoning capabilities under constrained decoding and how to reduce syntax errors while preserving unconstrained reasoning. The key methodology is a reasoning-augmented constrained decoding algorithm (CRANE) that alternates between unconstrained generation for reasoning and constrained generation for structurally correct outputs, supported by theoretical analysis of LLM expressivity. CRANE significantly outperforms state-of-the-art constrained decoding strategies and unconstrained decoding, showing up to a 10% accuracy improvement on the GSM-symbolic and FOLIO benchmarks. AI practitioners can use CRANE to improve the accuracy and syntactic correctness of LLM outputs in tasks requiring formal constraints, such as code generation and symbolic reasoning. |
| Intuitive physics understanding emerges from self-supervised pretraining on natural videos (Read more on arXiv or HuggingFace) | Laurent Najman, Adrien Bardes, Mahmoud Assran, Nicolas Ballas, Quentin Garrido | V-JEPA, a video joint embedding predictive architecture, demonstrates an understanding of intuitive physics when pretrained on natural videos. The main research objective was to investigate the emergence of intuitive physics understanding in deep neural networks trained to predict masked regions in natural videos. Researchers leveraged the violation-of-expectation framework and compared video prediction models in a learned representation space with pixel-space prediction and multimodal large language models. A V-JEPA model trained on natural videos achieved 98% zero-shot accuracy on the IntPhys benchmark. AI practitioners can apply the principle of joint learning of abstract representation space with sensory input prediction, as a robust objective for acquiring intuitive physics understanding in AI models, challenging the reliance on core knowledge. |
| Cuckoo: An IE Free Rider Hatched by Massive Nutrition in LLM’s Nest (Read more on arXiv or HuggingFace) | Jingbo Shang, Feng Yao, Zilong Wang, Letian Peng | Cuckoo is a novel information extraction (IE) model that leverages large language model (LLM) resources for pre-training via a new paradigm called Next Tokens Extraction (NTE). The main research objective is to demonstrate that IE models can be effectively pre-trained using the same data and a similar paradigm as LLMs, overcoming data scarcity limitations in traditional IE pre-training. The key methodology is converting next token prediction in LLMs to next token extraction (NTE) using BIO tags, applied to 102.6M instances derived from the C4 and TuluV3 datasets. Cuckoo outperforms existing pre-trained IE models in few-shot settings, achieving a 70.63 average F1 score across six basic IE tasks, surpassing baselines significantly. AI practitioners can leverage the NTE paradigm to train versatile and efficient IE models using readily available LLM pre-training resources, avoiding expensive manual annotation and enabling adaptation to a variety of IE tasks. |
| Dyve: Thinking Fast and Slow for Dynamic Process Verification (Read more on arXiv or HuggingFace) | Qiang Xu, Xiangyu Wen, Zhijian Xu, Zeju Li, Jianyuan1 | Dyve is a dynamic process verifier that enhances reasoning error detection in large language models by integrating fast and slow thinking. The main research objective is to improve the accuracy and efficiency of process verification in large language models’ reasoning. The key methodology is a dual-system approach, adaptively applying “System 1” (fast, token-level) and “System 2” (slow, comprehensive) verification, supported by step-wise consensus-filtered process supervision using Monte Carlo estimation, LLM-as-a-Judge, and specialized reasoning models. Dyve achieved an F1 score of 68.5 on the GSM8K subset of ProcessBench, outperforming existing process-based verifiers. AI practitioners can use Dyve’s dual-system approach for more reliable and efficient process verification in LLM-based reasoning systems, as it offers superior error detection to traditional process-based methods. |
| PhysReason: A Comprehensive Benchmark towards Physics-Based Reasoning (Read more on arXiv or HuggingFace) | Jiaxing Huang, Yanrui Wu, Yuxuan Dong, Xinyu Zhang, ChengyouJia | PhysReason is a new benchmark for evaluating physics-based reasoning capabilities of large language models (LLMs). The main research objective is to create a comprehensive benchmark to assess LLMs’ ability to solve physics problems requiring multi-step reasoning and application of physics theorems. The methodology involves compiling 1,200 physics problems categorized by difficulty and knowledge/reasoning type, and proposing the Physics Solution Auto Scoring Framework (PSAS) for evaluation. Primary results showed that even top-performing models like Deepseek-R1 achieved less than 60% on answer-level evaluation, with performance dropping from 75.11% on knowledge questions to 31.95% on hard problems. Principal implication for AI practitioners: the benchmark highlights limitations of current LLMs and can help to improve future models on tasks for physics-based reasoning and applications such as robotics. |
| System Message Generation for User Preferences using Open-Source Models (Read more on arXiv or HuggingFace) | Teakgyu Hong, Dawoon Jung, Minsoo Khang, Jungho Cho, Minbyul Jeong | SYSGEN, a data construction pipeline, generates system messages and aligned assistant responses for large language models using open-source models. The main research objective is to address the scarcity and license restrictions of existing datasets with system messages by automatically generating diverse, instruction-aligned system messages. The key methodology involves a four-phase pipeline: generating system messages with eight key functionalities, filtering mis-specified tags, verifying functionalities using an LLM-as-a-judge approach, and generating new, aligned assistant responses. Training on SYSGEN data improved model alignment, with LLaMA-3.1-8B-instruct and Phi-4 models achieving +0.9 and +0.13 absolute improvements, respectively, on the Multifacet benchmark. AI practitioners can leverage SYSGEN to enhance model alignment with user instructions and preferences while minimizing performance degradation on unseen benchmarks and avoiding licensing issues related to training data. |
| video-SALMONN-o1: Reasoning-enhanced Audio-visual Large Language Model (Read more on arXiv or HuggingFace) | Yixuan Li, Changli Tang, Jimin Zhuang, Yudong Yang, Guangzhi Sun | video-SALMONN-01 is an open-source audio-visual large language model designed for enhanced reasoning in general video understanding tasks. The main research objective is to improve the reasoning capabilities of audio-visual LLMs for general video understanding, beyond the existing focus on mathematical problems and visual graphical inputs. The key methodology involves developing a reasoning-intensive dataset with step-by-step solutions, proposing process direct preference optimization (pDPO) for step-level reward modeling, and introducing RivaBench, a new video understanding benchmark. Primary results show that video-SALMONN-01 achieves 3-8% accuracy improvements over the LLaVA-OneVision baseline across different video reasoning benchmarks, and pDPO achieves 6-8% improvements compared to the supervised fine-tuning model on RivaBench. AI practitioners can utilize video-SALMONN-01 and the pDPO method for building applications requiring advanced audio-visual reasoning, such as complex video comprehension and synthetic video detection. |
| Building A Proof-Oriented Programmer That Is 64% Better Than GPT-4o Under Data Scarsity (Read more on arXiv or HuggingFace) | Tianran Sun, Justin Wang, Dylan Zhang | This paper introduces PoPilot, a fine-tuned language model designed to address data scarcity in proof-oriented programming with F. The main research objective is to improve language models’ performance on project-level proof generation and repair in F under data-scarce conditions. The key methodology involves synthetic data augmentation, creating new proof-oriented programming problems, incorporating diverse coding data, and generating repair data within existing repositories. The primary result shows that the 14B parameter model, PoPilot, outperforms GPT-4o in project-level proof-oriented programming by a 64% relative margin. AI practitioners can leverage the proposed synthetic data generation strategies to create specialized verification assistants capable of both synthesizing and repairing proofs to reduce the cost of adaptation of language model. |
| MagicArticulate: Make Your 3D Models Articulation-Ready (Read more on arXiv or HuggingFace) | Yiwen Chen, Fan Yang, Xiu Li, Jianfeng Zhang, chaoyue7 | MagicArticulate is a framework that automatically converts static 3D models into animation-ready assets with skeletons and skinning weights. The main research objective is to develop a scalable method for automatically generating articulation-ready 3D models, addressing the limitations of manual annotation and existing template-based or template-free approaches. The key methodology involves a two-stage pipeline: an auto-regressive transformer for skeleton generation formulated as a sequence modeling problem, followed by a functional diffusion process for skinning weight prediction that incorporates volumetric geodesic distance priors. The method achieves a Chamfer Distance (CD-J2J) of 2.586 on the Articulation-XL dataset for skeleton generation, outperforming existing methods. For AI practitioners, MagicArticulate provides a scalable solution to automatically rig 3D models, significantly reducing the manual effort required for animation content creation and potentially accelerating the development of animation pipelines. |
| Talk Structurally, Act Hierarchically: A Collaborative Framework for LLM Multi-Agent Systems (Read more on arXiv or HuggingFace) | Shingo Takamatsu, Briti Gangopadhyay, Wei-Yao Wang, Sota Moriyama, Zhao Wang | i) The paper introduces TalkHier, a novel framework for LLM Multi-Agent (LLM-MA) systems designed to improve communication and refinement in complex collaborative tasks. ii) The research aims to address challenges in managing communication and refinement among agents in LLM-MA systems. iii) The methodology involves a structured communication protocol and a hierarchical refinement system. iv) TalkHier achieves 88.38% accuracy on the MMLU benchmark when built on GPT40, outperforming inference scaling models and open-source multi-agent models. v) The principal implication for AI practitioners is a new standard for LLM-MA systems, providing a more effective, adaptable, and collaborative framework. |
| One Example Shown, Many Concepts Known! Counterexample-Driven Conceptual Reasoning in Mathematical LLMs (Read more on arXiv or HuggingFace) | Xinnian Liang, Zhikun Xu, Haojing Huang, Jiayi Kuang, Yinghui Li | This paper introduces COUNTERMATH, a new benchmark for evaluating counterexample-driven conceptual reasoning in mathematical Large Language Models (LLMs). The main research objective is to assess and enhance LLMs’ ability to understand mathematical concepts through counterexample-driven proofs, moving beyond reliance on “drill-based” learning. The key methodology involves creating a dataset of 1,216 university-level mathematical statement-rationale pairs from textbooks and developing a data engineering framework for automatically acquiring training data. Primary results show that even advanced LLMs like OpenAI o1 achieve a relatively low F1 score (60.1) on COUNTERMATH, and a fine-tuned model with only 1,025 training samples significantly outperformed baseline models. The principal implication for AI practitioners is that strengthening LLMs’ counterexample-driven reasoning is crucial for improving their overall mathematical capabilities, and this work provides a benchmark and methodology to pursue this. |
| Better Embeddings with Coupled Adam (Read more on arXiv or HuggingFace) | Tobias Stollenwerk, flxst | The paper introduces Coupled Adam, a modification of the Adam optimizer, to address the anisotropy problem in language model embeddings. The main research question is whether the second moment in the Adam optimizer contributes to anisotropic word embeddings in language models and how this can be mitigated. The key methodology involves analyzing the embedding update vectors under SGD and Adam, proposing a modified Adam optimizer (“Coupled Adam”) that averages the second moment across vocabulary items, and empirically evaluating its impact on embedding quality and model performance. Primary results show Coupled Adam improves embedding isotropy significantly, achieving values above 0.90 in most small-scale experiments, and enhances upstream/downstream performance on sufficiently large datasets. For AI practitioners, using Coupled Adam instead of standard Adam can improve the quality of word embeddings and boost model performance, particularly for large language models. |
| Towards Data-Efficient Pretraining for Atomic Property Prediction (Read more on arXiv or HuggingFace) | Bernard Ghanem, Yasir Ghunaim, hammh0a | This paper investigates data-efficient pretraining for atomic property prediction, showing that strategic dataset selection can match or surpass large-scale pretraining with significantly reduced computational cost. The main research objective is to determine if pretraining on a smaller, task-relevant dataset can achieve comparable or superior performance to large-scale pretraining in atomic property prediction. The key methodology introduces the Chemical Similarity Index (CSI), a metric inspired by Fréchet Inception Distance, to quantify the alignment between upstream pretraining datasets and downstream tasks, and uses this to select pretraining data. A primary result is that models pretrained on the ANI-1x dataset (using the CSI for selection) achieved a Mean Absolute Error (MAE) of 5.4 on rMD17, outperforming JMP-S (MAE of 6.7) with 24 times less computational budget. Principal implication for AI practitioners is that strategic selection of pretraining data based on task relevance, assessed using metrics like CSI, can achieve competitive performance with significantly reduced computational resources in atomic property prediction, favoring quality over quantity. |
| Large Language Models and Mathematical Reasoning Failures (Read more on arXiv or HuggingFace) | birgermoell, jboye | This paper evaluates the mathematical reasoning capabilities of large language models (LLMs) using newly constructed word problems and identifies common failure modes. The main research question is: How good are LLMs at mathematical reasoning when evaluated on both answer correctness and solution steps? The key methodology involved creating a dataset of 50 high-school-level mathematical word problems and manually assessing the answers and solutions provided by eight LLMs, including Mixtral, Llama, Gemini, and GPT-4o. The primary result was that the o1 model achieved the highest accuracy, correctly solving 37 out of 50 problems, while all models exhibited errors in spatial reasoning, strategic planning, and arithmetic. The principal implication for AI practitioners is the need to evaluate LLMs’ reasoning processes, not just their final answers, to avoid overestimating their problem-solving proficiency. |
| Language Complexity Measurement as a Noisy Zero-Shot Proxy for Evaluating LLM Performance (Read more on arXiv or HuggingFace) | jboye, birgermoell | This paper evaluates the capability of Large Language Models (LLMs) to measure language complexity as a proxy for general LLM performance. The main research objective is to examine the performance of state-of-the-art LLMs on computing the LIX readability metric and performing dependency parsing to calculate Average Dependency Distance (ADD). The methodology involves evaluating six LLMs using Swedish essays, comparing their LIX and ADD computations against ground truth values, and correlating these with MMLU benchmark scores. A primary result is a strong significant correlation of -0.875 (p=0.026) between the models’ accuracy in computing LIX and their MMLU performance. For AI practitioners, language complexity measurement abilities, specifically LIX computation, can serve as a practical, noisy zero-shot proxy for assessing general LLM capabilities, without needing extensive benchmarking datasets. |
Papers for 2025-02-17
| Title | Authors | Summary |
|---|---|---|
| Region-Adaptive Sampling for Diffusion Transformers (Read more on arXiv or HuggingFace) | Lili Qiu, Yiqi Zhang, Chengruidong Zhang, Yifan Yang, Ziming Liu | Region-adaptive sampling (RAS) improves the efficiency of Diffusion Transformers (DiTs) by dynamically adjusting sampling ratios across image regions. The main objective is to accelerate the sampling process of DiTs without significant quality degradation by focusing computational resources on semantically meaningful regions. RAS identifies “focus” regions in each sampling step using output noise from the previous step, updating only these, and caches the rest, based on attention continuity. RAS achieves speedups of up to 2.36x and 2.51x on Stable Diffusion 3 and Lumina-Next-T2I, respectively, with minimal generation quality degradation. AI practitioners can use RAS to significantly improve the sampling speed of Diffusion Transformers, facilitating real-time applications that require high-quality image generation. |
| Step-Video-T2V Technical Report: The Practice, Challenges, and Future of Video Foundation Model (Read more on arXiv or HuggingFace) | Nan Duan, Liangyu Chen, Kun Yan, Haoyang Huang, Guoqing Ma | i) Step-Video-T2V, a 30B parameter text-to-video model, achieves state-of-the-art results via a novel architecture and training strategy. ii) The research objective is to develop a high-performance and high-quality text-to-video generation model surpassing existing open-source and commercial engines. iii) The methodology involves a deep compression Video-VAE, a DiT with 3D full attention trained using Flow Matching, and a video-based DPO for visual quality enhancement. iv) Evaluated on Step-Video-T2V-Eval, Step-Video-T2V demonstrates state-of-the-art performance with 16x16 spatial and 8x temporal compression ratios while generating videos up to 204 frames. v) AI practitioners can leverage Step-Video-T2V as a strong baseline for further innovations in video foundation models, particularly in improving motion dynamics, aesthetics, and content consistency. |
| ZeroBench: An Impossible Visual Benchmark for Contemporary Large Multimodal Models (Read more on arXiv or HuggingFace) | Samuel Roberts, Akash Gupta, Ansh Sharma, Mohammad Reza Taesiri, Jonathan Roberts | ZeroBench is a new visual reasoning benchmark of 100 questions designed to be impossible for current large multimodal models (LMMs). The main research objective is to create a lightweight yet challenging visual benchmark to evaluate and differentiate the capabilities of LMMs. The methodology involves manually curating and reviewing a set of diverse, multi-step visual reasoning questions, and then adversarially filtering them based on the performance of 20 contemporary LMMs. The primary result is that all evaluated LMMs scored 0.0% on the main questions of ZeroBench, although they achieved non-zero scores on the easier sub-questions, such as 24.30% pass@1 by Claude 3.5 Sonnet v2. The principle implication is that this benchmark highlights limitations to assist in the development of improved LMMs. |
| Large Language Diffusion Models (Read more on arXiv or HuggingFace) | Jingyang Ou, Xiaolu Zhang, Zebin You, Fengqi Zhu, Shen Nie | LLaDA, a diffusion model trained from scratch, achieves performance comparable to autoregressive LLMs like LLaMA3 8B. The main research question is whether diffusion models can achieve the capabilities of large language models (LLMs) without relying on the autoregressive paradigm. Key methodology used is a masked diffusion model (MDM) trained with a forward data masking process and a reverse process parameterized by a vanilla Transformer to predict masked tokens, optimizing a likelihood bound. Primary result is that LLaDA 8B surpasses LLaMA2 7B on nearly all 15 standard zero/few-shot learning tasks and is on par with LLaMA3 8B, and it achieves a 70.7% accuracy on the GSM8K benchmark. Principal implication is that AI practitioners can explore diffusion models as a viable alternative to autoregressive models for large-scale language modeling, potentially offering advantages in bidirectional context understanding and parallel token generation. |
| MM-RLHF: The Next Step Forward in Multimodal LLM Alignment (Read more on arXiv or HuggingFace) | Peiyan Li, Chaoyou Fu, Haochen Tian, Tao Yu, Yi-Fan Zhang | i) The paper introduces MM-RLHF, a new dataset and methodology for aligning multimodal large language models (MLLMs) with human preferences. ii) The research aims to enhance MLLM capabilities across multiple dimensions by aligning models with human preferences. iii) The methodology includes curating a 120k comparison pair dataset, developing a critique-based reward model, and employing dynamic reward scaling within DPO. iv) Fine-tuning LLaVA-ov-7B with MM-RLHF and the proposed alignment algorithm achieves a 19.5% increase in conversational abilities and a 60% improvement in safety. v) AI practitioners can leverage the MM-RLHF dataset and associated techniques to improve MLLM alignment, leading to safer and more capable multimodal models; the critique based reward model can be used to provide more informative feedback for training. |
| Precise Parameter Localization for Textual Generation in Diffusion Models (Read more on arXiv or HuggingFace) | Adam Dziedzic, Kamil Deja, Franziska Boenisch, Bartosz Cywiński, Łukasz Staniszewski | This research localizes and utilizes the parameters in diffusion models responsible for generating and editing textual content within images. The main research objective is to identify the specific parameters within diffusion models that control the generation of textual content in images. The key methodology involves activation patching of cross and joint attention layers and fine-tuning using Low-Rank Adaptation (LoRA). The primary result is that less than 1% of diffusion models’ parameters (0.61% of Stable Diffusion XL, 0.21% of DeepFloyd IF, and 0.23% of Stable Diffusion 3), specifically within attention layers, are responsible for textual content generation. This implies that AI practitioners can improve text generation in diffusion models, and enable precise text editing by fine-tuning or manipulating only this small subset of parameters, conserving computational resources and preserving overall image generation quality. |
| Diverse Inference and Verification for Advanced Reasoning (Read more on arXiv or HuggingFace) | Yuke Zhang, Seunghwan Hyun, Mao Mao, Gaston Longhitano, Iddo Drori | i) The paper presents a diverse inference approach to improve the performance of Reasoning LLMs on challenging tasks. ii) The research aims to enhance reasoning LLMs’ accuracy on complex benchmarks like IMO combinatorics, ARC puzzles, and HLE questions. iii) Key methods include combining multiple models/methods at test time, verifying solutions automatically, test-time simulations, reinforcement learning, and meta-learning of agent graphs. iv) The approach increases IMO combinatorics accuracy from 33.3% to 77.8%, HLE accuracy from 8% to 37%, and solves 80% of ARC puzzles unsolvable by 948 humans. v) AI practitioners can leverage diverse inference and verification techniques to improve the robustness and accuracy of reasoning LLMs on advanced problem-solving tasks. |
| We Can’t Understand AI Using our Existing Vocabulary (Read more on arXiv or HuggingFace) | Been Kim, Robert Geirhos, John Hewitt | This position paper argues that understanding and controlling AI requires developing new vocabulary (neologisms) to represent concepts unique to machines or humans. The main research objective is to argue for developing neologisms to bridge the communication gap between humans and AI, stemming from their differing conceptualizations of the world. The key methodology used is a conceptual argument supported by a proof-of-concept, “neologism embedding learning,” which trains new word embeddings representing human or machine concepts to control model behavior. The primary results demonstrated that using a “length neologism,” responses that meet the length contraints went from near 0% with regular instructions, to a vast majority of generations, shown in figure 5. The authors presented a new “diversity neologism”, increasing response variety in a number-guessing task. Principal implication for AI practitioners is that creating and incorporating neologisms into prompts can improve control over language model behavior and potentially provide a more precise way to interact with and understand AI systems. |
| AdaPTS: Adapting Univariate Foundation Models to Probabilistic Multivariate Time Series Forecasting (Read more on arXiv or HuggingFace) | Maurizio Filippone, Albert Thomas, Giuseppe Paolo, Vasilii Feofanov, abenechehab | AdaPTS is a framework for adapting pre-trained univariate time series foundation models to probabilistic multivariate forecasting using trainable feature-space transformations. The main research objective is to develop a method for leveraging pre-trained univariate time series foundation models (FMs) for multivariate forecasting tasks while addressing challenges like inter-feature dependencies and uncertainty quantification. The key methodology involves introducing “adapters”—stochastic, invertible feature-space transformations—that project multivariate inputs into a latent space where a frozen, pre-trained univariate FM can be applied independently to each dimension, followed by an inverse transformation. Primary results show that AdaPTS improves the forecasting accuracy of the Moment model in 5 out of 8 considered tasks; for example on the Illness dataset (H=24), the VAE adapter achieved a 15% MSE improvement, reducing it from 2.902 to 2.461. AI practitioners can use AdaPTS as a modular and scalable solution for leveraging existing time series FMs in multivariate contexts, enhancing forecasting performance, and uncertainty quantification without requiring FM fine-tuning. |
| FoNE: Precise Single-Token Number Embeddings via Fourier Features (Read more on arXiv or HuggingFace) | Vatsal Sharan, Robin Jia, Mahdi Soltanolkotabi, Deqing Fu, Tianyi Zhou | FoNE introduces a novel method to represent numbers as single tokens in large language models using Fourier features. The main research objective is to develop a more precise and efficient number embedding method that overcomes the limitations of traditional subword and digit-wise tokenization in LLMs. FoNE maps numbers directly into the embedding space using their Fourier features, encoding each digit with two embedding dimensions. On 6-digit decimal addition, FoNE requires 64x less data to achieve 99% accuracy than subword and digit-wise embeddings and is the only method that yields 100% accuracy on over 100,000 test examples. The principal implication is that AI practitioners can leverage FoNE to improve LLM performance on number-related tasks, achieving higher accuracy with reduced computational overhead and training data. |
| Jailbreaking to Jailbreak (Read more on arXiv or HuggingFace) | Bijan Varjavand, Robert Vacareanu, Vaughn Robinson, Jeremy Kritz, ZifanScale | This paper introduces “Jailbreaking-to-Jailbreak” (J2), a novel approach where a refusal-trained Large Language Model (LLM) is jailbroken to assist in jailbreaking other LLMs. The main research objective is to evaluate the capability of jailbroken LLMs to act as effective red teamers and to compare their performance against existing automated and human-led red teaming methods. Key methodology involves creating J2 attackers by jailbreaking frontier LLMs through human-crafted prompts, then using these J2 attackers in an iterative, multi-turn red teaming workflow with in-context learning. Primary results show that J2 attackers (specifically Sonnet-3.5 and Gemini-1.5-pro) achieve 93.0% and 91.0% attack success rates (ASRs) respectively against GPT-40 on Harmbench, approaching human-level performance. Principal implication for AI practitioners is that LLM safeguards can be bypassed by leveraging a jailbroken version of an LLM, highlighting a new failure mode and emphasizing the need for enhanced safeguard mechanisms against LLM-assisted jailbreaking. |
| STMA: A Spatio-Temporal Memory Agent for Long-Horizon Embodied Task Planning (Read more on arXiv or HuggingFace) | Shuguang Cui, Zhixin Mai, Ge Wang, Yiming Zhao, Mingcong Lei | The paper introduces the Spatio-Temporal Memory Agent (STMA), a framework designed to enhance task planning and execution in dynamic environments for embodied AI. The main objective is to enable agents to perform long-horizon tasks by improving decision-making and adaptability through integrated spatio-temporal memory. The methodology involves a spatio-temporal memory module, a dynamic knowledge graph for spatial reasoning, and a planner-critic mechanism for iterative strategy refinement. Results from evaluations in the TextWorld environment show STMA achieved a 31.25% improvement in success rate and a 24.7% increase in average score compared to state-of-the-art models. For AI practitioners, STMA offers a new way to approach memory within AI Agents. |
| MRS: A Fast Sampler for Mean Reverting Diffusion based on ODE and SDE Solvers (Read more on arXiv or HuggingFace) | Ge Yang, Le Lu, Hongbo Zhao, Wei Fang, Ao Li | Mean Reverting Sampler (MRS) accelerates sampling for Mean Reverting (MR) Diffusion models. The main research objective is to reduce the sampling NFEs (number of function evaluations) of MR Diffusion, which currently requires hundreds of steps. The methodology involves solving the reverse-time SDE and probability flow ODE associated with MR Diffusion, deriving semi-analytical solutions consisting of an analytical function and a neural network parameterized integral. Primary results demonstrate that the MR Sampler maintains high sampling quality with a speedup of 10 to 20 times across ten different image restoration tasks. Principal implication for AI practitioners is that they can leverage MRS for faster and more efficient controllable generation using MR Diffusion models, making them more practical in applications. |
| V2V-LLM: Vehicle-to-Vehicle Cooperative Autonomous Driving with Multi-Modal Large Language Models (Read more on arXiv or HuggingFace) | Yu-Chiang Frank Wang, Stephen F. Smith, Chien-Yi Wang, Ryo Hachiuma, Hsu-kuang Chiu | i) This paper introduces V2V-LLM, a large language model for cooperative autonomous driving. ii) The research aims to explore the problem of integrating LLMs into cooperative autonomous driving systems to improve safety. iii) The methodology involves creating a new dataset, V2V-QA, and developing a baseline method, V2V-LLM, that fuses perception information from multiple connected autonomous vehicles using scene-level and object-level features. iv) The V2V-LLM outperforms other fusion methods on notable object identification and planning tasks in the V2V-QA dataset, achieving a collision rate of 3.00% compared to 4.57% for the “No Fusion” baseline. v) The primary implication for AI practitioners is the potential of V2V-LLM to serve as a foundation model for cooperative autonomous driving, particularly in scenarios with sensor occlusion. |
| Agentic End-to-End De Novo Protein Design for Tailored Dynamics Using a Language Diffusion Model (Read more on arXiv or HuggingFace) | Markus J. Buehler, Bo Ni | VibeGen is a generative AI framework for de novo protein design conditioned on normal mode vibrations. The main research objective is to develop a model that can generate novel protein sequences that exhibit specified dynamic properties, specifically low-frequency vibrational modes. The key methodology involves an agentic dual-model architecture, comprising a protein designer (PD) based on a protein language diffusion model that generates sequences and a protein predictor (PP) that evaluates their dynamic accuracy. Primary results showed that the generated proteins accurately reproduced prescribed normal mode amplitudes, with a median Pearson correlation coefficient of 0.53 between designed and target vibration profiles across a large test set. Principal implication for AI practitioners is the demonstration of a viable approach for integrating protein dynamics into generative protein design, enabling the creation of biomolecules with targeted motion-based functionalities. |
Papers for 2025-02-14
| Title | Authors | Summary |
|---|---|---|
| InfiniteHiP: Extending Language Model Context Up to 3 Million Tokens on a Single GPU (Read more on arXiv or HuggingFace) | Sung Ju Hwang, Losif63, geonp, gmlwns5176 | InfiniteHiP enables extremely long-context language model inference on a single GPU without significant performance loss. The main research objective is to develop a training-free framework that allows large language models (LLMs) to handle context lengths significantly exceeding their pre-trained limits on a single GPU. The key methodology involves a hierarchical pruning algorithm to optimize key-value (KV) cache, combined with a novel block sparse attention mechanism and dynamic RoPE adjustments. The primary result is that InfiniteHiP achieves a 7.24x speedup in the SGLang framework with only 0.34% of the VRAM used by FlashAttention2, while extending context to 3 million tokens on a single GPU. A Principal implication for AI practitioners, is that it can be a framework of efficient, long context inference that utilizes modularized pruning algorithm. |
| Skrr: Skip and Re-use Text Encoder Layers for Memory Efficient Text-to-Image Generation (Read more on arXiv or HuggingFace) | Se Young Chun, Jae-sun Seo, Wongi Jeong, Agorium | Skrr is a method for reducing text encoder memory usage in text-to-image diffusion models by selectively skipping or reusing layers. The main research question is how to reduce the memory footprint of text encoders in text-to-image (T2I) diffusion models without significantly impacting image quality or text alignment. The key methodology, Skrr, involves two phases: “Skip” identifies and prunes redundant transformer sub-blocks using a T2I diffusion-tailored discrepancy metric and beam search, and “Re-use” recycles remaining layers to mitigate performance loss. Skrr maintains image quality comparable to the original model, and achieves up to 20.4% improvement in GenEval scores at over 40% sparsity. The principal implication for AI practitioners is that Skrr offers an effective strategy for constructing memory-efficient T2I models, which could also help the development and deployment of text-to-image diffusion models, especially in resource-constrained environments. |
| SelfCite: Self-Supervised Alignment for Context Attribution in Large Language Models (Read more on arXiv or HuggingFace) | Hu Xu, Shannon Zejiang Shen, ZhaofengWu, bencw, voidism | SelfCite is a self-supervised framework that aligns large language models (LLMs) to generate accurate, fine-grained citations by leveraging their own probabilities for necessity and sufficiency rewards through context ablation. The main research objective is to improve the accuracy and quality of citations generated by LLMs without relying on annotation processes. The key methodology involves using context ablation to calculate a reward signal based on two metrics, necessity score (probability drop) and sufficiency score (probability hold), and best-of-N sampling to generate better citations. The primary result is that SelfCite significantly improves citation correctness on the LongBench-Cite benchmark, increasing citation F1 up to 5.3 points on the LongBench-Cite benchmark across five long-form question answering tasks. For AI practitioners, SelfCite offers a method to improve citation quality in LLM-generated text without requiring human annotation, potentially leading to more reliable and trustworthy LLM applications. |
| An Open Recipe: Adapting Language-Specific LLMs to a Reasoning Model in One Day via Model Merging (Read more on arXiv or HuggingFace) | Kasima Tharnpipitchai, potsawee, pittawat, kunato | This paper demonstrates a method for enhancing reasoning capabilities in language-specific large language models (LLMs) using model merging and data selection within a limited computational budget. The main research objective is to incorporate the advanced reasoning abilities of a model like DeepSeek R1 into a Thai language-specific LLM while preserving its target language performance. The key methodology involves supervised fine-tuning of the language-specific LLM on a curated dataset, followed by ability-aware model merging with a reasoning-focused LLM, optimizing the merge ratio across layers. A primary result is that the merged model, Typhoon2-R1-70B, achieved 76.5% average performance across all evaluation metrics, 41.6% above Typhoon2 70B Instruct and 12.8% above DeepSeek R1 70B Distill. This approach allows AI practitioners to improve reasoning in low-resource language LLMs efficiently, using publicly available datasets and modest computational resources. |
| Exploring the Potential of Encoder-free Architectures in 3D LMMs (Read more on arXiv or HuggingFace) | delinqu, Tavish9, zhuhaow, Purple1288, IvanTang | This paper investigates encoder-free architectures for 3D Large Multimodal Models (LMMs), demonstrating comparable performance to encoder-based models. The main research objective is to determine if 3D LMMs can effectively function without dedicated 3D encoders, directly integrating 3D understanding capabilities within the Large Language Model (LLM). The key methodology involves proposing LLM-embedded Semantic Encoding during pre-training and Hierarchical Geometry Aggregation during instruction tuning, replacing the traditional 3D encoder with learnable LLM layers and self-supervised losses. The primary result is that the proposed ENEL model, without a 3D encoder, achieved a GPT-4 score of 50.92% on 3D object captioning, which is similar with the state-of-the-art ShapeLLM-13B. The principal implication is that AI practitioners can explore encoder-free 3D LMMs as a potentially more efficient and scalable alternative to encoder-based architectures, potentially simplifying model design and reducing computational overhead. |
| Can this Model Also Recognize Dogs? Zero-Shot Model Search from Weights (Read more on arXiv or HuggingFace) | Yedid Hoshen, Or Nathan, Jonathan Kahana, Eliahu | This paper introduces ProbeLog, a method for retrieving classification models capable of recognizing a specific target concept based on model weights, without access to training data or metadata. The main research question is how to efficiently and accurately search for models in large repositories that can recognize a given concept (e.g., “Dog”) in a zero-shot manner. ProbeLog uses a probing-based approach, computing logit-level descriptors by observing model responses to a fixed set of input probes, and extends this to zero-shot search via text alignment models. The method achieved a top-1 retrieval accuracy of 43.8% on the INet-Hub dataset when searching for models recognizing ImageNet concepts from text prompts. AI practitioners can use ProbeLog to search for suitable pre-trained models based on specific concept recognition capabilities, potentially reducing the need for training or fine-tuning. |
| CoSER: Coordinating LLM-Based Persona Simulation of Established Roles (Read more on arXiv or HuggingFace) | Rui Xu, Xinfeng Yuan, Yifei Zhang, Heng Wang, Xintao Wang | CoSER is a framework for simulating established characters using large language models (LLMs), including a dataset, models, and an evaluation protocol. The main research objective is to address the lack of authentic character datasets and nuanced evaluation methods for simulating established characters with LLMs. The key methodology is given-circumstance acting (GCA), where LLMs sequentially portray multiple characters in book scenes, used for both training and evaluation. Primary results show that CoSER 70B achieves 75.80% and 93.47% accuracy on the InCharacter and LifeChoice benchmarks, respectively, surpassing or matching GPT-4o. The principal implication for AI practitioners is that they can leverage the CoSER dataset and GCA framework to train and evaluate LLMs for more faithful and nuanced role-playing of established characters, improving applications like character chatbots and agents in games. |
| TripoSG: High-Fidelity 3D Shape Synthesis using Large-Scale Rectified Flow Models (Read more on arXiv or HuggingFace) | Yuan Liang, Dehu Wang, Zexiang Liu, Zi-Xin Zou, Yangguang Li | TripoSG is a new image-to-3D generation model that leverages large-scale rectified flow transformers to achieve high-fidelity 3D shape synthesis. The main research objective is to determine the optimal paradigm for generating high-fidelity 3D models with precise alignment to input images. The key methodology involves a large-scale rectified flow transformer trained on 2 million high-quality 3D samples, a hybrid supervised 3D VAE training strategy, and a dedicated data processing pipeline. Primary results show that TripoSG achieves a Normal-FID score of 3.36 when trained on a large-scale dataset with 4096 tokens and a mixture-of-experts model. The model demonstrates that AI practitioners can now utilize large-scale generative techniques to effectively generate detailed, high-fidelity and accurate 3D models from single input images which are consistent with the input. |
| EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agents (Read more on arXiv or HuggingFace) | Cheng Qian, Mark Zhao, Junyu Zhang, Rui Yang, Hanyang81 | EmbodiedBench is a benchmark for evaluating vision-driven embodied agents based on multi-modal large language models (MLLMs). Main research question or objective: How do existing MLLMs perform as vision-driven embodied agents across a variety of tasks and capabilities, and what are their limitations? Key methodology used: Developed a benchmark (EMBODIEDBENCH) with 1,128 testing instances across four environments, hierarchical action levels (high-level and low-level), and six capability-oriented subsets, then evaluated 13 proprietary and open-source MLLMs using a unified agent framework. Primary results: MLLMs excel at high-level tasks but struggle with low-level manipulation; the best model, GPT-4o, scored only 28.9% on average across all tasks in the benchmark, and performance degrades by 40%-70% when vision input is removed in low-level tasks. Principal implication for AI practitioners: AI practitioners should focus on improving MLLMs’ low-level manipulation, long-horizon planning and use additional approaches for leveraging visual input for high-level embodied tasks since the best model performs poorly in low-level tasks. |
| Typhoon T1: An Open Thai Reasoning Model (Read more on arXiv or HuggingFace) | Kunat Pipatanakul, Kasima Tharnpipitchai, Potsawee Manakul, pittawat | Typhoon T1 is an open-source Thai reasoning model built on a large language model, demonstrating a method for developing reasoning capabilities in low-resource languages. The primary research objective was to develop a Thai reasoning model and investigate effective strategies for its creation, including thinking formats and data composition. The key methodology involved supervised fine-tuning of a pre-trained language model (Typhoon 2 3B Instruct) using synthetically generated datasets with structured, semi-structured, and unstructured reasoning chains. A primary result was that the structured thinking format achieved a GSM8K score of 62.02, outperforming unstructured and semi-structured formats. The principal implication for AI practitioners is that supervised fine-tuning with structured synthetic data can effectively create reasoning models, particularly in low-resource languages, providing a viable alternative to reinforcement learning. |
| Logical Reasoning in Large Language Models: A Survey (Read more on arXiv or HuggingFace) | Chaoli Zhang, Mengru Ding, Hanmeng Liu, ruoxining, HarryFu | This survey synthesizes advancements in logical reasoning within large language models (LLMs), covering paradigms, benchmarks, enhancement methods, and future directions. The main research objective is to provide a comprehensive overview of logical reasoning capabilities in LLMs, focusing on formal symbolic logic rather than general heuristic approaches. The key methodology involves a literature review analyzing existing capabilities across deductive, inductive, abductive, and analogical reasoning, as well as assessing strategies like data-centric tuning, reinforcement learning, and neuro-symbolic approaches. A primary result is that while GPT-4 outperforms ChatGPT on benchmarks like LogiQA and ReClor, both models struggle with out-of-distribution tasks. The principal implication for AI practitioners is the need for hybrid architectures and improved evaluation frameworks that stress-test robustness and generalization in logical reasoning, moving beyond simple accuracy metrics to assess consistency and explainability. |
| MME-CoT: Benchmarking Chain-of-Thought in Large Multimodal Models for Reasoning Quality, Robustness, and Efficiency (Read more on arXiv or HuggingFace) | Yu Qi, Yanwei Li, Ziyu Guo, Renrui Zhang, CaraJ | MME-CoT is a benchmark for evaluating Chain-of-Thought (CoT) reasoning in Large Multimodal Models (LMMs), assessing quality, robustness, and efficiency. The main research objective is to investigate to what extent and how CoT reasoning benefits multimodal challenges in LMMs. Researchers curated a dataset with six domains and proposed novel metrics that meticulously examines LMMs reasoning quality, robustness and efficiency at a fine-grained level. The evaluation reveals that Kimi k1.5 achieved the best CoT quality with 64.2 F1-score, surpassing GPT-4o, and CoT prompting often degrades LMM performance on perception-heavy tasks. For AI practitioners, the results provide insights into the strengths and weaknesses of applying CoT to LMMs, especially highlighting that careful consideration is needed when employing CoT in tasks requiring strong perceptual capabilities. |
| CoT-Valve: Length-Compressible Chain-of-Thought Tuning (Read more on arXiv or HuggingFace) | Xinchao Wang, Gongfan Fang, Runpeng Yu, Guangnian Wan, Xinyin Ma | CoT-Valve introduces a method for tuning language models to generate reasoning chains of controllable lengths, improving efficiency and adaptability. The main research objective is to enable a single model to dynamically adjust the length of its Chain-of-Thought (CoT) reasoning based on task difficulty. The key methodology involves identifying and manipulating a direction in the parameter space (using LoRA) that controls CoT length, along with a “MixChain” dataset for training. A primary result is that on GSM8K, the QwQ-32B-Preview model reduced reasoning chains from 741 to 225 tokens with a minor performance drop (95.07% to 94.92%). Principal implication for AI practioners is that it enables more efficient inference by allowing models to use shorter reasoning paths for simpler tasks, which can improve the cost-effectiveness of reasoning-based application. |
| SQuARE: Sequential Question Answering Reasoning Engine for Enhanced Chain-of-Thought in Large Language Models (Read more on arXiv or HuggingFace) | Moshe Wasserblat, Gad Markovits, Moshe Berchansky, danf | SQuARE is a prompting technique that improves large language model reasoning by generating and answering sub-questions before addressing the main query. The main research objective is to assess if decomposing queries into iterative steps via self-interrogation enhances the reasoning capabilities of LLMs. The key methodology is prompting LLMs (Llama 3 and GPT-4o) to generate and resolve multiple auxiliary question-answer pairs before answering the original question, across multiple QA datasets (TriviaQA, HotpotQA, ASQA). Primary results show that SQuARE improves performance on TriviaQA by 6.5% over Retrieval-Augmented Generation (RAG) using the Llama-3.2 3B model. For AI practitioners, SQuARE presents a method for improving response accuracy in reasoning tasks by systematically decomposing questions, particularly beneficial for smaller-scale models. |
| mmE5: Improving Multimodal Multilingual Embeddings via High-quality Synthetic Data (Read more on arXiv or HuggingFace) | Ziliang Zhao, Yutao Zhu, Nan Yang, Liang Wang, Haon-Chen | mmE5 enhances multimodal multilingual embeddings through a novel synthetic data generation framework. The research objective is to improve multimodal embedding performance by addressing the scarcity of high-quality labeled multimodal data. The methodology involves synthesizing datasets using an MLLM, guided by principles of broad scope, robust cross-modal alignment, and high fidelity, incorporating deep thinking, self-evaluation, and refinement. mmE5 achieves a state-of-the-art average score of 58.6 on the MMEB benchmark in a zero-shot setting, surpassing previous methods. AI practitioners can leverage mmE5’s synthetic data generation approach to create more robust and generalizable multimodal embedding models, particularly in multilingual contexts. |
| The Stochastic Parrot on LLM’s Shoulder: A Summative Assessment of Physical Concept Understanding (Read more on arXiv or HuggingFace) | Shunchi Zhang, Tsz Ting Chung, Junjie Wu, Lemao Liu, Mo Yu | The paper introduces PHYSICO, a benchmark to evaluate large language models’ (LLMs) understanding of physical concepts, revealing significant gaps compared to human performance. The primary research objective is to investigate whether LLMs truly understand physical concepts or merely act as “stochastic parrots.” The key methodology is a summative assessment using grid-format inputs to represent physical phenomena, and comparing LLM performance with human performance across various subtasks. Results indicate that state-of-the-art LLMs, like GPT-4, perform perfectly on low-level tasks(>95% accuracy) but lag behind humans on high-level tasks (~40% less in accuracy) . For AI practitioners, the principal implication is that LLMs still lack robust physical concept understanding beyond memorization, suggesting a need for new methods to improve their reasoning ability. |
| DexTrack: Towards Generalizable Neural Tracking Control for Dexterous Manipulation from Human References (Read more on arXiv or HuggingFace) | Li Yi, Yuzhe Qin, Qianwei Han, Jianibieke Adalibieke, Xueyi Liu | DexTrack is a neural tracking controller that learns to manipulate objects with a robotic hand by following human-provided kinematic references. The main research objective is to develop a generalizable neural tracking controller for dexterous manipulation that can mimic human-object interaction trajectories. The key methodology involves iteratively training the controller with reinforcement and imitation learning, using a homotopy optimization method to mine high-quality robot tracking demonstrations from human references. The primary results show that DexTrack achieves over a 10% improvement in success rates compared to leading baselines in both simulation and real-world evaluations. AI practitioners can leverage DexTrack’s approach of combining imitation learning with high-quality demonstrations to create versatile and robust controllers for complex robotic manipulation tasks. |
| 3CAD: A Large-Scale Real-World 3C Product Dataset for Unsupervised Anomaly (Read more on arXiv or HuggingFace) | Yuanwei Ma, Wenbo Guo, Hanyang Sun, Peng Xing, enquan2022 | 3CAD, a large-scale real-world dataset for unsupervised anomaly detection in 3C products, is introduced along with a coarse-to-fine detection paradigm. The main research objective is to create a challenging benchmark dataset of 3C product defects and develop an effective unsupervised anomaly detection method. The key methodology, CFRG, combines knowledge distillation, recovery guidance, and a segmentation network for coarse-to-fine localization of anomalies. CFRG achieves 93.4% AUROC, 86.5% AUPRO, and 82.0% AP on the 3CAD dataset. The principal implication for practitioners is the 3CAD dataset and CFRG model provide a challenging benchmark and an effective baseline for unsupervised anomaly detection in real-world 3C product manufacturing. |
Papers for 2025-02-13
| Title | Authors | Summary |
|---|---|---|
| TextAtlas5M: A Large-scale Dataset for Dense Text Image Generation (Read more on arXiv or HuggingFace) | Zhuobai Dong, Weiming Han, Jiawei Zhang, Dongxing Mao, Alex Jinpeng Wang | TextAtlas5M is a large-scale dataset designed for generating images with dense, complex, and long-form text. The main research objective is to address the limitations of existing datasets, which often focus on shorter and simpler text, thereby hindering the development of models capable of generating images with comprehensive textual content. The key methodology involves curating 5 million long-text generated and collected images across diverse data types, including synthetic and real-world images, and creating a human-improved test set (TextAtlasEval) of 3,000 samples across 3 data domains. Primary results include the finding that evaluations demonstrate even advanced proprietary models (e.g., GPT4o with DallE-3) are significantly challenged by TextAtlasEval benchmarks, while showing an even large gap in their open-source counterparts. This dataset and benchmarks provide AI practitioners with a valuable resource for training and evaluating text-conditioned image generation models, specifically focusing on dense and long-form text rendering, thus, advancing the capacity to control visual outputs. |
| Light-A-Video: Training-free Video Relighting via Progressive Light Fusion (Read more on arXiv or HuggingFace) | Pan Zhang, Pengyang Ling, Jiazi Bu, Yujie Zhou, yuhangzang | Light-A-Video is a training-free approach for temporally smooth video relighting that leverages image relighting and video diffusion models. The main research objective is to achieve temporally consistent video relighting without requiring training or optimization, addressing the limitations of existing methods. The key methodology involves a Consistent Light Attention (CLA) module for stable light source generation and a Progressive Light Fusion (PLF) strategy to blend relighted appearances, incorporating motion priors from a video diffusion model. Primary results show that Light-A-Video achieves a FID score of 29.63 while maintaining a temporal consistency CLIP score of 0.9655, superior to baseline methods that apply image relighting frame-by-frame. For AI practitioners, Light-A-Video provides a training-free pipeline for high-quality video relighting, directly applicable with existing image relighting and video diffusion models, enabling zero-shot illumination control of video sequences. |
| BenchMAX: A Comprehensive Multilingual Evaluation Suite for Large Language Models (Read more on arXiv or HuggingFace) | Lei Li, Conghui He, Hanxu Hu, Wenhao Zhu, ggdcr | BenchMAX is a multi-way multilingual evaluation benchmark for assessing advanced capabilities of large language models (LLMs) across 17 languages. The main research objective is to create a benchmark that fairly compares LLM capabilities like instruction following, reasoning, and code generation across diverse languages and script systems. The methodology involves machine-translating English tasks into 16 other languages, followed by independent annotation by three native speakers for each sample and task, and final version selection using a strong LLM. A key finding is that DeepSeek-V3 671B model achieved 84.2% on Math and 47.4 on Science reasoning tasks, respectively. For AI practitioners, BenchMAX provides a platform to evaluate LLM performance across languages to improve their multilingual capabilities. |
| CineMaster: A 3D-Aware and Controllable Framework for Cinematic Text-to-Video Generation (Read more on arXiv or HuggingFace) | Huchuan Lu, Xu Jia, Xiaoyu Shi, Yawen Luo, Qinghe Wang | CineMaster is a novel framework for 3D-aware and controllable text-to-video generation, enabling cinematic video creation with precise object placement and camera control. The main research objective is to provide users with 3D-aware and intuitive control over text-to-video generation, similar to the control wielded by film directors. The proposed two-stage framework first allows users to construct 3D scenes and camera movements via an interactive workflow, then uses the generated depth maps, camera trajectories, and object labels to guide a text-to-video diffusion model. CineMaster achieves a mean Intersection over Union (mIoU) of 0.551 and a trajectory deviation (Traj-D) of 66.29, outperforming existing methods in object-box alignment. For AI practitioners, this framework provides a new paradigm for controllable video generation, using a 3D-native approach to enable precise manipulation of scene elements and camera movement directly from textual input and 3D scene descriptions. |
| WorldGUI: Dynamic Testing for Comprehensive Desktop GUI Automation (Read more on arXiv or HuggingFace) | Mike Zheng Shou, Difei Gao, Henry Hengyuan Zhao | WorldGUI introduces a new benchmark and framework, GUI-Thinker, for dynamic testing of desktop GUI automation agents. The main research objective is to evaluate and improve GUI agents’ ability to handle diverse initial states and dynamic environments in real-world computer interactions. The methodology involves creating a benchmark (WorldGUI) with 315 tasks across 10 applications, each with varied starting states, and proposing a critical-thinking-based framework (GUI-Thinker) with five core components: Planner, Planner-Critic, Step-Check, Actor, and Actor-Critic. Experimental results demonstrate that GUI-Thinker significantly outperforms existing agents, with the Claude-3.5 based GUI-thinker achieving a 32.4% overall success rate, and GPT-40 based agent achieving 36.2%, exceeding a baseline by 14.9%. For AI practitioners, WorldGUI provides a robust benchmark to test and enhance agent adaptability in varied, dynamic states. |
| LASP-2: Rethinking Sequence Parallelism for Linear Attention and Its Hybrid (Read more on arXiv or HuggingFace) | Yu Cheng, Xiaoye Qu, Yiran Zhong, landisen, weigao266 | LASP-2 improves sequence parallelism for linear attention in transformers by optimizing communication and computation. The main research objective is to enhance the efficiency of sequence parallelism (SP) when training linear attention transformer models with very long input sequences. The key methodology is LASP-2, which reorganizes the communication-computation workflow to require only one AllGather collective communication on intermediate memory states independent of sequence length, and extends this to hybrid models (LASP-2H). Primary results show that LASP-2 achieves training speed improvements of 15.2% over LASP and 36.6% over Ring Attention on a Linear-Llama3 model with a 2048K sequence length across 64 GPUs. For AI practitioners, LASP-2 provides a more efficient way to train linear attention-based and hybrid transformer models on long sequences, reducing training time and resource consumption. |
| TransMLA: Multi-head Latent Attention Is All You Need (Read more on arXiv or HuggingFace) | Muhan Zhang, Zengwei Yao, fxmeng | TransMLA converts GQA-based language models to MLA-based models, improving expressiveness without increasing KV cache size. The main research objective is to demonstrate that Multi-head Latent Attention (MLA) offers greater expressive power than Group Query Attention (GQA) for the same key-value (KV) cache overhead. The key methodology involves transforming pre-trained GQA models (e.g., LLaMA, Qwen) into equivalent MLA models via low-rank matrix factorization, followed by fine-tuning. Primary results show that the transformed TransMLA model outperformed the original Qwen2.5-7B GQA model on the GSM8K benchmark (87% vs 81%). The main implication is that the TransMLA transformation provides AI practitioners using open-source, GQA-based LLMs with a low cost method to shift to more effective MLA architecture without changes in KV cache size, enhancing performance. |
| Fino1: On the Transferability of Reasoning Enhanced LLMs to Finance (Read more on arXiv or HuggingFace) | Yan Wang, Weipeng Zhou, Lingfei Qian, QianqianXie1994, jiminHuang | The paper evaluates the performance of reasoning-enhanced and general large language models (LLMs) on financial tasks and introduces a new financial reasoning-enhanced model. The main research question is how transferable general-domain reasoning enhancements in LLMs are to the financial domain, and what impact they have across different financial tasks. The methodology involves a comprehensive evaluation of 16 LLMs on three financial datasets (FinQA, DocMath-Simplong, XBRL-Math) encompassing numerical reasoning, tabular interpretation, and financial terminology, followed by developing a model called Fino1. A primary result is that Finol-8B achieved an average score of 61.03 across all datasets, outperforming Llama3.1-8B-Instruct by 10.91 points, with an XBRL-Math score reaching 82.22. The key implication for AI practitioners is that domain-specific fine-tuning with curated financial data, even on a small scale, can significantly improve LLM performance on financial reasoning tasks, surpassing general reasoning enhancements. |
| Ignore the KL Penalty! Boosting Exploration on Critical Tokens to Enhance RL Fine-Tuning (Read more on arXiv or HuggingFace) | lecraquito, Nbeau, supertardigrade | This paper investigates how varying pre-training levels affect language model exploration in reinforcement learning (RL) fine-tuning, and proposes a modified KL penalty to improve exploration. The main research question is how pre-training data distribution impacts exploration efficiency during RL fine-tuning of language models on tasks requiring out-of-distribution generalization. The key methodology involves pre-training a small language model on an arithmetic addition task with varying digit lengths, then fine-tuning it with RL and a modified KL penalty that prioritizes exploration on “critical tokens”. Primary results show the model with the prioritized KL penalty achieved higher accuracy; for example the accuracy during testing with N=7 was higher when the KL penalty took into account the confidence of the old policy. The principal implication for AI practitioners is that adjusting the KL penalty based on pre-trained model certainty on specific tokens can enhance the efficiency of RL fine-tuning, particularly for tasks requiring generalization beyond the pre-training distribution. |
| Distillation Scaling Laws (Read more on arXiv or HuggingFace) | Etai Littwin, Jason Ramapuram, Floris Weers, Amitis Shidani, Dan Busbridge | This paper provides a distillation scaling law that estimates distilled model performance based on compute budget and student/teacher allocation. The main research objective is to determine optimal distillation recipes and understand how to allocate compute resources between teacher and student models to maximize student performance. The key methodology involves a large-scale, controlled study of distillation with students and teachers ranging from 143M to 12.6B parameters, trained on up to 512B tokens, fitting a distillation scaling law to predict student cross-entropy. The primary result is that distillation outperforms supervised pretraining only when the total compute is below a student-size-dependent threshold and a teacher already exists or has uses beyond a single distillation, and student cross-entropy follows a broken power law. The principal implication for AI practitioners is that distillation is beneficial for resource-constrained scenarios or when leveraging existing teachers, guiding optimal model and data scaling during distillation pretraining. |
| SARChat-Bench-2M: A Multi-Task Vision-Language Benchmark for SAR Image Interpretation (Read more on arXiv or HuggingFace) | HaiPeng Wang, Peidong Wang, Sihao Dong, Xiayang Xiao, JimmyMa99 | SARChat-Bench-2M is a new benchmark for evaluating vision-language models (VLMs) on synthetic aperture radar (SAR) image interpretation tasks. The main research objective is to develop a large-scale multimodal dialogue dataset and benchmark for evaluating VLMs’ capabilities in SAR image understanding. The key methodology involves constructing a dataset (SARChat-2M) of 2 million SAR image-text pairs and defining six core tasks (classification, description, counting, localization, recognition, and referring) with specific evaluation metrics. Primary results show that the mPLUG-Owl3-7B model achieved the best performance among tested VLMs, with single-target and multi-target cross-modal identification accuracy rates reaching 99.27% and 99.51%, respectively. The principal implication is that AI practitioners can use SARChat-2M and SARChat-Bench to train, evaluate, and advance VLMs for SAR-specific applications, addressing the existing gap in large-scale, high-quality aligned SAR image-text datasets. |
| LLM Pretraining with Continuous Concepts (Read more on arXiv or HuggingFace) | Andrew Cohen, Jane Yu, Jack Lanchantin, Jihoon Tack, xlxxl | LLM Pretraining with Continuous Concepts introduces a novel pretraining framework, CoCoMix, that combines discrete next-token prediction with continuous concept learning to enhance language models. The main research objective is to investigate whether augmenting the next token prediction objective with explicit concept modeling in a latent space can improve language model pretraining. The key methodology involves extracting concepts from a pretrained sparse autoencoder, predicting these concepts, and mixing them into the model’s hidden state by interleaving them with token hidden representations. The primary results show that CoCoMix achieves comparable performance to standard next-token prediction with 21.5% fewer training tokens on a 1.38B parameter model. For AI practitioners, CoCoMix offers a more sample-efficient pretraining approach, enhances model interpretability and steerability by allowing direct inspection and modification of the predicted concept, and improves performance in weak-to-strong supervision scenarios. |
| Animate Anyone 2: High-Fidelity Character Image Animation with Environment Affordance (Read more on arXiv or HuggingFace) | Dechao Meng, Xin Gao, Zhen Shen, Guangyuan Wang, Hookszdp | Animate Anyone 2 introduces a diffusion-based framework for character image animation that incorporates environmental context to achieve realistic character-environment interactions. The main research objective is to animate characters with environment affordance, ensuring consistent and interactive relationships between the character and its surroundings. The key methodology involves extracting both motion signals and environmental representations from a source video, using a shape-agnostic mask strategy, an object guider with spatial blending for object interactions, and depth-wise pose modulation. Primary results include a superior SSIM score of 0.812 and FVD of 144.65 on the TikTok benchmark, outperforming existing methods in quantitative evaluations. For AI practitioners, this framework offers a robust method to generate high-fidelity character animations that seamlessly integrate with their environments, useful for applications in filmmaking and advertising. |
| NoLiMa: Long-Context Evaluation Beyond Literal Matching (Read more on arXiv or HuggingFace) | Ryan A. Rossi, Trung Bui, Hanieh Deilamsalehy, Franck-Dernoncourt, amodaresi | NOLIMA, a new benchmark, evaluates large language models’ (LLMs) long-context understanding by minimizing literal keyword overlap between questions and answers, emphasizing associative reasoning. Main research question/objective: To assess how well LLMs perform long-context reasoning when they cannot rely on simple literal matches between the question and the context, unlike typical Needle-In-A-Haystack (NIAH) tests. Key methodology: The authors created the NOLIMA benchmark, extending NIAH, where questions and corresponding “needles” (answers) have minimal lexical overlap, requiring models to infer latent associations to locate the needle within a long “haystack” (irrelevant text). They tested 12 LLMs, including GPT-40, and conducted analyses with variations of reasoning complexity, context length, needle placement, and with the presence/absence of literal matching. Primary results: Model performance degraded significantly with increasing context length; at 32K tokens, 10 of the 12 models dropped below 50% of their short-length baseline scores. GPT-4o’s performance decreased from 99.3% baseline to 69.7% at 32K. The presence of literal matches drastically simplified the task, and distractors with literal matches drastically impaired the task. Principal implication for AI practitioners: Current LLMs, even those claiming to support very long contexts, struggle with long-context associative reasoning tasks that lack surface-level (literal) cues, indicating a critical limitation that practitioners should consider when deploying these models in long-context applications. |
| Mediator: Memory-efficient LLM Merging with Less Parameter Conflicts and Uncertainty Based Routing (Read more on arXiv or HuggingFace) | Peijie Dong, Xinglin Pan, Zhenheng Tang, Kunfeng Lai, Dominic789654 | Mediator is a framework for merging multiple fine-tuned large language models (LLMs) efficiently by adaptively averaging layers with minimal parameter conflicts and routing layers with significant conflicts. The main research objective is to develop a method for merging LLMs that minimizes parameter conflicts and system costs while preserving performance across diverse tasks. The key methodology involves quantifying layer-wise parameter conflicts, adaptively averaging layers with low conflict and routing layers with high conflict, employing sparse expert decomposition, and using uncertainty-based routing for out-of-distribution samples. Primary results show that Mediator achieves significant performance improvements over existing methods; e.g. on LLaMA-3.2-8B, it achieved 71.80% average on multiple tasks. The principal implication is that AI practitioners can merge fine-tuned LLMs more efficiently to improve the performance and adaptability while reducing the storage and computational costs compared to maintaining separate models. |
| Next Block Prediction: Video Generation via Semi-Autoregressive Modeling (Read more on arXiv or HuggingFace) | Furu Wei, Xu Sun, Shuming Ma, Shuhuai Ren | The paper proposes a semi-autoregressive framework called Next-Block Prediction (NBP) for video generation that improves upon traditional next-token prediction. The main research objective is to develop a video generation framework that improves spatial dependency modeling and inference efficiency compared to autoregressive next-token prediction models. The key methodology shifts the generation unit from individual tokens to blocks (e.g., rows or frames), using bidirectional attention within each block and predicting multiple tokens in parallel. The NBP model achieved FVD scores of 103.3 on UCF101 and 25.5 on K600, outperforming the vanilla NTP model by an average of 4.4, with an 11x inference speedup. For AI practitioners, this framework provides a more efficient and scalable solution for video generation, maintaining or improving quality while accelerating inference through parallelization. |
| DPO-Shift: Shifting the Distribution of Direct Preference Optimization (Read more on arXiv or HuggingFace) | Xiao Li, Lei Zhao, Qianen Zhang, Feng Jiang, Xiliang Yang | DPO-Shift controllably shifts the distribution of chosen probabilities in Direct Preference Optimization (DPO) to mitigate likelihood displacement. The main research objective is to address the likelihood displacement issue in DPO, where probabilities of chosen responses decrease during training. The key methodology is introducing a parameter function, f(x), added to the rejected reward in the Bradley-Terry model, called DPO-Shift. Experimentally, DPO-Shift with f(x)=0.95 achieved a reward accuracy of 0.743 on the UltraFeedback test set, comparable to DPO’s 0.739, while demonstrably increasing chosen response probability. For AI practioners, DPO-Shift offers a simple, theoretically grounded solution to improve alignment with human preferences by mitigating the likelihood displacement of standard DPO, enabling a trade-off between chosen probability and reward margin. |
| LLM Modules: Knowledge Transfer from a Large to a Small Model using Enhanced Cross-Attention (Read more on arXiv or HuggingFace) | kkolomeitsev | The paper introduces LLM Modules, an architecture for transferring knowledge from a large, frozen language model to a smaller, trainable one using Enhanced Cross-Attention. The main objective is to develop a method that enables smaller models to achieve performance comparable to larger models by leveraging the knowledge of pre-trained large language models (LLMs) without full fine-tuning. The key methodology involves using a frozen Qwen2-1.5B model as a “knowledge source” and a GPT-Neo-125M model as a “generation module,” connected by Enhanced Cross-Attention layers that include linear projections, an adapter block, and a gating mechanism. Training on the Bespoke-Stratos-17k dataset for 15 epochs reduced training loss from 13.8 to 2.3 in the first epoch and to 1.1 in subsequent ones. For AI practitioners, the principal implication is that this modular approach can significantly reduce computational costs associated with training large language models while still achieving substantial performance improvements on specific tasks. |
| MetaSC: Test-Time Safety Specification Optimization for Language Models (Read more on arXiv or HuggingFace) | vicgalle | MetaSC is a framework that optimizes language model safety reasoning at inference time by dynamically updating safety prompts. The research objective is to improve language model safety performance without modifying model weights. The key methodology is a “meta-critique” mechanism that iteratively updates safety prompts (specifications) to adaptively drive the critique and revision process of a self-critique loop. Primary results show that MetaSC significantly improves safety scores compared to fixed system prompts and static self-critique defenses, achieving a safety score of 1.00 on the jailbreak defense task using the Hermes-3-Llama-3.1-405B model. For AI practitioners, MetaSC offers a way to enhance model safety dynamically at inference time, without retraining or fine-tuning. |
Papers for 2025-02-12
| Title | Authors | Summary |
|---|---|---|
| Competitive Programming with Large Reasoning Models (Read more on arXiv or HuggingFace) | Borys Minaev, Andre Saraiva, Alexander Wei, Ahmed El-Kishky, OpenAI | Reinforcement learning significantly improves large language models’ performance on complex coding and reasoning tasks. The main research question is how domain-specific, hand-engineered inference strategies compare to learned approaches in competitive programming. The key methodology involved fine-tuning large language models with reinforcement learning and comparing performance with and without hand-crafted test-time strategies. The primary result was that OpenAI’s o3 model achieved a Codeforces rating of 2724 (99.8th percentile) and an IOI 2024 score of 395.64, surpassing a gold medal threshold without hand-engineered strategies. Scaling general-purpose reinforcement learning presents a robust method toward state-of-the-art AI in reasoning tasks like competitive programming. |
| CodeI/O: Condensing Reasoning Patterns via Code Input-Output Prediction (Read more on arXiv or HuggingFace) | Yu Wu, Runxin Xu, Dejian Yang, Daya Guo, Junlong Li | CODEI/O systematically condenses diverse reasoning patterns in code for improved performance on reasoning tasks. The main research objective is to improve the performance of Large Language Models (LLMs) on a broad range of reasoning tasks by leveraging code-based training data. The key methodology involves transforming raw code files into an input-output prediction format and training LLMs to predict either the output given code and input, or feasible input given code and output, entirely in natural language as Chain-of-Thought rationales. Primary results demonstrate consistent improvements across 14 benchmarks spanning symbolic, scientific, logic, math & numerical, and commonsense reasoning, with CODEI/O++ achieving an average score improvement of 2.9 points, compared to single stage training on Qwen 2.5 Coder 7B. For AI practitioners, this implies that training on code input-output prediction tasks can enhance LLMs’ general reasoning capabilities beyond code-specific applications. |
| Magic 1-For-1: Generating One Minute Video Clips within One Minute (Read more on arXiv or HuggingFace) | Qingyu Yin, Jiantong Zhao, Shitong Shao, Hongwei Yi, Owen777 | Magic 1-For-1 is an efficient video generation model that optimizes memory consumption and inference latency. The main objective is to reduce the computational cost and time required for text-to-video generation while maintaining high video quality. The key methodology involves factorizing the text-to-video task into text-to-image and image-to-video subtasks, alongside model convergence speedup, adversarial step distillation, and parameter sparsification. The primary results show the model can generate 5-second video clips within 3 seconds, and achieves an average score of 0.8134 on a customized VBench, outperforming other models. The principal implication for AI practitioners is that it offers an approach for generating minute-long videos within one minute, optimizing the tradeoff between computational cost and video quality for diffusion-based video generation. |
| Teaching Language Models to Critique via Reinforcement Learning (Read more on arXiv or HuggingFace) | Jingjing Xu, Weichao Mao, Liyu Chen, Jie chen, Zhihui | CTRL trains large language models (LLMs) to provide effective feedback on code, improving iterative code generation. The main research objective is to develop a framework, CTRL, that trains a critic model to generate feedback that maximizes correction performance for a fixed generator model, without human supervision. The methodology uses a two-stage approach: supervised finetuning using execution feedback to synthesize critiques, followed by reinforcement learning with Group Relative Policy Optimization (GRPO) to optimize the critic. The results demonstrate that critics trained with CTRL significantly enhance pass rates, achieving up to 106.1% relative improvement on the CodeContests benchmark when using the same base model for generation and critique, and 23.5% improvement when paired with a better generator. For AI practitioners, CTRL provides a method to create specialized critics that can substantially improve code generation performance through effective, targeted feedback, enabling more autonomous AI systems. |
| Expect the Unexpected: FailSafe Long Context QA for Finance (Read more on arXiv or HuggingFace) | Mateusz Russak, Dmytro Mozolevskyi, Melisa Russak, muayad, kiranr | FailSafeQA, a new long-context financial benchmark, evaluates LLM robustness and context-awareness against variations in human-interface interactions. i) This paper introduces FailSafeQA, a new benchmark for evaluating the robustness of Large Language Models (LLMs) in financial question-answering systems, particularly when dealing with long contexts and imperfect user inputs. ii) The main research objective is to assess the resilience of LLMs against six variations in human-input interactions, such as query failure (misspelled, incomplete and out-of-domain) and context failure (degraded, irrelevant, and missing). iii) The key methodology uses the LLM-as-a-Judge approach with Qwen2.5-72B-Instruct and defines fine-grained rating criteria to calculate Robustness, Context Grounding, and Compliance scores for 24 LLMs. The input consists of truncated 10k filings. iv) The most robust model, OpenAI 03-mini, fabricated information in 41% of tested cases, while Palmyra-Fin-128k-Instruct, the most compliant model, failed robust predictions in 17% of test cases. v) AI practitioners should be aware that high-performing LLMs still have significant room for improvement in terms of balancing robustness and context grounding. Practitioners must carefully assess the trade-off between a model’s ability to handle imperfect inputs and its tendency to hallucinate. |
| LLMs Can Easily Learn to Reason from Demonstrations Structure, not content, is what matters! (Read more on arXiv or HuggingFace) | Xiangxi Mo, Shu Liu, Tyler Griggs, Shiyi Cao, Dacheng Li | Large language models (LLMs) can be efficiently fine-tuned to perform complex reasoning by learning the structural patterns of long chain-of-thought (CoT) demonstrations. The main research question is how to effectively elicit Long CoT reasoning capabilities in LLMs and what aspects of training data are most important. The key methodology involved supervised fine-tuning and low-rank adaptation (LoRA) on LLMs, with controlled experiments perturbing either the content or structure of Long CoT training samples. A primary result was that a Qwen2.5-32B-Instruct model achieved 56.7% accuracy on AIME 2024 after fine-tuning with only 17k Long CoT samples. AI practitioners can elicit strong reasoning performance in LLMs with relatively small, structurally sound datasets, without needing perfect accuracy in the content of individual reasoning steps. |
| Éclair – Extracting Content and Layout with Integrated Reading Order for Documents (Read more on arXiv or HuggingFace) | Lukas Voegtle, Ilia Karmanov, jseppanen, katerynaCh, amalad | ÉCLAIR, a multi-modal large language model (MLLM), extracts structured text, bounding boxes, and semantic classes from documents in integrated reading order. The main research objective is to develop a general-purpose text-extraction tool capable of processing diverse document types and extracting formatted text, spatial information, and semantic class labels simultaneously. The key methodology involves a transformer encoder-decoder architecture with a ViT-like encoder and an autoregressive decoder, pre-trained on a newly generated arXiv-5M dataset and fine-tuned on diverse public datasets. The primary results include achieving state-of-the-art accuracy on the new DROBS benchmark with a 0.937 Counting F1 score and outperforming other methods on established benchmarks. The principal implication for AI practitioners is that ÉCLAIR provides a new model for document OCR, enabling the extraction of more structured data from documents. |
| CAD-Editor: A Locate-then-Infill Framework with Automated Training Data Synthesis for Text-Based CAD Editing (Read more on arXiv or HuggingFace) | Jiang Bian, Qi Liu, Yu Yuan, ShizhaoSun | CAD-Editor is a framework for automatically modifying CAD models based on textual instructions, using an automated data synthesis pipeline and a locate-then-infill approach. The main research objective is to develop a system for text-based editing of CAD models, addressing the lack of support for text-based control in existing design variation methods and the absence of consideration for existing CAD models as constraints. The methodology involves generating synthetic training data using design variation models and LVLMs and decomposing the task into locating regions for modification and infilling those regions with LLMs. Primary results show that CAD-Editor achieves a 95.6% Valid Ratio and a 0.27 Directional CLIP Score, outperforming baseline methods in generation validity, text-CAD alignment, and overall quality. AI practitioners can leverage the proposed framework and data synthesis pipeline to enable more intuitive and efficient CAD model editing through natural language instructions, accelerating the design workflow. |
| Enhance-A-Video: Better Generated Video for Free (Read more on arXiv or HuggingFace) | Wenqi Shao, Kaipeng Zhang, Mengzhao Chen, Xuanlei Zhao, Yang Luo | Enhance-A-Video is a training-free method to improve the temporal consistency and visual quality of diffusion transformer (DiT)-based video generation. The main research objective is to develop a method to enhance the coherence and quality of DiT-based generated videos without retraining or fine-tuning. The key methodology involves introducing a “Enhance Block” that calculates a Cross-Frame Intensity (CFI) from temporal attention maps and uses an “enhance temperature” parameter to scale and integrate this CFI, thereby strengthening cross-frame correlations. User studies demonstrated that models incorporating Enhance-A-Video were preferred across metrics including temporal consistency, prompt-video consistency, and overall visual quality, and VBench scores consistently improved across all tested models. AI practitioners can integrate this plug-and-play method into existing DiT-based video generation frameworks to improve video quality at minimal computational cost, without any retraining or fine tuning of models. |
| NatureLM: Deciphering the Language of Nature for Scientific Discovery (Read more on arXiv or HuggingFace) | Chuan Cao, Liang He, Shufang Xie, Peiran Jin, Yingce Xia | NatureLM is a sequence-based science foundation model designed for scientific discovery across multiple domains. Main research question or objective: To develop a unified, versatile model capable of handling various scientific applications, including generation and optimization, across multiple scientific domains using a sequence-based approach. Key methodology used: A Transformer decoder architecture pre-trained on 143 billion tokens from multiple scientific domains (small molecules, proteins, DNA, RNA, materials, and text), followed by post-training with instruction-response pairs. Primary results: NatureLM (8x7B) achieved state-of-the-art performance in retrosynthesis (71.9% top-1 accuracy on USPTO-50K) and SMILES-to-IUPAC translation (0.607 top-5 accuracy), significantly outperforming general-purpose foundation models. Principal implication for AI practitioners: Practitioners can utilize NatureLM as a foundation model for diverse scientific tasks, particularly where cross-domain interactions and sequence-based representations are crucial, potentially accelerating scientific discovery through a generalist model approach. |
| Hephaestus: Improving Fundamental Agent Capabilities of Large Language Models through Continual Pre-Training (Read more on arXiv or HuggingFace) | Kewei Cheng, Xin Liu, Haoming Jiang, Jingfeng Yang, yczhuang | Hephaestus introduces a continual pre-training method to enhance the fundamental capabilities of LLM-based agents. Main research question or objective: How can continual pre-training on a large-scale, agent-oriented corpus improve the API function calling, intrinsic reasoning, and environmental feedback adaptation capabilities of large language models? Key methodology used: A two-stage continual pre-training framework on the Hephaestus-Forge corpus (103B tokens, 76,537 APIs), leveraging scaling law experiments to optimize data mixing ratios, followed by instruction fine-tuning. Primary results: Hephaestus-8B outperforms LLAMA-3-8B by 9.6% and rivals commercial LLMs on three agent benchmarks, achieves comparable performance with GPT-3.5-turbo, excelling particularly in complex multi-turn tasks (BFCL-v3). Principal implication for AI practitioners: Continual pre-training with a well-curated, agent-specific corpus like Hephaestus-Forge can significantly enhance fundamental agent capabilities of open-source LLMs, bridging the performance gap with commercial models and providing a more robust and generalizable foundation for LLM-based agent development. |
| Forget What You Know about LLMs Evaluations - LLMs are Like a Chameleon (Read more on arXiv or HuggingFace) | Seffi Cohen, Lior Rokach, Bracha Shapira, Yehonatan Elisha, Nurit Cohen-Inger | This paper introduces a meta-evaluation framework, Chameleon Benchmark Overfit Detector (C-BOD), to detect overfitting in Large Language Models (LLMs) on benchmark datasets. The central research question is whether LLMs over-rely on benchmark-specific cues, exhibiting surface-level performance rather than true language understanding. The methodology involves systematically perturbing benchmark prompts using a parametric transformation (controlled by parameter µ) and assessing performance changes with statistical significance tests (McNemar’s test). A primary result is that 20 out of 26 tested LLMs showed statistically significant performance degradation on the MMLU benchmark under modest perturbations, with an average accuracy drop of 2.15%. AI practitioners should integrate C-BOD’s perturbation methods into evaluation pipelines to ensure robust generalization and mitigate superficial memorization in LLMs, prioritizing model resilience over high scores on fixed benchmarks. |
| VidCRAFT3: Camera, Object, and Lighting Control for Image-to-Video Generation (Read more on arXiv or HuggingFace) | Hang Xu, Yi Zhu, Yanpeng Zhou, Zimian Peng, Sixiao Zheng | VidCRAFT3 is a novel image-to-video generation framework enabling precise control over camera motion, object motion, and lighting direction. The main research objective is to develop a model that can simultaneously control multiple visual elements (camera motion, object motion, and lighting) in image-to-video generation, overcoming the limitations of existing methods. The key methodology involves a Spatial Triple-Attention Transformer integrating lighting, text, and image features, along with 3D point cloud rendering and trajectory-based motion encoding, and using a three-stage training process. Primary results show the model achieves a CamMC score of 4.07 on the RealEstate10K dataset, outperforming existing methods like CameraCtrl, CamI2V and MotionCtrl. The principal implication is that AI practitioners can use VidCRAFT3 to create high-quality videos with fine-grained and disentangled control over multiple aspects. |
| Retrieval-augmented Large Language Models for Financial Time Series Forecasting (Read more on arXiv or HuggingFace) | Yueru He, Zhengyu Chen, Lingfei Qian, Zihao Jiang, Mengxi Xiao | This paper introduces a retrieval-augmented generation (RAG) framework, FinSeer, for financial time-series forecasting, specifically stock movement prediction. The main research objective is to develop a RAG framework that effectively integrates financial time-series data with large language models (LLMs) to improve stock movement prediction accuracy. The key methodology involves a fine-tuned 1B parameter LLM (StockLLM), a novel candidate selection method using LLM feedback, and a training objective maximizing similarity between queries and historically significant sequences. The RAG framework with FinSeer achieved an 8% higher accuracy on the BIGDATA22 benchmark compared to a general-purpose LLM-feedback-based retriever. For AI practitioners, this framework demonstrates the importance of using dedicated retrieval models designed to process and filter financial time-series data, to improve the performance of the LLMs in financial forecasting tasks. |
| Mask-Enhanced Autoregressive Prediction: Pay Less Attention to Learn More (Read more on arXiv or HuggingFace) | Li Shen, Zhenyu Zhang, Jianjin Li, Zhikai Jia, Xialie Zhuang | Mask-Enhanced Autoregressive Prediction (MEAP) integrates masked language modeling into next-token prediction to improve large language models’ in-context retrieval capabilities without extra computational cost. The main research objective is to enhance LLMs’ ability to retrieve key information and perform long-context reasoning without compromising their fundamental language modeling capabilities. MEAP randomly masks a fraction of input tokens and then performs standard next-token prediction using a decoder-only Transformer. In pre-training, MEAP outperformed NTP on the Needle in a Haystack evaluation by 11% on average using 140B less training token. This demonstrates MEAP’s superior performance in key information retrieval tasks, and thus provides AI practitioners with a more data- and compute-efficient training paradigm for large language models. |
| FocalCodec: Low-Bitrate Speech Coding via Focal Modulation Networks (Read more on arXiv or HuggingFace) | Mirco Ravanelli, Cem Subakan, Francesco Paissan, lucadellalib | FocalCodec is a low-bitrate speech codec based on focal modulation that uses a single binary codebook for compression. The research objective is to develop a speech codec that achieves high compression rates while preserving both semantic and acoustic information for downstream tasks. The key methodology involves a compressor-quantizer-decompressor architecture utilizing focal modulation, binary spherical quantization (BSQ), and a pretrained self-supervised encoder (WavLM). Primary results show that FocalCodec@50 achieves a dWER of 2.18 on the LibriSpeech test-clean set, outperforming several baselines at comparable bitrates. AI practitioners can use FocalCodec as an efficient and low-bitrate option that can be deployed to preserve sufficient semantic and acoustic information for downstream tasks, such as speech resynthesis, voice conversion, or speech enhancement model development. |
| Auditing Prompt Caching in Language Model APIs (Read more on arXiv or HuggingFace) | Percy Liang, Rohith Kuditipudi, Xiang Lisa Li, Chenchen Gu, thashim | Prompt caching in large language model APIs can leak private and proprietary information through timing differences, which can be detected by auditing. The main research objective was to develop and conduct statistical audits to detect prompt caching and determine the level of cache sharing (per-user, per-organization, or global) in real-world LLM API providers. The key methodology was using statistical hypothesis testing on response times from two procedures: one to generate cache hits, and one to generate cache misses, analyzing differences using the two-sample Kolmogorov-Smirnov test. The primary results revealed that prompt caching was detected in 8 out of 17 API providers, with 7 exhibiting global cache sharing across users, where it was detected with an average precision of around 0.8. AI practitioners should be aware of prompt caching implementation details and cache-sharing levels in LLM APIs to mitigate potential privacy leakage, since the caching can be identified from timing data. |
| Gemstones: A Model Suite for Multi-Faceted Scaling Laws (Read more on arXiv or HuggingFace) | Abhinav Bhatele, Siddharth Singh, David Yu Miller, John Kirchenbauer, smcleish | Gemstones provides a dataset of over 4000 transformer checkpoints to study scaling laws across various architectural and training hyperparameters. The main research question is how model design (width, depth) and model selection impact scaling law parameters and interpretations. The key methodology involves training transformers, up to 2 billion parameters, with diverse widths, depths, learning rates, and cooldown schedules, then fitting and analyzing scaling laws on this data. The primary results show scaling law prescriptions are highly sensitive to model selection and fitting procedures; for example, the optimal tokens-per-parameter ratio is slightly higher than that proposed in previous works. The principal implication for AI practitioners is that scaling laws should be approached with awareness for fragility, with a recommendation to err on wider and, surprisingly, over-trained models, especially when considering time optimality. |
| Skill Expansion and Composition in Parameter Space (Read more on arXiv or HuggingFace) | Yixing Lan, Haoyi Niu, Yinan Zheng, Jianxiong Li, LTL07 | i) The paper introduces Parametric Skill Expansion and Composition (PSEC), a framework for iteratively expanding agent capabilities. ii) The research aims to develop an autonomous agent that can efficiently acquire new skills by leveraging prior knowledge and dynamically composing existing skills. iii) PSEC employs parameter-efficient finetuning using Low-Rank Adaptation (LoRA) modules for skill expansion and a context-aware module for skill composition in parameter space. iv) Experiments on D4RL show PSEC demonstrates the superior capacity to efficiently tackle new challenges. v) PSEC provides AI practitioners with a method for continual learning and efficient skill transfer in reinforcement learning agents, mitigating catastrophic forgetting through parameter isolation. |
Papers for 2025-02-11
| Title | Authors | Summary |
|---|---|---|
| SynthDetoxM: Modern LLMs are Few-Shot Parallel Detoxification Data Annotators (Read more on arXiv or HuggingFace) | Alexander Panchenko, tlenusik, memyprokotow, chameleon-lizard, etomoscow | This paper introduces SynthDetoxM, a multilingual synthetic parallel text detoxification dataset, and a framework for generating such data using large language models (LLMs). The main research objective is to address the scarcity of parallel multilingual datasets for training text detoxification models. The key methodology involves few-shot prompting of multiple open-source LLMs to rewrite toxic sentences sourced from existing toxicity datasets across German, French, Spanish, and Russian, followed by a filtering and ranking process. Models trained on the full SynthDetoxM achieved a J score (combining style transfer accuracy, similarity, and fluency) of 0.484, 0.521, and 0.471 on German, Russian and Spanish respectively. The principal implication is that AI practitioners can leverage the proposed framework and the SynthDetoxM dataset to train more effective multilingual text detoxification models, even with limited human-annotated parallel data. |
| Exploring the Limit of Outcome Reward for Learning Mathematical Reasoning (Read more on arXiv or HuggingFace) | Yuzhe Gu, Songyang Gao, Chengqi Lyu, zsytony, ZwwWayne | This paper introduces OREAL, a new reinforcement learning (RL) framework for enhancing mathematical reasoning in large language models (LLMs) using only binary outcome rewards. The main research objective is to push the performance limit achievable through Outcome REwArd-based reinforcement learning (OREAL) for mathematical reasoning tasks. The key methodology involves behavior cloning on positive trajectories from Best-of-N sampling, reward shaping for negative samples, and a token-level reward model for credit assignment. OREAL achieves a 95.0 pass@1 accuracy on MATH-500 with a 32B model, and a 7B model can obtain 94.0 pass@1 accuracy on MATH-500. AI practitioners can utilize OREAL’s techniques to improve LLM performance on mathematical reasoning tasks using readily available binary outcome feedback, emphasizing the importance of policy model initialization and proper training data selection. |
| Can 1B LLM Surpass 405B LLM? Rethinking Compute-Optimal Test-Time Scaling (Read more on arXiv or HuggingFace) | Xiu Li, Jian Zhao, Junqi Gao, iseesaw, RyanLiu112 | This paper investigates compute-optimal test-time scaling (TTS) strategies for Large Language Models (LLMs), demonstrating that smaller LLMs can outperform larger ones with appropriate scaling. The main research question is what is the optimal approach to scaling test-time computation across different policy models, Process Reward Models (PRMs), and problem difficulty levels, and to what extent can it improve performance. The key methodology involves comprehensive experiments on MATH-500 and AIME24 tasks using various LLMs (0.5B to 72B) and PRMs (1.5B to 72B), evaluating different TTS methods like Best-of-N, beam search, and Diverse Verifier Tree Search. The primary results show that a 3B LLM with compute-optimal TTS can surpass a 405B LLM, achieving 75.6% on MATH-500 and 30.0% on AIME24, compared to 71.4% and 23.3% for the 405B model with Chain-of-Thought prompting. The principal implication for AI practitioners is that applying compute-optimal, reward-aware TTS strategies can significantly enhance the reasoning abilities of smaller LLMs, potentially leading to more efficient and effective deployment compared to using much larger models. |
| Lossless Acceleration of Large Language Models with Hierarchical Drafting based on Temporal Locality in Speculative Decoding (Read more on arXiv or HuggingFace) | Soyeong Jeong, Jeongyeon Seo, Sangjin Choi, doubleyyh, zomss | Hierarchy Drafting (HD) accelerates large language model (LLM) inference by organizing token sources into hierarchical databases based on temporal locality and accessing them sequentially during speculative decoding. Main research question or objective: To address the limitations of existing speculative decoding methods, which rely on a single database, require additional fine-tuning or deliver inconsistent acceleration gains. Key methodology used: The proposed method, Hierarchy Drafting (HD), organizes diverse token sources into three databases (context-dependent, model-dependent, and statistics-dependent) based on temporal locality and accesses them sequentially during speculative decoding, starting from the smallest to largest. Primary results: Experiments on Spec-Bench using LLMs with 7B and 13B parameters demonstrate that HD outperforms existing lossless drafting methods, achieving over 1.5x faster inference speed compared to autoregressive decoding when the temperature is 0.0. Principal implication for AI practitioners: AI practitioners can achieve significant and consistent lossless inference acceleration in LLMs without model retraining or modification, using readily accessible data sources, by employing HD, making it suitable for real-world deployment. |
| Show-o Turbo: Towards Accelerated Unified Multimodal Understanding and Generation (Read more on arXiv or HuggingFace) | Yishun Li, Zhenyi Liao, zhijie3, asunalove, UnhurriedDawn | Show-o Turbo accelerates the unified multimodal understanding and generation model Show-o by extending consistency distillation to its multimodal denoising trajectories. The main research question is whether a unified approach exists to enhance the efficiency of Show-o’s inference, which involves denoising image tokens and autoregressively decoding text tokens. The key methodology involves viewing text generation as a denoising process using Jacobi decoding, extending consistency distillation (CD) to multimodal discrete sampling trajectories, and employing trajectory segmentation and curriculum learning. Show-o Turbo achieves a GenEval score of 0.625 at 4 sampling steps without classifier-free guidance (CFG), outperforming the original Show-o with 8 steps and CFG, in text-to-image generation and 1.5 speedup on image-to-text task. AI practitioners can leverage this approach to deploy more efficient multimodal models that achieve significant speedups in both image and text generation tasks with minimal performance trade-offs. |
| Training Language Models for Social Deduction with Multi-Agent Reinforcement Learning (Read more on arXiv or HuggingFace) | Dorsa Sadigh, C. Karen Liu, Warren Xia, bidiptas | Language models are trained to communicate effectively in a multi-agent social deduction game without human demonstrations, enhancing their ability to reason and strategize. The main research objective is to train language models to have productive natural language discussions about their environment, leveraging the agent’s goal for predicting useful information. The methodology decomposes communication into listening and speaking, using a dense reward signal based on imposter prediction and influence on other agents’ beliefs to guide multi-agent reinforcement learning. Crewmate agents trained with the proposed technique achieve double the win rate compared to standard reinforcement learning, illustrating the value of the communication strategy. AI practitioners can utilize the described approach to enable self-improving discussions in multi-agent settings without requiring task-specific human data, potentially broadening the application of language models in cooperative AI. |
| ReasonFlux: Hierarchical LLM Reasoning via Scaling Thought Templates (Read more on arXiv or HuggingFace) | Mengdi Wang, Bin Cui, Zhaochen Yu, Ling Yang | ReasonFlux is a hierarchical LLM reasoning framework that optimizes mathematical reasoning by scaling thought templates. The main research objective is to improve LLMs’ mathematical reasoning capabilities beyond existing models like OpenAI’s o1-preview and DeepSeek V3. The key methodology involves a structured thought template library, hierarchical reinforcement learning on template sequences, and an inference scaling system that adaptively retrieves and applies templates. On the MATH benchmark, ReasonFlux-32B achieves an accuracy of 91.2%, surpassing o1-preview by 6.7%. AI practitioners can leverage ReasonFlux’s hierarchical template-based approach for more efficient and generalizable reasoning in complex problem-solving applications, requiring less computational resources. |
| The Hidden Life of Tokens: Reducing Hallucination of Large Vision-Language Models via Visual Information Steering (Read more on arXiv or HuggingFace) | Zhenting Wang, Di Liu, Yunhe Gao, Haizhou Shi, Zhuowei Li | This paper introduces VISTA, a training-free framework to reduce hallucination in Large Vision-Language Models (LVLMs) by steering token generation with visual information. The main research objective is to investigate and mitigate the phenomenon of LVLMs generating syntactically coherent but visually ungrounded content. The key methodology, VISTA, combines a Visual Steering Vector (VSV) to reinforce visual cues in activation space and Self-Logits Augmentation (SLA) to leverage early-layer activations for semantically meaningful decoding. Primary results show that VISTA reduces hallucination by about 40% on average in open-ended generation tasks, outperforming existing methods across multiple architectures and decoding strategies. The principal implication for AI practitioners is that VISTA provides an efficient, inference-time intervention to improve the visual grounding and reliability of LVLMs without requiring additional training or model modification. |
| Matryoshka Quantization (Read more on arXiv or HuggingFace) | Aditya Kusupati, Prateek Jain, Jeff Dean, Puranjay Datta, Pranav Nair | Matryoshka Quantization (MatQuant) is a multi-scale quantization technique that trains a single model capable of operating at various integer bit-widths. The main research question is whether a single model can be trained to extract multiple accurate lower-precision models, addressing the challenges of accuracy loss in low-precision quantization and the need for maintaining multiple models. The key methodology is Matryoshka Quantization, which jointly optimizes model weights across multiple precision levels (e.g., int8, int4, int2) using shared most significant bits and leveraging the inherent nested structure of integer data types. Primary results show that MatQuant-derived int2 models outperform standard int2 quantization techniques by up to 10% in accuracy, and an int2 FFN-quantized Gemma-2 9B model is more accurate than an int8 FFN-quantized Gemma-2 2B model. The principal implication is that AI practitioners can train and maintain a single quantized model that can be served at different precision levels, offering a spectrum of accuracy-versus-cost options and improving accuracy, especially in very low precision regimes like int2. |
| EVEv2: Improved Baselines for Encoder-Free Vision-Language Models (Read more on arXiv or HuggingFace) | Yueze Wang, Yufeng Cui, Xiaotong Li, Haiwen Diao, PhyscalX | EVEv2.0 is a new family of encoder-free vision-language models (VLMs) that improve upon existing baselines through architectural and training enhancements. The main research objective is to systematically investigate and improve the performance of encoder-free VLMs, addressing challenges like cross-modal interference and visual perception learning from scratch. The key methodology involves a “Divide-and-Conquer” architecture that decomposes the model into modality-specific components within a unified decoder-only framework, along with a progressive training strategy utilizing an enhanced captioning engine. Primary results show that EVEv2.0 achieves 71.4% accuracy on ScienceQA-IMG, outperforming prior encoder-free models, while approaching the performance of encoder-based counterparts with similar capacity, using only 100M publicly available data. The principal implication for AI practitioners is that properly decomposing and associating modalities, combined with a well-designed training strategy, allows for effective optimization of decoder-only VLMs, providing superior data efficiency and strong visual-reasoning capability, and thereby improving performance of large language models. |
| LM2: Large Memory Models (Read more on arXiv or HuggingFace) | Fraser Greenlee, Alex J. Chan, Filippos Christianos, Wenqi Wu, Jikun Kang | LM2 is a memory-augmented Transformer architecture designed to improve long-context reasoning in language models. The main research objective is to address the limitations of standard Transformers in processing long contexts with distributed information, particularly for tasks involving multi-step reasoning and relational argumentation. The key methodology involves integrating a dynamic memory module into the decoder-only Transformer, using cross-attention and gating mechanisms to update and retrieve contextual representations. Experimental results on the BABILong benchmark show LM2 outperforms the memory-augmented RMT model by 37.1% and the baseline Llama-3.2 model by 86.3% on average across tasks. The principal implication for AI practitioners is that incorporating explicit memory modules, as done in LM2, can enhance a Transformer’s ability to handle long-context reasoning tasks without sacrificing performance on general tasks, which has significance for NLP applications. |
| Lumina-Video: Efficient and Flexible Video Generation with Multi-scale Next-DiT (Read more on arXiv or HuggingFace) | Kai Wang, Zhen Li, Yutong Liu, Shicheng Li, Dongyang Liu | Lumina-Video is a novel framework for efficient and flexible video generation based on an enhanced Diffusion Transformer architecture. The main research objective is to address the spatiotemporal complexity and computational challenges of video generation using Diffusion Transformers (DiTs). The key methodology involves a Multi-scale Next-DiT architecture with multiple patch sizes, motion score conditioning, progressive training, and multi-source training. Lumina-Video achieves a total score of 82.94% on the VBench benchmark, demonstrating competitive performance in generating high-quality videos. AI practitioners can leverage Lumina-Video’s Multi-Scale Next-DiT and training strategies to build efficient and flexible video generation models with controllable dynamics. |
| History-Guided Video Diffusion (Read more on arXiv or HuggingFace) | Russ Tedrake, Yilun Du, Max Simchowitz, Boyuan Chen, Kiwhan Song | The paper introduces a video diffusion model, DFoT, and a family of guidance methods, History Guidance (HG), that improve video generation quality and consistency by leveraging variable-length historical frames. The main research question is how to effectively use different portions of video history as a form of guidance for improved video generation. The key methodology involves the Diffusion Forcing Transformer (DFoT), which allows conditioning on flexible history lengths, and History Guidance methods, which combine scores from different history windows and noise levels. A primary result is that DFoT with history guidance achieves a Fréchet Video Distance (FVD) of 170.4 on Kinetics-600, outperforming baselines. AI practitioners can use DFoT and History Guidance to improve the quality, consistency, and length of generated videos, especially for tasks requiring long-term coherence. |
| CustomVideoX: 3D Reference Attention Driven Dynamic Adaptation for Zero-Shot Customized Video Diffusion Transformers (Read more on arXiv or HuggingFace) | Zhen Yang, Jin Wang, Jingxuan Pang, Mushui Liu, D. She | CustomVideoX is a zero-shot personalized video generation framework based on the Video Diffusion Transformer, enhancing video quality and temporal coherence. The main research objective is to develop a method for generating customized videos from a reference image and text prompt, addressing temporal inconsistencies and quality degradation issues. The key methodology involves integrating 3D Reference Attention for direct interaction between reference image and video frames, Time-Aware Attention Bias to modulate reference feature influence, and Entity Region-Aware Enhancement for focused feature injection. Primary results show that CustomVideoX achieves a CLIP-I score of 90.26 and DINO-I score of 91.49 on the VideoBench benchmark, outperforming other methods. AI practitioners can leverage CustomVideoX’s architecture for improved zero-shot personalized video generation, specifically benefiting from the 3D Reference Attention and time-aware mechanisms for better fidelity and consistency. |
| APE: Faster and Longer Context-Augmented Generation via Adaptive Parallel Encoding (Read more on arXiv or HuggingFace) | Beidi Chen, Tianqi Chen, Hanyuezhuohua | APE improves context-augmented generation by enabling faster and longer context processing through adaptive parallel encoding. The main research objective is to address the computational burden and performance degradation of existing context-augmented generation (CAG) techniques when handling multiple, lengthy contexts. The key methodology, Adaptive Parallel Encoding (APE), uses a shared prefix, attention temperature, and scaling factor to align the distribution of parallel encoding with sequential encoding. Results show that APE preserves 98% of sequential encoding performance on RAG tasks while enabling an end-to-end 4.5x speedup by reducing prefilling time by 28x for a 128K-length context. The principal implication for AI practitioners is that APE enables more efficient and scalable deployment of CAG systems, particularly those dealing with long and numerous contexts, by reducing computational costs and improving response times. |
| Efficient-vDiT: Efficient Video Diffusion Transformers With Attention Tile (Read more on arXiv or HuggingFace) | Peiyuan Zhang, Runlong Su, Dacheng Li, zhijie3, foreverpiano | EFFICIENT-VDIT accelerates video diffusion transformers by sparsifying 3D attention and reducing sampling steps. The main research objective is to address the computational inefficiency of 3D full attention diffusion transformers (DiTs) during video generation. The key methodology involves identifying and leveraging a “tile-style” repetitive pattern in 3D attention maps to create sparse attention masks, combined with multi-step consistency distillation. The primary result is that EFFICIENT-VDIT achieves up to a 7.8x speedup on Open-Sora-Plan-1.2 models for 29 and 93 frame video generation with minimal performance degradation on VBench. For AI practitioners, this method provides a way to significantly speed up video generation with 3D DiTs, enabling faster inference and potentially reducing computational costs. |
| MetaChain: A Fully-Automated and Zero-Code Framework for LLM Agents (Read more on arXiv or HuggingFace) | Chao Huang, Tianyu Fan, Jiabin Tang | MetaChain is a framework enabling fully-automated, zero-code development and deployment of LLM agents through natural language alone. The main research question is: Can we enable everyone, regardless of technical background, to build their own LLM agents using natural language alone? The key methodology involves a novel LLM Agent Framework with four components: Agentic System Utilities, LLM-powered Actionable Engine, Self-Managing File System, and Self-Play Agent Customization module, enabling automated agent generation, customization, and workflow optimization. Primary results include ranking #1 among open-source solutions on the GAIA benchmark and achieving 73.51% accuracy on a MultiHop-RAG task. The principal implication for AI practitioners is that MetaChain democratizes agent development, allowing non-programmers to create and customize LLM agents and workflows, potentially accelerating the adoption of agent technology. |
| Steel-LLM:From Scratch to Open Source – A Personal Journey in Building a Chinese-Centric LLM (Read more on arXiv or HuggingFace) | Zhaoxiang Zhang, Shu Li, Qingshui Gu, aaabiao | Steel-LLM is a fully open-source, 1-billion-parameter, Chinese-centric language model developed with limited computational resources. The main objective was to create a high-quality, transparent, and resource-efficient language model, primarily trained on Chinese data, with a small proportion of English. The methodology involved adapting a Qwen-based Transformer architecture with Soft Mixture of Experts and an enhanced Feed-Forward Network, trained using a modified TinyLlama framework on 8 A100/H800 GPUs. The model achieved a CEVAL accuracy of 41.90% and a CMMLU accuracy of 36.08% after supervised finetuning. AI practitioners can use the provided training pipeline, datasets, model architecture, and intermediate checkpoints to develop or extend similar language models with limited resources, facilitating reproducibility and further research. |
| The Curse of Depth in Large Language Models (Read more on arXiv or HuggingFace) | Yefeng Zheng, Lu Yin, Xinyuan Song, Wenfang Sun, pengxiang | The paper introduces “Curse of Depth” in large language models (LLMs), where deeper layers contribute less than expected due to Pre-Layer Normalization (Pre-LN), and proposes LayerNorm Scaling to address it. The main research objective is to identify and rectify the phenomenon where deeper layers in LLMs are less effective, specifically investigating the role of Pre-LN in this issue. The key methodology involves theoretical analysis of Pre-LN’s impact on variance and gradient flow, alongside empirical evaluations via layer pruning experiments and comparisons of different normalization techniques. A primary result is that LayerNorm Scaling reduces perplexity by 1.31 on LLaMA-1B compared to standard Pre-LN. The principal implication for AI practitioners is that applying LayerNorm Scaling, which inversely scales the output of Pre-LN by the square root of the layer depth, can improve LLM performance by enhancing the contribution of deeper layers during training, creating more resource-efficient models. |
| DreamDPO: Aligning Text-to-3D Generation with Human Preferences via Direct Preference Optimization (Read more on arXiv or HuggingFace) | Yi Yang, Hehe Fan, Fan Ma, Xiaobo Xia, Zhenglin Zhou | DreamDPO is an optimization-based framework for text-to-3D generation that aligns 3D content with human preferences through direct preference optimization. The main research objective is to improve the alignment of text-to-3D generated content with human preferences and enhance controllability. The methodology involves constructing pairwise examples, comparing their alignment with human preferences using reward or large multimodal models, and optimizing the 3D representation with a preference-driven loss function. DreamDPO achieved a GPTEval3D overall score of 1203.1, outperforming 13 state-of-the-art methods, including MVDream (1097.7). AI practitioners can utilize DreamDPO to generate higher-quality and more controllable 3D content, moving beyond pointwise quality evaluations by utilizing pairwise comparisons and preference optimization. |
| Dual Caption Preference Optimization for Diffusion Models (Read more on arXiv or HuggingFace) | Bimsara Pathiraja, Shamanthak Hegde, Agneet Chatterjee, Yiran Luo, sahsaeedi | Dual Caption Preference Optimization (DCPO) improves text-to-image diffusion models by using distinct captions for preferred and less preferred images during training. The main research objective is to address the issues of conflict distribution and irrelevant prompts in existing preference optimization methods for diffusion models. The key methodology involves generating distinct captions for preferred and less-preferred images using captioning, perturbation, or hybrid methods, and introducing a modified objective function that leverages these dual captions. Primary results show that DCPO-h outperforms Stable Diffusion 2.1, SFT, Diffusion-DPO, and MaPO, achieving a +0.21 improvement in Pickscore. The principal implication for AI practitioners is that using dual, distinct captions for preferred and less-preferred image pairs during preference optimization can significantly enhance the alignment and performance of diffusion models. |
Papers for 2025-02-10
| Title | Authors | Summary |
|---|---|---|
| VideoRoPE: What Makes for Good Video Rotary Position Embedding? (Read more on arXiv or HuggingFace) | Pan Zhang, Xiaoyi Dong, Xilin Wei, yuhangzang, LiuXR | VideoRoPE introduces a novel rotary position embedding method for video data that outperforms existing methods by preserving spatio-temporal relationships. The main research objective is to identify and address the limitations of existing Rotary Position Embedding (RoPE) methods when applied to video data with complex spatio-temporal structures. The key methodology involves analyzing four essential characteristics (2D/3D structure, frequency allocation, spatial symmetry, temporal index scaling) for effective RoPE adaptation to video and proposing VideoRoPE, which features a 3D structure, low-frequency temporal allocation, diagonal layout, and adjustable temporal spacing. Primary results show that VideoRoPE outperforms previous RoPE variants on various benchmarks, achieving a 12.44% performance improvement over M-ROPE on the Video Retrieval task in both V-NIAH and V-NIAH-D settings. The principal implication for AI practitioners is that VideoRoPE provides a more robust and effective positional encoding scheme for video-based models, enhancing performance in tasks such as video retrieval, understanding, and hallucination reduction. |
| Fast Video Generation with Sliding Tile Attention (Read more on arXiv or HuggingFace) | Ion Stoica, Hangliang Ding, Runlong Su, Peiyuan Zhang, BrianChen1129 | Sliding Tile Attention (STA) accelerates video diffusion models by efficiently computing attention within local spatiotemporal windows. The paper introduces STA to address the high computational cost of 3D full attention in video diffusion transformers (DiTs). STA operates tile-by-tile, utilizing a hardware-aware sliding window design and kernel-level optimizations. STA reduces end-to-end latency of a video DiT (HunyuanVideo) from 945s to 685s without quality degradation, and to 268s with finetuning (0.09% drop on VBench). AI practitioners can deploy STA to significantly reduce inference time for video generation DiTs while maintaining output quality, or trade minimal quality loss for substantial speed gains. |
| AuraFusion360: Augmented Unseen Region Alignment for Reference-based 360° Unbounded Scene Inpainting (Read more on arXiv or HuggingFace) | Jie-Ying Lee, Ying-Huan Chen, Yang-Jung Chen, Chung-Ho Wu, cmhungsteve | AuraFusion360 is a reference-based method for 360° unbounded scene inpainting that removes objects and fills holes in 3D scenes represented by Gaussian Splatting. The main research objective is to achieve high-quality object removal and hole filling in 360° unbounded scenes, maintaining view consistency and geometric accuracy. The methodology introduces depth-aware unseen mask generation, Adaptive Guided Depth Diffusion for initial point placement, and SDEdit-based detail enhancement for multi-view coherence. The method achieves an average PSNR of 17.661 and LPIPS of 0.388 on the 360-USID dataset, outperforming existing methods. AI practitioners can use this method and the provided 360-USID dataset for improved 3D scene inpainting, particularly in applications requiring consistent and accurate object removal in 360° environments. |
| Goku: Flow Based Video Generative Foundation Models (Read more on arXiv or HuggingFace) | Fengda Zhu, Yida Zhang, Yuqi Zhang, Chongjian Ge, ShoufaChen | Goku is a family of rectified flow Transformer models for joint image-and-video generation that achieves industry-leading performance. The main research objective is to develop a state-of-the-art joint image-and-video generation model with industry-leading performance using rectified flow Transformers. The key methodology involves a data curation pipeline, a 3D joint image-video variational autoencoder (VAE), a Transformer architecture with full attention, rectified flow formulation, and infrastructure optimization for large-scale training. Goku achieves 0.76 on GenEval and 83.65 on DPG-Bench for text-to-image generation, and 84.85 on VBench for text-to-video tasks. This work demonstrates a pathway toward industry-grade performance in visual generation, enabling practitioners to build more efficient and high-performing generative models using Rectified Flows. |
| QuEST: Stable Training of LLMs with 1-Bit Weights and Activations (Read more on arXiv or HuggingFace) | Jiale Chen, d-alistarh, mnikdan97, soroushtabesh, BlackSamorez | QuEST introduces a quantization-aware training method for large language models (LLMs) enabling stable training with extremely low-precision weights and activations. The main research objective is to determine the Pareto-optimal frontier for training LLMs with low-bitwidth weights and activations, minimizing representation size while maintaining accuracy. The key methodology, QuEST, combines Hadamard normalization and MSE-optimal fitting for quantization, with a “trust” gradient estimator minimizing the difference between quantized and full-precision gradients. Primary results show stable training of Llama-family models down to 1-bit weights and activations, with 4-bit QuEST models achieving superior accuracy compared to BF16 models almost 4x larger in size. The principal implication for AI practitioners is that QuEST enables training and deploying accurate LLMs at significantly reduced precision and model size, potentially leading to more efficient inference. |
| Agency Is Frame-Dependent (Read more on arXiv or HuggingFace) | Shi Dong, Will Dabney, Michael Bowling, André Barreto, David Abel | i) The paper argues that agency, a system’s capacity to steer outcomes toward a goal, is fundamentally frame-dependent. ii) The main objective is to demonstrate that the attribution of agency to a system is relative to the choice of a reference frame. iii) The methodology involves a philosophical argument, illustrating that the essential properties of agency (individuality, source of action, normativity, adaptivity) are frame-dependent. iv) The paper does not present specific quantitative findings. v) Any basic science of agency requires frame-dependence, impacting how AI practitioners should approach reinforcement learning. |
| FlashVideo:Flowing Fidelity to Detail for Efficient High-Resolution Video Generation (Read more on arXiv or HuggingFace) | Peize Sun, Chongjian Ge, Wenbo Li, Shilong Zhang, ShoufaChen | FlashVideo introduces a two-stage framework for efficient high-resolution text-to-video generation. The research aims to decouple prompt fidelity and visual quality optimization in video generation. It utilizes a two-stage DiT architecture with a large model for low-resolution generation followed by flow matching with a smaller model for high-resolution detail enhancement. FlashVideo achieves a top-tier performance on VBench-Long (82.99 score) with significantly reduced function evaluation time (102.3s for 1080p video generation). The two-stage design allows AI practitioners to preview initial output before committing to full-resolution generation, reducing computational costs and wait times. |
| Linear Correlation in LM’s Compositional Generalization and Hallucination (Read more on arXiv or HuggingFace) | Chengyu Dong, Shibo Hao, Chenyang An, Letian Peng, shangjingbo | i) This paper unveils linear correlations in language models (LMs) during knowledge composition. ii) The research investigates the extent to which linear transformations can approximate the relationships between the output logits of related next token prediction (NTP) tasks. iii) The methodology involves fitting a linear transformation between logits of source and target knowledge prompts using a subset of data, then evaluating the transformation on the remaining data using Pearson correlation. iv) Results indicate that the fitted linear transformation is resilient to fine-tuning, with successful generalization for simultaneous knowledge updates requiring high correlation intensity and transformation precision; in City-Country relationships, 42% of cities learn the top-1 weight with their influenced countries. v) The implication for AI practitioners is the understanding that compositional generalization in LMs relies on linear correlations between vocabulary representations, which can be leveraged for knowledge composition tasks but also may lead to hallucinations when misaligned. |
| Generating Symbolic World Models via Test-time Scaling of Large Language Models (Read more on arXiv or HuggingFace) | Fuxiang Frank Xia, Tim Z. Xiao, Yuhuan Yuan, Zhouliang Yu, zhangysk | i) This paper introduces a test-time scaling approach for generating Planning Domain Definition Language (PDDL) domains using Large Language Models (LLMs). ii) The main objective is to enhance PDDL reasoning in LLMs for generating high-quality PDDL domains without additional training data. iii) The methodology employs a Best-of-N sampling approach followed by iterative refinement using Instance Verbalized Machine Learning (iVML). iv) The method achieves an 85.2% success rate on the NL2Domain task and 71.4% on Prob2Domain with Qwen2.5-Coder-7B, exceeding ol-mini’s performance. v) AI practitioners can leverage this approach to generate symbolic world models for robust planning, particularly in complex domains where existing LLM-based planners struggle. |
| On-device Sora: Enabling Diffusion-Based Text-to-Video Generation for Mobile Devices (Read more on arXiv or HuggingFace) | Yeojin Lee, Jungmin Cheon, Isu Jeong, Kyuhwan Lee, Bosung Kim | On-device Sora is a framework for diffusion-based text-to-video generation that operates efficiently on smartphone-grade devices. The main research objective is to enable efficient and high-quality text-to-video generation on resource-constrained mobile devices, addressing limitations of current diffusion-based video generation models. Key methodologies include Linear Proportional Leap (LPL) to reduce denoising steps, Temporal Dimension Token Merging (TDTM) to minimize token-processing computation, and Concurrent Inference with Dynamic Loading (CI-DL) for efficient model inference. Results demonstrate that On-device Sora generates videos on an iPhone 15 Pro with quality comparable to Open-Sora running on NVIDIA A6000 GPUs, achieving up to 1.94x speedup with LPL. AI practitioners can leverage On-device Sora’s techniques to deploy and accelerate diffusion-based video generation models on mobile and embedded devices, expanding accessibility and enabling on-device applications. |
| CMoE: Fast Carving of Mixture-of-Experts for Efficient LLM Inference (Read more on arXiv or HuggingFace) | Wulong Liu, Xianzhi Yu, Hui-Ling Zhen, Lancheng Zou, Eleven-P | CMoE is a framework that efficiently creates sparse Mixture-of-Experts models from dense large language models (LLMs) for improved inference efficiency. The main objective is to transform dense LLMs into sparse MoE architectures without extensive retraining. The methodology involves grouping feed-forward network (FFN) neurons into shared and routed experts based on activation rates, constructing a training-free routing mechanism using representative neurons, and optional lightweight adaptation. Results show that, with a 25% activation ratio, CMoE achieved 76.59% of the dense model’s accuracy on some downstream benchmarks with lightweight fine-tuning on 2,048 samples. For AI practitioners, CMoE offers a method to deploy LLMs more efficiently in resource-constrained environments by significantly reducing computational overhead while maintaining performance. |
| Step Back to Leap Forward: Self-Backtracking for Boosting Reasoning of Language Models (Read more on arXiv or HuggingFace) | Jie-Jing Shao, Ding-Chu Zhang, Wen-Da Wei, Xuan-Yi Zhu, yangxw | This paper introduces Self-Backtracking, a technique that enables language models to autonomously backtrack during reasoning. The main research objective is to address the limitations of current slow-thinking mechanisms in large language models, specifically inefficient overthinking and over-reliance on auxiliary reward models. The key methodology involves training the model to recognize suboptimal reasoning paths and backtrack to earlier states, using a specialized dataset format and a modified loss function during training, and an inference algorithm combining expansion, backtracking, and selection steps during inference. The primary result shows that Self-Backtracking improves reasoning accuracy on the Countdown task by over 40% compared to optimal-path supervised fine-tuning, using the Llama3.2-1B model. The principal implication for AI practitioners is that integrating self-backtracking into language models can significantly enhance reasoning capabilities and efficiency, and reduce the need for external reward models. |
| Scaling Laws in Patchification: An Image Is Worth 50,176 Tokens And More (Read more on arXiv or HuggingFace) | Yuyin Zhou, Wei Shao, Guoyizhe Wei, Yaodong Yu, Feng Wang | This paper investigates the impact of patchification, an image tokenization method, on the performance of vision models. The main research objective is to examine the information loss caused by the patchification-based compressive encoding paradigm in vision models and how it affects visual understanding. The key methodology involves extensive scaling experiments by varying patch sizes in ViT and Mamba-based architectures across different vision tasks and input scales. The primary result is that model performance consistently improves as patch size decreases, achieving a test accuracy of 84.6% on ImageNet-1k with a base-sized model using a 1x1 patch size (50,176 tokens). The principal implication is that AI practitioners should consider reducing or eliminating spatial compression in vision encoders to improve model accuracy, as computational resources allow. |
| QLIP: Text-Aligned Visual Tokenization Unifies Auto-Regressive Multimodal Understanding and Generation (Read more on arXiv or HuggingFace) | Yuke Zhu, Linxi Fan, Scott Reed, Fuzhao Xue, zhaoyue-zephyrus | QLIP is a visual tokenization method that combines state-of-the-art reconstruction quality with state-of-the-art zero-shot image understanding. The main research objective is to develop a visual tokenizer that excels at both capturing image semantics and reconstructing high-quality visuals for multimodal language modeling. The key methodology involves training a Binary Spherical Quantization (BSQ)-based autoencoder with a contrastive objective for text-image alignment, using a two-stage training process to balance reconstruction and alignment. A primary result is that QLIP-B achieves a zero-shot classification accuracy of 74.3% on ImageNet, while achieving a reconstruction FID of 3.21, comparable to state-of-the-art methods. AI practitioners can use QLIP as a drop-in replacement for visual encoders in existing models like LLaVA or image tokenizers in models like LlamaGen, achieving improved or comparable performance in multimodal understanding and generation tasks. |
| ARR: Question Answering with Large Language Models via Analyzing, Retrieving, and Reasoning (Read more on arXiv or HuggingFace) | Giuseppe Carenini, yuweiyin | ARR is a zero-shot prompting method that improves question-answering (QA) performance of Large Language Models (LLMs) by explicitly guiding them through analyzing, retrieving, and reasoning steps. The main research objective is to evaluate the effectiveness of the ARR prompting method compared to baseline and Chain-of-Thought (CoT) prompting in multiple-choice QA tasks. The key methodology involves comparing the accuracy of LLMs using different trigger sentences representing ARR, baseline (no specific trigger), and zero-shot CoT prompting across ten multiple-choice QA datasets. Primary results show that ARR achieves an average accuracy of 69.58% across all datasets, outperforming the baseline (65.48%) and CoT (68.14%) when using the LLaMA3-8B-Chat model. AI practitioners can leverage the ARR prompting strategy to enhance LLM performance in QA tasks without needing model fine-tuning or few-shot examples, leading to better results in various applications, including information retrieval and decision support. |
Papers for 2025-02-07
| Title | Authors | Summary |
|---|---|---|
| Analyze Feature Flow to Enhance Interpretation and Steering in Language Models (Read more on arXiv or HuggingFace) | Yaroslav Aksenov, kefirski, elephantmipt, dlaptev | This paper introduces a data-free method to track the evolution of features learned by sparse autoencoders across layers of large language models, enabling improved interpretability and steering of model behavior. The main research question is how to systematically map and understand the progression of features discovered by sparse autoencoders across consecutive layers of large language models. The key methodology involves using cosine similarity between decoder weights of SAEs trained on different modules (MLP, attention, residual) and layers to trace feature persistence, transformation, or emergence. The primary results show that deactivating a single predecessor feature causes a greater activation strength drop if that predecessor is in a group with single predecessor: for example, in layer 8, this probability is approximately 0.75, 0.55 and 0.6 for “From RES”, “From MLP” and “From ATT”, respectively . The principal implication for AI practitioners is that this method provides a means for more precise control over model behavior by identifying and manipulating multi-layer feature circuits, offering improvements over single-layer steering approaches. |
| UltraIF: Advancing Instruction Following from the Wild (Read more on arXiv or HuggingFace) | Ning Ding, Li Sheng, ssz1111, ganqu, kkk-an | ULTRAIF is a scalable approach for building LLMs that can follow complex instructions with open-source data by training a composer model to synthesize instructions and evaluation questions. Main research question or objective: How to effectively align open-source LLMs with complex instructions using a scalable approach and open-source data. Key methodology used: Decomposing real-world user prompts into simplified queries, constraints, and evaluation questions; training an “UltraComposer” model to compose constraint-associated prompts with evaluation questions; using the composer to synthesize complex instructions and filter responses based on the evaluation questions. Primary results: ULTRAIF successfully aligns LLaMA-3.1-8B-Base to match the instruct version on 5 instruction-following benchmarks without benchmark-specific data, achieving a score of 69.63 (DRFR) on InfoBench and outperforming comparable baselines. Principal implication for AI practitioners: AI/ML engineers can use ULTRAIF as an effective and scalable method to improve the instruction-following capabilities of LLMs using open-source data, potentially reducing reliance on expensive, proprietary datasets, and simplifying the training and evaluation processes. |
| DynVFX: Augmenting Real Videos with Dynamic Content (Read more on arXiv or HuggingFace) | talidekel, omerbartal, RafailFridman, DanahY | DynVFX augments real-world videos with new dynamic content described by user-provided text instructions. The main research objective is to develop a method for seamlessly integrating synthesized dynamic objects or complex scene effects into existing real-world videos, accounting for camera motion, occlusions, and interactions. The key methodology is a zero-shot, training-free framework leveraging a pre-trained text-to-video diffusion transformer and a Vision Language Model (VLM) for content synthesis and scene understanding, using a novel inference-based method with “Anchor Extended Attention” to manipulate attention features for localization and integration. The primary results show that the proposed method outperforms baselines like SDEdit and LORA fine-tuning, achieving a masked Structural Similarity Index (SSIM) of 0.860 and a CLIP Directional score of 0.311, indicating better original content preservation and edit fidelity. For AI practitioners, this method provides a framework that facilitates generating and harmonizing dynamic video effects without the need for creating and tracking masks, enabling improved video editing and synthesis capabilities using pre-trained diffusion models. |
| Ola: Pushing the Frontiers of Omni-Modal Language Model with Progressive Modality Alignment (Read more on arXiv or HuggingFace) | jiwenlu, WinstonHu, liuziwei7, THUdyh, Zuyan | Ola is an omni-modal language model achieving competitive performance across image, video, and audio understanding using a progressive modality alignment strategy. The main research objective is to develop an omni-modal language model that achieves competitive performance across image, video, and audio understanding compared to specialized single-modality models, while maintaining efficiency. The key methodology is a progressive modality alignment strategy that trains the model sequentially on image-text, then video, and finally audio data, along with a dual-encoder approach for audio input and sentence-wise streaming decoding for speech generation. The model achieves a mean accuracy of 72.6% on the OpenCompass benchmark and 68.4% on the VideoMME benchmark, outperforming existing open-source omni-modal LLMs and many specialized models. The principal implication is that AI practitioners can build more efficient and cost-effective omni-modal models by leveraging progressive modality training, starting with the most distinct modalities, which reduces the cross-modal alignment data demand. |
| MotionLab: Unified Human Motion Generation and Editing via the Motion-Condition-Motion Paradigm (Read more on arXiv or HuggingFace) | De Wen Soh, Na Zhao, zeyuhu, ZiyanGuo | MotionLab is a unified framework for human motion generation and editing that leverages a novel Motion-Condition-Motion paradigm and rectified flows. The main research objective is to determine if human motion generation and editing can be effectively unified within a single framework. The key methodology involves a MotionFlow Transformer with Aligned Rotational Position Encoding, Task Specified Instruction Modulation, and Motion Curriculum Learning for multi-task training. The framework achieved a text-based editing R@1 score of 56.34 on the MotionFix dataset, demonstrating editing capabilities. For AI practitioners, MotionLab provides a versatile framework capable of handling both human motion generation and editing tasks, promoting knowledge sharing and efficiency. |
| Great Models Think Alike and this Undermines AI Oversight (Read more on arXiv or HuggingFace) | AmeyaPrabhu, douwekiela, iaa01, Klingspor, shash42 | This paper studies how model similarity affects AI oversight, finding that greater similarity biases evaluations and reduces gains from training on Language Model (LM) annotations, with model errors becoming more correlated as capabilities increase. The main research question is how model similarity impacts the effectiveness of AI oversight, both in evaluation (LLM-as-a-judge) and training (using LM annotations). The key methodology involves proposing Chance Adjusted Probabilistic Agreement (CAPA), a new metric for LM similarity based on the overlap in model mistakes, and using it to analyze LLM-as-a-judge and training on LM annotation scenarios. Primary results show LLM-as-a-judge scores are significantly correlated with model similarity (average Pearson r=0.84), and gains from weak-to-strong generalization are higher when the supervisor and student models are more dissimilar. For AI practitioners, increasing model similarity poses a risk due to correlated failures, indicating a need for measuring and reporting model similarity and developing methods for training diverse models. |
| Gold-medalist Performance in Solving Olympiad Geometry with AlphaGeometry2 (Read more on arXiv or HuggingFace) | Miroslav Olšák, Trieu H. Trinh, Yuri Chervonyi, lmthang, mmenegali | AlphaGeometry2 achieves gold-medal-level performance in solving Olympiad geometry problems. The main research objective is to improve upon the previous AlphaGeometry system to solve a broader range of, and more difficult, Olympiad geometry problems. Key methodologies include expanding the domain-specific language, optimizing the symbolic deduction engine (DDAR) with a C++ implementation, developing a novel search algorithm (SKEST) that utilizes multiple search trees with knowledge sharing, and employing a larger, Gemini-based language model trained on more diverse synthetic data. AlphaGeometry2 achieves an 84% solve rate on 2000-2024 IMO geometry problems (42 out of 50), compared to 54% for the original AlphaGeometry. AI practitioners can leverage the demonstrated techniques, such as enhanced neuro-symbolic reasoning, knowledge sharing between search agents and improved synthetic data generation, to build more powerful AI systems for complex mathematical reasoning tasks. |
| ScoreFlow: Mastering LLM Agent Workflows via Score-based Preference Optimization (Read more on arXiv or HuggingFace) | Bryon Aragam, Ling Yang, Edify-Kd2024, lightaime, yinjiewang | ScoreFlow is a framework for optimizing multi-agent workflows of large language models (LLMs) using a novel score-based preference optimization method. The main research objective is to develop an automated, adaptive, and cost-efficient framework for generating and optimizing LLM agent workflows, addressing limitations of existing methods like inflexibility and poor scalability. The key methodology involves representing workflows as code, generating multiple workflows per task, evaluating them with quantitative scores, and optimizing the workflow generator using Score-DPO, a variant of direct preference optimization that incorporates evaluation scores. Across six benchmarks, ScoreFlow achieved an 8.2% average improvement over existing baselines. AI practitioners can utilize ScoreFlow to automate and enhance the creation of high-performance, scalable, and adaptable LLM agent workflows, resulting in improved model performance and lower inference costs. |
| MAGA: MAssive Genre-Audience Reformulation to Pretraining Corpus Expansion (Read more on arXiv or HuggingFace) | Chenggang Li, Ke Shen, haoxintong | The paper introduces MAGA, a method for expanding pretraining corpora by reformulating existing text into diverse genres and audience styles using large language models. The main research question is how effective MAGA-generated synthetic data is for expanding pretraining corpus and aiding model scaling under data-constrained scenarios. The key methodology involves a two-stage synthesis process using a 3.3B MoE model to generate multiple genre-audience reformulations of documents, followed by heuristic cleaning. Primary results show that models trained with MAGA-expanded data (MAGA-Mix) achieved consistent improvements across model sizes (134M-1.7B parameters), with a +2.15 average performance gain on the 1.7B model, and substantial gains in TriviaQA (+15.47) and GSM8K (+6.06). For AI practitioners, MAGA offers a scalable method to expand training datasets and improve model performance, particularly when high-quality natural language data is scarce, providing an avenue for model scaling. |
| Llasa: Scaling Train-Time and Inference-Time Compute for Llama-based Speech Synthesis (Read more on arXiv or HuggingFace) | Xinsheng Wang, Chi-Min Chan, Xinfa Zhu, HKUST-Audio, ZhenYe234 | Llasa explores scaling train-time and inference-time compute for Llama-based text-to-speech (TTS) synthesis, demonstrating improvements in naturalness, prosody, and expressiveness. The main research objective is to investigate the effects of scaling both training and inference computation on the performance of a simplified, Llama-based TTS system. The key methodology involves using a single Transformer architecture with a vector quantizer (VQ) codec (X-codec2) and evaluating performance under varying model sizes, training data sizes, and inference-time search strategies (e.g., beam search, best-of-N). Primary results show that increasing training data from 80k to 250k hours improves the mean expert score on Chinese polyphonic characters from below 2.00 to around 2.25; scaling inference compute using a mixed strategy of PRM and ORM achieved higher SIM and kept WER near ground truth, on seed-tts-eval test-hard testset. For AI practitioners, this implies that both train-time and inference-time compute scaling are viable strategies for improving TTS quality, and that inference-time scaling can be a useful approach for balancing competing objectives like speaker similarity and content accuracy. |
| MotionCanvas: Cinematic Shot Design with Controllable Image-to-Video Generation (Read more on arXiv or HuggingFace) | ttwong, aniruddha26398, heiwang1997, cusuh, Doubiiu | MotionCanvas is an image-to-video generation system that enables cinematic shot design with controllable camera and object motions. The main research objective is to develop a method that allows users to intuitively design cinematic video shots from a static image, controlling both camera and object movements in a scene-aware manner. The key methodology involves a Motion Signal Translation module that converts user-specified 3D motion intentions (camera paths, object bounding boxes, point trajectories) into 2D screen-space motion signals (point trajectories, bbox sequences) to condition a video diffusion model. The method achieved a Camera Motion Consistency (CamMC) score of 0.9453 on the RealEstate10K test set. AI practitioners can use MotionCanvas to enhance creative workflows in digital content creation with precise control over camera and object movements in image-to-video generation, avoiding costly 3D-related training data. |
| ChartCitor: Multi-Agent Framework for Fine-Grained Chart Visual Attribution (Read more on arXiv or HuggingFace) | Kanika Goswami, Franck-Dernoncourt, ryanrossi, puneetm | ChartCitor is a multi-agent LLM framework that provides fine-grained bounding box citations for answers generated from chart images. The main research objective is to identify chart elements (e.g., bars, lines) that support factual claims in LLM-generated responses to user questions about charts. The methodology involves orchestrating multiple LLM agents to perform chart-to-table extraction, answer reformulation, table augmentation, evidence retrieval via pre-filtering and re-ranking, and table-to-chart mapping. The primary result shows that ChartCitor achieves an Intersection over Union (IoU) of 27.4, outperforming existing baselines such as direct bounding box decoding and other LLM-based models, by 9-15%. The principal implication is that AI practitioners can enhance the trustworthiness and explainability of chart question-answering systems by using this framework to provide visual evidence for LLM-generated answers, directly linking claims to specific chart components. |
| BOLT: Bootstrap Long Chain-of-Thought in Language Models without Distillation (Read more on arXiv or HuggingFace) | cxiong, yingbozhou, jcxu, hendrydong, bpucla | BOLT is a method to develop long chain-of-thought (LongCoT) reasoning in large language models (LLMs) without knowledge distillation or human annotations. The main research question is whether LLMs can develop LongCoT capabilities from standard instruct models without relying on existing LongCoT models or expensive human annotations. The key methodology is a three-stage process: 1) LongCoT data bootstrapping with in-context learning; 2) LongCoT supervised finetuning; and 3) online training using DPO to refine LongCoT capacities. The method applied to Llama-3.1-70B-Instruct achieved impressive performance, evaluated through MT-Bench and Arena-Hard, showcasing improved reasoning ability. The principal implication is that AI practitioners can develop strong LongCoT reasoning capabilities from existing ShortCoT models and reduce the cost to train the models, thereby making advanced reasoning more accessible without reliance on proprietary models. |
| Beyond Prompt Content: Enhancing LLM Performance via Content-Format Integrated Prompt Optimization (Read more on arXiv or HuggingFace) | Xuan Feng, Qi Chen, Yuanye Liu, lynazhang, Jiahang | The paper introduces Content-Format Integrated Prompt Optimization (CFPO), a method to improve Large Language Model (LLM) performance by jointly optimizing prompt content and format. The main research question is whether integrating prompt content and format optimization can enhance LLM performance compared to content-only optimization methods. The key methodology involves iterative refinement using component-wise content optimization (case-diagnosis, Monte-Carlo sampling) and dynamic format exploration (LLM-assisted format generation, UCT-based selection). Primary results show that CFPO achieves an 8.6% absolute improvement in GSM8K accuracy using the LLaMA-3.1-8B model compared to the baseline prompt (50.03 to 63.38). For AI/ML engineers and data scientists, CFPO highlights that jointly optimizing both prompt content and format presents a practical approach to significantly boosting LLM performance and can be done using only open-source LLMs. |
| PlotGen: Multi-Agent LLM-based Scientific Data Visualization via Multimodal Feedback (Read more on arXiv or HuggingFace) | Ryan Rossi, Puneet Mathur, Kanika Goswami, Franck-Dernoncourt | PlotGen is a multi-agent framework that automates scientific data visualization generation using multimodal feedback for iterative refinement. The main research objective is to automate the creation of precise scientific visualizations from user specifications and raw data, addressing the limitations of current Large Language Models (LLMs) in this area. The key methodology involves orchestrating multiple LLM-based agents, including a Query Planning Agent, a Code Generation Agent, and three feedback agents (Numeric, Lexical, and Visual) that leverage multimodal LLMs for self-reflection. Primary results show that PlotGen outperforms strong baselines, achieving a 4-6% improvement on the MatPlotBench dataset. For AI practitioners, PlotGen provides a framework to improve accuracy and reduce debugging of LLM-generated visualizations. |
| Enhancing Code Generation for Low-Resource Languages: No Silver Bullet (Read more on arXiv or HuggingFace) | gbavota, AML14, Devy1 | This paper investigates methods to improve code generation by Large Language Models (LLMs) for low-resource programming languages, finding no single superior technique across all contexts. The primary research question is: Which techniques are best suited to improve LLM-based code generation capabilities in low-resource programming languages? The study empirically evaluated in-context learning (translation examples, translation rules, few-shot) and fine-tuning (with/without pre-training on code translation) on six LLMs, using the MultiPL-E benchmark for R and Racket. Results show fine-tuning benefits smaller models (e.g., DeepSeek Coder 1B), increasing Racket pass@1 from 7.0% to 18.4% with pre-training & fine-tuning, while in-context learning, specifically with translation examples, generally improves performance for larger models and GitHub Copilot, with deltas over baseline reaching +6.3% in some test cases. AI practitioners should consider model size when boosting performance on low-resource languages, with in-context learning representing a generally effective and low-cost strategy, especially for larger LLMs. |
| Weak-to-Strong Diffusion with Reflection (Read more on arXiv or HuggingFace) | Zeke Xie, Masashi Sugiyama, Lichen Bai | The paper introduces Weak-to-Strong Diffusion (W2SD), a framework that enhances diffusion model inference by leveraging the difference between weak and strong models. The main research objective is to reduce the gap between the learned distribution of diffusion models and the real data distribution. The key methodology involves using a reflective operation that alternates between denoising and inversion, guided by the estimated difference between existing weak and strong models (weak-to-strong difference). Experiments demonstrate W2SD significantly improves human preference, with Juggernaut-XL and W2SD improving the HPSv2 winning rate up to 90% over the original results. AI practitioners can use W2SD as a general-purpose framework to improve the performance of diffusion models by defining appropriate weak-to-strong model pairs, leading to better alignment with real data distributions. |
| Speak Easy: Eliciting Harmful Jailbreaks from LLMs with Simple Interactions (Read more on arXiv or HuggingFace) | Marzyeh Ghassemi, Yik Siu Chan, YuxinXiao, narutatsuri | SPEAK EASY demonstrates that large language models (LLMs) can be jailbroken through simple, everyday human-LLM interactions to produce harmful content. The main research objective is to investigate whether harmful jailbroken responses, both actionable and informative, can be elicited from LLMs through common interaction patterns. The key methodology involves proposing HARMSCORE, a metric for evaluating jailbreak harmfulness, and SPEAK EASY, a framework using multi-step reasoning and multilingual querying to simulate realistic user interactions. Results show that incorporating SPEAK EASY into direct request and jailbreak baselines increased the Attack Success Rate (ASR) of GPT-4o by an average of 0.463 and HARMSCORE by 0.579 across four safety benchmarks. For AI practitioners, this implies that current safety alignment techniques in LLMs are vulnerable to simple, realistic interaction patterns, making careful consideration of such patterns in both red-teaming and defense necessary. |
| PILAF: Optimal Human Preference Sampling for Reward Modeling (Read more on arXiv or HuggingFace) | duanyq, Knykny, Kunhao, RedTachyon, Coolfyz | The paper introduces Policy-Interpolated Learning for Aligned Feedback (PILAF), a novel response sampling strategy for Reinforcement Learning from Human Feedback (RLHF) that aligns preference learning with maximizing underlying oracle reward. The main research question is how to design an optimal sampling scheme for generating response pairs in RLHF to improve sample efficiency and model performance. The key methodology is T-PILAF, a theoretically grounded sampling method generating responses by interpolating the policy and reference models, and its practical variant PILAF which implements this. Primary results show PILAF outperforms baselines in iterative and online Direct Preference Optimization (DPO) settings, achieving a final reward of -9.80 vs -10.16 for Vanilla sampling in the iterative setting, with a 40% reduction in training time. The principal implication is that AI practitioners can use PILAF to improve the efficiency and performance of RLHF by optimizing the data sampling process, resulting in higher rewards and lower divergence from the reference model. |
| Towards Physical Understanding in Video Generation: A 3D Point Regularization Approach (Read more on arXiv or HuggingFace) | ZanyRumata, vidit98, anilkagak2, jlcao2, yunuoch | The paper introduces a video generation framework that incorporates 3D geometry and dynamics by augmenting 2D videos with 3D point trajectories and using them to regularize the video diffusion process. The main research objective is to improve the physical plausibility and temporal consistency of generated videos, especially in contact-rich scenarios. The key methodology involves creating a 3D-aware video dataset (PointVid) by tracking 3D points in videos, fine-tuning a latent diffusion model on this dataset, and regularizing the generation process using 3D point information. Primary results show that, compared to I2VGen-XL, their method has background consistency score improvement of +0.061 on the VBench benchmark, along with other improvements such as better object permanence and more accurate hand-object interactions. For AI practitioners, this means adding a 3D spatial component to the video generation process creates better video quality. |
| Learning Real-World Action-Video Dynamics with Heterogeneous Masked Autoregression (Read more on arXiv or HuggingFace) | Kevin Zhao, endernewton, chaoqi-liu, liruiw | Here’s a summary of the paper following your guidelines: The paper introduces Heterogeneous Masked Autoregression (HMA) for modeling action-conditioned video dynamics in robotics using diverse datasets. The main research objective is to develop a general and efficient model for action-video dynamics across heterogeneous robotic embodiments, domains, and tasks. The key methodology is masked autoregression, which uses a Transformer architecture to predict masked video tokens and actions from heterogeneous datasets, with variants for discrete (VQ tokens) and continuous (soft tokens) video representations. HMA achieves better visual fidelity and controllability than previous models, with a 15x faster inference speed of 22.72 FPS on the presented hardware setup (measured in Table 1). For AI practitioners, HMA offers a framework for building interactive video simulators and generating synthetic data for robot learning, which can have real-time robotic applications. |
Papers for 2025-02-06
| Title | Authors | Summary |
|---|---|---|
| SmolLM2: When Smol Goes Big – Data-Centric Training of a Small Language Model (Read more on arXiv or HuggingFace) | Gabriel Martín Blázquez, Elie Bakouch, Anton Lozhkov, Loubna Ben Allal, lvwerra | SmolLM2 is a 1.7 billion parameter language model trained on 11 trillion tokens to achieve state-of-the-art performance among small language models. The main research objective was to develop a performant small language model (SmolLM2) through a data-centric approach, optimizing for resource-constrained settings. The key methodology involved multi-stage training with a curated dataset mixing web text, code, math data, and instruction-following data, including newly created datasets (FineMath, Stack-Edu, SmolTalk) and manual refinement of mixing rates. A primary result is that SmolLM2 outperforms other small LMs like Qwen2.5-1.5B and Llama3.2-1B on several benchmarks; for instance achieving a score of 68.7 on HellaSwag compared to 66.4 by Qwen. AI practitioners can leverage the released SmolLM2 model and associated datasets to deploy or further research efficient, high-performing small LMs, particularly beneficial in settings with limited computational resources. |
| TwinMarket: A Scalable Behavioral and Social Simulation for Financial Markets (Read more on arXiv or HuggingFace) | Yunmiao Zhang, Kaidi Zhang, Minghao Wu, Yifei Zhang, Yuzhe Yang | TwinMarket, a multi-agent framework leveraging large language models (LLMs), simulates investor behavior and socio-economic dynamics in a stock market environment. The main research objective is to examine how individual behaviors, through interactions and feedback mechanisms in a simulated stock market, give rise to collective dynamics and emergent phenomena such as financial bubbles. The key methodology involves using LLMs within a Belief-Desire-Intention (BDI) framework to structure agent cognitive processes, coupled with a simulated social network for information exchange and social influence. Primary results show that in a 100-agent simulation, the model replicates stylized facts of financial markets, and rumor-exposed markets experienced a 2.02x increase in Sell/Buy ratio compared to the baseline, indicating amplified panic-driven selling behavior. Principal implication for AI practitioners: simulating human financial behavior by leveraging BDI framework to structure the cognitive process of agents can better predict market behavior under stress. |
| Demystifying Long Chain-of-Thought Reasoning in LLMs (Read more on arXiv or HuggingFace) | Xiang Yue, Graham Neubig, Morry Niu, Yuxuan Tong, Edward Yeo | This paper investigates the mechanics of long chain-of-thought (CoT) reasoning in large language models (LLMs) and identifies key factors influencing its generation and stability. The main research question is what factors enable LLMs to generate long CoT trajectories and how can their emergence be stabilized? The key methodology involves extensive supervised fine-tuning (SFT) and reinforcement learning (RL) experiments, including ablations on reward design and data composition. A primary result is that RL can improve long CoT SFT models by over 3% absolute accuracy on the MATH-500 benchmark, whereas short CoT SFT models showed minimal improvement. The principle implication for AI practitioners is that reward shaping, particularly using a cosine length-scaling reward with a repetition penalty, and scaling verifiable reward signals using a mix of gold and silver supervision data, are crucial for stabilizing long CoT growth and enhancing performance. |
| LIMO: Less is More for Reasoning (Read more on arXiv or HuggingFace) | Shijie Xia, Ethan Chern, Yang Xiao, Zhen Huang, Yixin Ye | LIMO demonstrates that large language models can achieve strong mathematical reasoning with surprisingly few, high-quality training examples. The main research question is whether minimal but precisely orchestrated demonstrations of cognitive processes can elicit sophisticated reasoning in foundation models with comprehensive domain knowledge. The key methodology involves curating a small, high-quality dataset (817 samples) of mathematical problems and solutions, and fine-tuning a pre-trained Qwen2.5-32B-Instruct model. The primary result is that LIMO achieves 57.1% accuracy on the AIME benchmark and 94.8% on MATH, significantly outperforming models trained on much larger datasets. The principal implication for AI practitioners is that focusing on the quality of reasoning demonstrations, rather than sheer data volume, is a more effective approach for developing robust reasoning capabilities in LLMs. |
| Boosting Multimodal Reasoning with MCTS-Automated Structured Thinking (Read more on arXiv or HuggingFace) | Feihu Che, Ruihan Jin, Shuai Zhang, Mingkuan Feng, Jinyang Wu | AStar, an automated structured thinking paradigm, enhances multimodal reasoning in large language models via Monte Carlo Tree Search (MCTS). The main research objective is to address the limitations of existing multimodal large language models (MLLMs) in complex visual reasoning, balancing performance and efficiency. The key methodology involves automatically deriving high-level cognitive reasoning patterns using MCTS-powered hierarchical structures, then integrating these patterns into a unified reasoning framework. The primary result is that AStar achieves a 54.0% accuracy on the MathVerse benchmark with a 7B backbone, surpassing GPT-4O (50.2%). For AI practitioners, AStar provides an effective way to boost MLLMs reasoning performance by leveraging structured patterns derived through the use of MCTS, which in turn, enhance the capability in solving complex problems that require structured thinking. |
| A Probabilistic Inference Approach to Inference-Time Scaling of LLMs using Particle-Based Monte Carlo Methods (Read more on arXiv or HuggingFace) | Akash Srivastava, Kai Xu, Guangxuan Xu, Shivchander Sudalairaj, ishapuri-mit | This paper introduces a probabilistic inference framework for scaling large language models (LLMs) at inference time using particle-based Monte Carlo methods. The main research objective is to develop a more robust inference-time scaling approach that is less susceptible to reward hacking compared to existing search-based methods. The key methodology is casting inference-time scaling as probabilistic inference over a state-space model and applying particle filtering to estimate the latent states, leveraging a language model and a process reward model. The primary result is that the proposed method achieves a 4-16x faster scaling rate than deterministic search counterparts on mathematical reasoning tasks, enabling Qwen2.5-Math-1.5B-Instruct to surpass GPT-40 accuracy with only 4 rollouts. The principal implication for AI practioners is that they can leverage this probabilistic inference approach for more efficient and robust inference-time scaling of LLMs, particularly in domains with imperfect reward models, achieving better performance with smaller models and limited compute budgets. |
| Jailbreaking with Universal Multi-Prompts (Read more on arXiv or HuggingFace) | Shang-Tse Chen, Hsuan Su, Yu-Ling Hsu | JUMP, a prompt-based method, jailbreaks Large Language Models (LLMs) using optimized universal multi-prompts and can also be adapted for defense. The main research objective is to optimize a universal attacker to achieve the best attack results on a set of malicious instructions, outperforming existing techniques. The methodology involves a prompt-based framework named JUMP, decomposing the training pipeline into Selector, Mutator, Constraints, and Evaluator stages, using an additional model as an attacker to generate adversarial suffixes through beam search. Primary results include JUMP++ achieving an Attack Success Rate (ASR@10) of 64.4% on Llama2-7b, significantly outperforming several baselines including AdvPrompter in the universal attack setting. Principal implication is to guide practitioners to use JUMP for a more efficient, high-performing method for jailbreaking and defending LLMs by optimizing universal multi-prompts, reducing computational costs when dealing with massive data. |
| LayerTracer: Cognitive-Aligned Layered SVG Synthesis via Diffusion Transformer (Read more on arXiv or HuggingFace) | Danze Chen, Yiren Song, mikeshou | LayerTracer is a diffusion transformer-based framework for generating layered Scalable Vector Graphics (SVGs) from text or images, mimicking professional design processes. The main research objective is to generate cognitive-aligned, editable layered SVGs that meet professional design standards, overcoming limitations of existing methods. The key methodology involves a dual-phase approach: first, a text-conditioned DiT generates multi-phase rasterized blueprints; second, layer-wise vectorization with path deduplication creates editable SVGs. In the SVG generation task, LayerTracer achieves the highest CLIP-Score of 33.76 with the lowest average number of paths (35.39) and shortest time cost (27s) relative to baselines such as VectorFusion and SVGDreamer. For AI practitioners, LayerTracer provides a novel approach and dataset for generating high-quality, editable layered SVGs, directly aligning AI-generated vectors with professional design cognition. |
| Token Assorted: Mixing Latent and Text Tokens for Improved Language Model Reasoning (Read more on arXiv or HuggingFace) | Yuandong Tian, Jiantao Jiao, Yingchen Xu, Hanlin Zhu, DiJia Su | This paper proposes a method to improve language model reasoning by mixing latent and text tokens in the reasoning trace. The main research question is whether representing initial reasoning steps with discrete latent tokens, while retaining later steps as text, can improve reasoning performance and efficiency in Large Language Models (LLMs). The key methodology involves training a VQ-VAE to convert text tokens into latent codes, then fine-tuning LLMs on reasoning traces where initial text tokens are replaced by these codes, using a randomized replacement strategy. The primary result is that the proposed approach outperforms baseline methods on various benchmarks, such as GSM8K (+4.1% accuracy with Llama-3.2-3B) and an average reduction of 17% of reasoning trace length. The principal implication for AI practioners is that using a mixed representation of latent and text tokens during reasoning trace training can lead to improved accuracy and efficiency compared to using text-only reasoning traces. |
| On Teacher Hacking in Language Model Distillation (Read more on arXiv or HuggingFace) | Nino Vieillard, Sarah Perrin, Johan Ferret, Daniele Calandriello, Daniil Tiapkin | Language model distillation can exhibit “teacher hacking,” where a student model exploits imperfections in the teacher instead of approximating the true data distribution. The main research question is whether teacher hacking occurs during knowledge distillation in language models, and if so, when and how it can be mitigated. A controlled experimental setup is used, involving an oracle (ground-truth) language model, a teacher model distilled from the oracle, and a student model distilled from the teacher. Results show that teacher hacking occurs when using a fixed offline dataset for distillation, observable when optimization deviates from polynomial convergence laws; for example KL divergence between student and teacher decreases, but divergence from Oracle increases. The implication for AI practioners is to utilize online data generation, prioritize prompt diversity, or increase generation budget to mitigate teacher hacking during language model distillation. |
Papers for 2025-02-05
| Title | Authors | Summary |
|---|---|---|
| Inverse Bridge Matching Distillation (Read more on arXiv or HuggingFace) | akorotin, dbaranchuk, apryc1, kekchpek, ngushchin | This paper introduces Inverse Bridge Matching Distillation (IBMD), a novel technique for accelerating the inference of diffusion bridge models (DBMs). The main research question is how to effectively distill both conditional and unconditional DBMs into fast, one-step or few-step generators while maintaining high generation quality. The key methodology is a distillation technique based on solving the inverse bridge matching problem using a tractable objective derived from the inverse formulation. The primary results show that IBMD can accelerate DBM inference by 4x to 100x, with a distilled one-step model achieving a FID score of 2.5 on a 4x super-resolution task, surpassing the teacher model’s score of 2.8 obtained using 1000 steps. The principal implication for AI practitioners is that IBMD provides a universal and efficient method for distilling DBMs, enabling their practical application in various image-to-image translation tasks by significantly reducing inference time. |
| VideoJAM: Joint Appearance-Motion Representations for Enhanced Motion Generation in Video Models (Read more on arXiv or HuggingFace) | Adam Polyak, Yuval Kirstain, Amit Zohar, Uriel Singer, Hila | VideoJAM enhances motion coherence in video generation models by introducing a joint appearance-motion representation. The main research question is how to improve the temporal coherence of generated videos, which often lag behind visual fidelity in current models. The key methodology involves training a diffusion model to predict both pixel appearance and optical flow from a unified latent representation, coupled with an inference-time “Inner-Guidance” mechanism that leverages the model’s own motion predictions to guide generation. Primary results show that VideoJAM outperforms state-of-the-art models on motion coherence, with human evaluators preferring VideoJAM’s motion in 82.0% of cases against the DiT-4B baseline. Principal implication for AI practitioners is that incorporating an explicit motion prior through joint appearance-motion modeling can significantly enhance the temporal consistency of generated videos, directly improving the realism and applicability of video generation models. |
| ACECODER: Acing Coder RL via Automated Test-Case Synthesis (Read more on arXiv or HuggingFace) | Xiaotong Chen, Haozhe Wang, Huaye Zeng, pingnieuk, DongfuJiang | ACECODER automates test-case synthesis to train coder models via reinforcement learning (RL). The main research question is whether leveraging automated large-scale test-case synthesis can enhance code model training through RL. The key methodology involves generating extensive question-test-case pairs from existing code data, constructing preference pairs based on program pass rates, and training reward models using the Bradley-Terry loss, followed by RL. A primary result is that the Qwen2.5-Coder-7B model, after RL fine-tuning, achieved a 25% improvement on HumanEval-plus when starting from the base model directly. The principal implication for AI practitioners is that automated test-case synthesis provides a viable path to enhance code generation models using RL, offering a scalable method to improve model performance without reliance on extensive human-annotated datasets. |
| QLASS: Boosting Language Agent Inference via Q-Guided Stepwise Search (Read more on arXiv or HuggingFace) | Ziniu Hu, Da Yin, Xingcheng Yao, Yao Tang, Zongyu Lin | QLASS is a novel method for enhancing language agent inference through Q-guided stepwise search. The main research question is how to improve the performance of language agents on complex interactive tasks by providing effective intermediate guidance during inference. The key methodology involves automatically generating annotations by estimating Q-values in a stepwise manner, constructing an exploration tree, and performing process reward modeling to guide a Q-guided generation strategy. Primary results show that QLASS outperforms baselines on WebShop, SciWorld, and ALFWorld, achieving a 70.3% success rate on WebShop compared to 67.9% for the next best method, and demonstrates robust performance even with almost half the annotated data. The principal implication for AI practitioners is that QLASS provides a more effective way to perform inference-time search for language agents by leveraging Q-value-based process rewards, leading to improved decision-making in complex interactive tasks. |
| Can LLMs Maintain Fundamental Abilities under KV Cache Compression? (Read more on arXiv or HuggingFace) | Zeyu Li, Peijie Dong, Hong Chen, Zhenheng Tang, Dominic789654 | This paper investigates the impact of KV cache compression methods on large language model (LLM) capabilities. The main research objective is to determine if LLMs retain fundamental abilities under various KV cache compression techniques. A comprehensive empirical study across diverse tasks, employing prominent KV cache compression methods, was conducted. Results showed arithmetic reasoning tasks were particularly sensitive to aggressive compression, with performance drops reaching 43.3%. A key implication for AI practitioners is the task-specific sensitivity to compression, which necessitates careful consideration of task requirements when implementing these methods, particularly for tasks involving arithmetic reasoning. |
| Satori: Reinforcement Learning with Chain-of-Action-Thought Enhances LLM Reasoning via Autoregressive Search (Read more on arXiv or HuggingFace) | Zhenfang Chen, Zhang-Wei Hong, Zhenting Qi, Guangtao Zeng, maohaos2 | Satori is a 7B parameter large language model (LLM) that enhances reasoning capabilities via autoregressive search. The research investigated whether a single LLM could internalize search capabilities to improve reasoning. A two-stage training paradigm was employed, using chain-of-action-thought (COAT) reasoning and reinforcement learning with a “Restart and Explore” strategy. Satori achieved state-of-the-art performance on mathematical reasoning benchmarks, outperforming the instruct model built on the same base model. The study’s principal implication is that reinforcement learning can effectively enhance LLMs’ reasoning abilities, particularly through the introduction of meta-actions and self-improvement techniques, thus providing a more efficient pathway for developing advanced reasoning LLMs. |
| Generating Multi-Image Synthetic Data for Text-to-Image Customization (Read more on arXiv or HuggingFace) | Samaneh Azadi, Ishan Misra, Jun-Yan Zhu, Xi Yin, Nupur Kumari | This paper introduces a method for generating multi-image synthetic data to improve text-to-image model customization. The main research question is how to create a dataset and training method that enables tuning-free customization models to generate high-fidelity images of specific objects in diverse contexts. The key methodology involves generating a synthetic dataset (SynCD) using 3D assets and shared attention mechanisms, and training an encoder-based model with a novel inference technique that normalizes text and image guidance vectors. The primary results show that the proposed method outperforms existing tuning-free methods on standard customization benchmarks, achieving a geometric score of 0.838 with 3 input images compared to 0.780 for the next best method (JeDi). The principal implication for AI practitioners is that using synthetic data with multi-image supervision and shared attention mechanisms can significantly improve the performance of tuning-free text-to-image customization models. |
Papers for 2025-02-04
| Title | Authors | Summary |
|---|---|---|
| The Differences Between Direct Alignment Algorithms are a Blur (Read more on arXiv or HuggingFace) | Boris Shaposhnikov, kefirski, ZeL1k7, ummagumm-a, Myashka | The paper investigates Direct Alignment Algorithms (DAAs) for aligning language models with human preferences, focusing on their performance and key distinctions. The main research objective is to clarify the relationships and comparative advantages among various DAAs, particularly regarding the impact of an explicit Supervised Fine-Tuning (SFT) phase and a scaling parameter, β. The methodology involves incorporating an SFT phase and the β parameter into single-stage DAAs (ORPO and ASFT) and empirically evaluating their performance on benchmarks like Alpaca Eval 2 using Llama 3.1 8B and Llama 3.2 3B models. A primary result is that these modifications improved ORPO’s performance on Alpaca Eval 2 by +3.46 and ASFT’s by +8.27. The principal implication for AI practitioners is that incorporating an explicit SFT phase and tuning the β parameter can significantly enhance the alignment quality of single-stage DAAs, making them competitive with two-stage methods like DPO, and that pairwise methods often outperform pointwise objectives. |
| Process Reinforcement through Implicit Rewards (Read more on arXiv or HuggingFace) | Wendi Li, Zefan Wang, Lifan Yuan, hanbin, ganqu | The paper introduces PRIME, a scalable reinforcement learning framework for enhancing reasoning in large language models using dense token-level rewards. The main research question is how to acquire and utilize high-quality dense rewards at scale for efficient online process reward model (PRM) updates in reinforcement learning of large language models (LLMs). The key methodology is the use of implicit process rewards derived from an Implicit PRM, which is trained with outcome labels only and allows online updates using policy rollouts and outcome labels. The primary result is that Eurus-2-7B-PRIME, trained using PRIME, achieves a 15.1% average improvement across several reasoning benchmarks over the SFT model. The principal implication for AI practitioners is that PRIME offers an efficient way to incorporate dense rewards into reinforcement learning for LLMs, improving sample efficiency and performance without the need for dedicated reward model training or step-level annotations. |
| OmniHuman-1: Rethinking the Scaling-Up of One-Stage Conditioned Human Animation Models (Read more on arXiv or HuggingFace) | Chao Liang, Zerong Zheng, Jiaqi Yang, Jianwen Jiang, Gaojie Lin | OmniHuman-1 is a diffusion-based model for generating human animation videos conditioned on multiple modalities, including text, audio, and pose. The main research objective is to address the challenge of scaling up training data for end-to-end human animation models. The key methodology is a mixed-condition training strategy using a Diffusion Transformer model that integrates text, audio, and pose as conditions, along with an “omni-conditions” approach to leverage data across different conditioning strengths. The primary results show that OmniHuman outperforms existing methods on portrait and body animation tasks, achieving a FID score of 16.970 on the RAVDESS dataset for portrait animation. The principal implication for AI practitioners is that the proposed omni-conditions training strategy effectively scales up human animation models by leveraging mixed-condition data, enabling the development of more versatile and realistic human video generation systems. |
| Preference Leakage: A Contamination Problem in LLM-as-a-judge (Read more on arXiv or HuggingFace) | Bohan Jiang, Ming Zhong, Yue Huang, Dawei Li, RLSNLP | This paper investigates preference leakage, a contamination issue in LLM-as-a-judge systems where evaluator LLMs exhibit biases towards related data generator LLMs. The main research question is whether preference leakage introduces systematic biases in LLM-based evaluations and, if so, to what extent. The key methodology involves training student models on synthetic data generated by different LLMs and then evaluating them using related and unrelated LLM judges, quantifying the bias through a “preference leakage score”. A primary result is that the average preference leakage score for the Mistral-GPT-40 vs Mistral-Gemini-1.5 model pair on AlpacaEval 2.0 was 18.4%, indicating significant bias. The principal implication for AI practitioners is that using closely related LLMs for data generation and evaluation can lead to significant biases, artificially inflating performance metrics and compromising the reliability of assessments. |
| SafeRAG: Benchmarking Security in Retrieval-Augmented Generation of Large Language Model (Read more on arXiv or HuggingFace) | Sensen Zhang, Zhiyu Li, Simin Niu, Xun Liang, UglyToilet | SafeRAG is a new benchmark to evaluate the security of retrieval-augmented generation (RAG) systems against data injection attacks. The main research question is: How vulnerable are RAG systems to attacks that manipulate external knowledge sources? The key methodology involves constructing a dataset, SafeRAG, with four attack types (silver noise, inter-context conflict, soft ad, and white Denial-of-Service) and evaluating 14 RAG components across different stages (indexing, retrieval, generation). A primary result is that the Baichuan 13B model achieved an attack failure rate (AFR) of 1.00 under the Denial-of-Service task, indicating complete resistance. The principal implication for AI practitioners is that current RAG systems, even advanced ones, are vulnerable to sophisticated data injection attacks, highlighting the need to develop more robust retrievers, filters, and generators when building RAG applications. |
| FastKV: KV Cache Compression for Fast Long-Context Processing with Token-Selective Propagation (Read more on arXiv or HuggingFace) | Jae-Joon Kim, Yulhwa Kim, jiwonsong, dongwonjo | FastKV introduces a novel KV cache compression method for large language models (LLMs) to improve efficiency in long-context processing. The main research question is how to enhance the latency and throughput of LLMs handling long-context sequences while maintaining accuracy. The key methodology is Token-Selective Propagation (TSP), which retains full context in initial layers and selectively propagates crucial tokens in deeper layers, alongside grouped-query attention (GQA)-aware KV cache compression. The primary results show that FastKV achieves 2.00x improvement in time-to-first-token (TTFT) and 1.40x improvement in throughput compared to HeadKV. The principal implication for AI practitioners is that FastKV can be used as a drop-in replacement in existing LLMs to significantly reduce latency and increase throughput in long-context processing without sacrificing accuracy. |
| Almost Surely Safe Alignment of Large Language Models at Inference-Time (Read more on arXiv or HuggingFace) | Jun Wang, Ilija Bogunovic, Matthieu Zimmer, Shyam Sundhar Ramesh, Xiaotong Ji | This paper introduces InferenceGuard, a novel inference-time alignment method that ensures large language models (LLMs) generate safe responses with a probability approaching one. The main research question is how to guarantee safe outputs from LLMs during inference without modifying model weights. The key methodology involves framing safe inference-time alignment as a constrained Markov decision process (cMDP), augmenting the state space with a safety constraint tracker, and training a critic in the latent space to guide a lookahead search algorithm. The primary results show that InferenceGuard achieved safety rates of 98.02% on Alpaca-7B and 100% on Beaver-7B-v3 while maintaining strong task performance. The principal implication for AI practitioners is that InferenceGuard offers a practical and theoretically sound approach for safely aligning LLMs during inference, enhancing their usability in real-world applications without the need for retraining. |
| DeepRAG: Thinking to Retrieval Step by Step for Large Language Models (Read more on arXiv or HuggingFace) | Yaojie Lu, Chunlei Xin, Fandong Meng, Jiali Zeng, xinyan233333 | DeepRAG is a retrieval-augmented generation framework that models retrieval-augmented reasoning as a Markov Decision Process for improved efficiency and accuracy. The main research question is how to optimize retrieval-augmented reasoning in large language models by dynamically determining when to retrieve external knowledge versus relying on parametric reasoning. The key methodology is a Markov Decision Process framework called DeepRAG, which uses binary tree search, imitation learning, and chain of calibration to enable strategic and adaptive retrieval. Primary results show that DeepRAG improves answer accuracy by 21.99% while also enhancing retrieval efficiency. The principal implication for AI practitioners is that DeepRAG provides a more effective framework for retrieval-augmented reasoning compared to existing methods, and it achieves superior performance by using dynamic cognitive decision-making. |
| ZebraLogic: On the Scaling Limits of LLMs for Logical Reasoning (Read more on arXiv or HuggingFace) | Radha Poovendran, Ashish Sabharwal, Kyle Richardson, ronanlb, yuchenlin | ZebraLogic is a framework for evaluating the logical reasoning abilities of large language models (LLMs) using logic grid puzzles. The main research question is how LLM performance on logical reasoning tasks scales with problem complexity. The key methodology involves generating logic grid puzzles with controllable complexity using constraint satisfaction problems and evaluating various LLMs’ performance. Primary results show a significant decline in accuracy as problem complexity increases, with most models struggling when the puzzle’s search space exceeds 10^7 possibilities (e.g., gpt-40-mini achieves only 20.1% overall accuracy). The principal implication for AI practitioners is that scaling model size or training data alone is insufficient for solving complex logical reasoning tasks, and increasing test-time compute via more reasoning steps can improve performance. |
| The Jumping Reasoning Curve? Tracking the Evolution of Reasoning Performance in GPT-[n] and o-[n] Models on Multimodal Puzzles (Read more on arXiv or HuggingFace) | Soujanya Poria, Deepanway Ghosal, Yew Ken Chia, Vernon Y. H. Toh | The paper tracks the evolution of multimodal reasoning in GPT-[n] and o-[n] models using visual puzzles. The main research question is how the reasoning performance of these models evolves over time on multimodal puzzles. The key methodology involves evaluating the models on PUZZLEVQA and ALGOPUZZLEVQA datasets using multiple-choice and open-ended questions, with a two-stage prompting strategy for answer extraction. Primary results show that the o1 model achieved 79.2% accuracy on PUZZLEVQA in the multiple-choice setting, but all models performed significantly worse in open-ended settings. The principal implication for AI practitioners is that despite improvements, current models still have limitations in visual perception and abstract reasoning, suggesting a need for further development in these areas. |
| Improving Transformer World Models for Data-Efficient RL (Read more on arXiv or HuggingFace) | Wolfgang Lehrach, Carter Wendelken, Xinghua Lou, Joseph Ortiz, Antoine Dedieu | This paper introduces a model-based reinforcement learning (MBRL) agent that achieves state-of-the-art performance on the Craftax-classic benchmark. The main research question is how to improve the sample efficiency of MBRL agents in complex, open-world environments like Craftax-classic. The key methodology involves combining a novel policy architecture (CNNs and RNNs) with three main improvements to transformer world models (TWMs): “Dyna with warmup”, “nearest neighbor tokenizer” on image patches, and “block teacher forcing”. The primary result is that the proposed MBRL agent achieves a reward of 67.42% after only 1 million environment steps, significantly outperforming DreamerV3, which achieves 53.2%. The principal implication for AI practitioners is that the combination of these techniques provides a more sample-efficient approach to training reinforcement learning agents in environments requiring strong generalization, deep exploration, and long-term reasoning. |
| Improved Training Technique for Latent Consistency Models (Read more on arXiv or HuggingFace) | Dimitris Metaxas, Di Liu, Khanh Doan, trungleuc, quandao10 | This paper introduces an improved training technique for latent consistency models (CMs) to address their suboptimal performance in the latent space compared to pixel space. The main research question is: How can the performance of consistency models in latent space be improved? The key methodology involves replacing Pseudo-Huber loss with Cauchy loss to mitigate the impact of impulsive outliers in latent data, introducing a diffusion loss at early timesteps, employing optimal transport (OT) coupling, using an adaptive scaling-c scheduler, and adopting Non-scaling LayerNorm. The primary result is that the proposed method achieves a FID score of 7.27 for 1-NFE sampling on the CelebA-HQ dataset, a significant improvement over the baseline iLCT model’s FID of 37.15. For AI practitioners, this improved training technique enables the development of more effective latent consistency models capable of generating high-quality samples with one or two steps. |
| Scaling Embedding Layers in Language Models (Read more on arXiv or HuggingFace) | Pritish Kamath, Yangsibo Huang, Badih Ghazi, Edith Cohen, Da Yu | The paper introduces SCONE, a method for scaling input embedding layers in language models without increasing inference-time cost. The main research question is how to enhance language model performance by extending input embedding layers while retaining the original vocabulary and avoiding increased decoding costs. The key methodology involves introducing embeddings for frequent n-grams (f-grams) that are learned with a separate model during training and precomputed/stored off-accelerator for inference. A primary result is that a 1B parameter model using SCONE with 1B f-grams outperformed a 1.9B parameter baseline on the OLMo evaluation mixture, achieving a perplexity of 14.581 compared to 14.598 for the baseline. The principal implication for AI practitioners is that SCONE enables more efficient scaling of language models by leveraging larger embedding layers without impacting inference-time FLOPS, allowing for improved performance within a fixed computational budget. |
| PhD Knowledge Not Required: A Reasoning Challenge for Large Language Models (Read more on arXiv or HuggingFace) | Molly Q Feldman, Federico Cassano, Aleksander Boruch-Gruszecki, Joydeep Biswas, Carolyn Jane Anderson | This paper introduces a benchmark based on the NPR Sunday Puzzle Challenge to evaluate reasoning in large language models using general knowledge questions. The main research objective is to develop a benchmark that tests reasoning capabilities of large language models on problems that are challenging yet require only general knowledge, unlike existing benchmarks that rely on specialized, “PhD-level” knowledge. The key methodology involves curating a dataset of nearly 600 problems from the NPR Sunday Puzzle, prompting models to answer these problems zero-shot, and evaluating their accuracy. The primary results show that OpenAI’s o1 model achieves 59% accuracy, significantly outperforming other models, including DeepSeek R1, which achieved 35% accuracy. The principal implication for AI practitioners is that this benchmark reveals capability gaps in reasoning models that are not evident in benchmarks requiring specialized knowledge, and it highlights specific failure modes like models “giving up” or getting stuck in reasoning. |
| Lifelong Sequential Knowledge Editing without Model Degradation (Read more on arXiv or HuggingFace) | Thomas Hartvigsen, Ahmed Alaa, Maochuan Lu, Phudish Prateepamornkul, akshat57 | This paper introduces a method for lifelong sequential knowledge editing in large language models without significant model degradation. The main research question is how to perform sequential knowledge edits on large language models without causing catastrophic forgetting or loss of downstream performance. The key methodology used is a novel approach called ENCORE, which combines Most-Probable Early Stopping (MPES) during gradient descent with a Frobenius-norm constraint on the weight updates during the least-squares optimization step. The primary results show that ENCORE can perform 10,000 sequential edits without loss of downstream performance and is 61% faster than MEMIT and 64% faster than AlphaEdit on Llama3-8B. The principal implication for AI practitioners is that ENCORE enables more efficient and robust sequential knowledge editing, allowing for continual updating of models without significant degradation in performance on downstream tasks. |
| Current Pathology Foundation Models are unrobust to Medical Center Differences (Read more on arXiv or HuggingFace) | Jonas Teuwen, Eric Marcus, EdwinDdeJong | Here is a concise summary of the research paper: i) This paper evaluates the robustness of current pathology foundation models (FMs) to medical center differences, finding significant sensitivity to this confounding factor. ii) The main research objective is to measure whether pathology FMs focus on biological features like tissue and cancer type, or on confounding medical center signatures. iii) The key methodology used is the introduction of a “Robustness Index” to quantify the degree to which biological features dominate confounding features in the FM embedding space, along with an analysis of the impact of unrobustness on downstream model performance. iv) The primary results show that all evaluated pathology FMs represent the medical center to a strong degree, with the Virchow2 model achieving the highest Robustness Index of 1.20, indicating that it is the only model where biological information dominated the medical center information for the first 50 neighbors. v) The principal implication for AI practitioners is that current pathology FMs are highly sensitive to medical center variations, and this sensitivity affects downstream tasks such as cancer type classification, highlighting the need for models that are more robust to such confounding factors for reliable clinical applications. |
| A Study on the Performance of U-Net Modifications in Retroperitoneal Tumor Segmentation (Read more on arXiv or HuggingFace) | Rebecca Scalabrino, Daniel Hsu, Alexander Manzella, Ehsan Khodapanah Aghdam, Moein Heidari | Here is a summary of the research paper based on the provided guidelines: This study evaluates U-Net variants for segmenting retroperitoneal tumors in CT images, introducing a novel architecture called ViLU-Net. The main research question is how the performance of U-Net-based models incorporating convolutional neural networks (CNNs), Vision Transformers (ViTs), Mamba, and xLSTM components compares in segmenting retroperitoneal tumors. The key methodology involves implementing and training various U-Net modifications, including the proposed ViLU-Net which integrates Vision x-LSTM (ViL) blocks within a U-shaped encoder-decoder framework, on a new dataset of 82 retroperitoneal tumor CT cases and the public FLARE 2022 dataset. The primary results show that ViLU-Net achieved the highest average Dice Similarity Coefficient (DSC) of 0.8594 on the abdomen CT dataset among the tested models. The principal implication for AI practitioners is that xLSTM-based architectures like ViLU-Net offer a promising approach for medical image segmentation, demonstrating superior performance with reduced complexity compared to existing models. |
Papers for 2025-02-03
| Title | Authors | Summary |
|---|---|---|
| s1: Simple test-time scaling (Read more on arXiv or HuggingFace) | Xiang Lisa Li, percyliang, swj0419, zitongyang, Muennighoff | i) The paper introduces “s1”, a straightforward method for enhancing language model reasoning and achieving test-time scaling by using a small, carefully curated dataset and a novel budget-forcing technique. ii) Main research question or objective: What is the simplest approach to achieve both test-time scaling and strong reasoning performance in language models? iii) Key methodology used: The authors curated a 1,000-sample dataset (s1K) based on difficulty, diversity, and quality, and developed a test-time budget forcing technique to control model thinking time. iv) Primary results: The s1-32B model, finetuned on s1K and equipped with budget forcing, outperformed the o1-preview model on competition math questions by up to 27% on MATH and AIME24 benchmarks and demonstrated test-time scaling, improving from 50% to 57% on AIME24 with increased thinking time. v) Principal implication for AI practitioners: AI practitioners can leverage the s1K dataset and budget forcing technique to significantly improve the reasoning capabilities and test-time performance of language models with minimal training data and a simple test-time intervention. |
| Reward-Guided Speculative Decoding for Efficient LLM Reasoning (Read more on arXiv or HuggingFace) | doyensahoo, JunnanLi, hendrydong, yuhuixu, baohao | Reward-Guided Speculative Decoding (RSD) is introduced to improve the efficiency of large language model (LLM) inference, particularly for multi-step reasoning tasks. The main research question is how to balance efficiency and accuracy in LLM inference by integrating lightweight “draft” evaluations with reward-driven refinements from a more capable “target” model. The key methodology involves using a process reward model to evaluate intermediate decoding steps from a draft model and dynamically deciding whether to accept them or invoke the target model for correction based on reward thresholds. Primary results show that RSD achieves up to 4.4× fewer FLOPs compared to using the target model alone, while achieving up to 3.5 higher accuracy than standard speculative decoding on reasoning benchmarks. For AI practitioners, RSD provides a robust framework to deploy LLMs more efficiently in resource-intensive scenarios by optimizing the trade-off between computational cost and output quality. |
| Self-supervised Quantized Representation for Seamlessly Integrating Knowledge Graphs with Large Language Models (Read more on arXiv or HuggingFace) | Fangzhi Xu, Zhen Peng, Kai He, Tianzhe Zhao, Qika | The paper introduces a method for integrating Knowledge Graphs (KGs) with Large Language Models (LLMs) using quantized representations. The main research question is how to effectively bridge the gap between KG structures and the natural language format of LLMs to achieve seamless integration. The key methodology involves a self-supervised quantized representation (SSQR) method that compresses KG structural and semantic knowledge into discrete codes, followed by constructing KG instruction-following data to fine-tune LLMs. Primary results show that SSQR outperforms existing unsupervised quantized methods, achieving a 9.28% improvement in Mean Reciprocal Rank (MRR) compared to the previous best performance on the WN18RR dataset. The principal implication for AI practitioners is that they can leverage the SSQR method to seamlessly integrate KGs with LLMs by using the learned quantized codes as input features, enhancing model performance on KG-related tasks such as link prediction and triple classification without requiring significant architectural modifications. |
| Constitutional Classifiers: Defending against Universal Jailbreaks across Thousands of Hours of Red Teaming (Read more on arXiv or HuggingFace) | Primusa, euanong, sgoodfriend, jayelm, meg-tong | Constitutional Classifiers are synthetic safeguards that defend large language models (LLMs) against universal jailbreaks by using a constitution of natural language rules. The main research question is whether Constitutional Classifiers can effectively defend LLMs against universal jailbreak strategies that systematically bypass model safeguards and extract harmful information. The key methodology involves training classifiers on synthetic data generated by prompting LLMs with a constitution that specifies permitted and restricted content, followed by extensive red teaming to test robustness. The primary results show that in over 3,000 estimated hours of red teaming, no red teamer found a universal jailbreak that could extract information at a similar level of detail to an unguarded model across most target queries, and enhanced classifiers demonstrated robust defense against held-out domain-specific jailbreaks, with an absolute 0.38% increase in production-traffic refusals. The principal implication for AI practitioners is that Constitutional Classifiers offer a viable defense against universal jailbreaks while maintaining practical deployment feasibility, and thus can play a crucial role in safely deploying capable AI systems. |
| Trading Inference-Time Compute for Adversarial Robustness (Read more on arXiv or HuggingFace) | Sam Toyer, Stephanie Lin, Boaz Barak, Evgenia Nitishinskaya, Wojciech Zaremba | Here is a concise summary of the research paper: This paper investigates the impact of increased inference-time computation on the adversarial robustness of reasoning models. The main research question is whether increasing inference-time compute can improve the robustness of large language models (LLMs) against adversarial attacks without adversarial training. The key methodology involves testing various adversarial attacks on OpenAI’s reasoning models (01-preview and 01-mini) and measuring attack success rates as a function of inference-time compute. The primary results show that increased inference-time compute generally improves robustness across a range of attacks, with the attack success rate often decreasing to zero as test-time compute grows; for example, in a many-shot attack on a math task, increasing inference-time compute reduced the success rate of an adversary aiming to output the correct answer multiplied by 7 to near zero. The principal implication for AI practitioners is that scaling inference-time compute can be a viable strategy for enhancing the adversarial robustness of LLMs, offering a complementary approach to traditional adversarial training. |
| INT: Instance-Specific Negative Mining for Task-Generic Promptable Segmentation (Read more on arXiv or HuggingFace) | Shaogang Gong, Zixu Cheng, Jian Hu | Instance-specific Negative Mining for Task-Generic Promptable Segmentation (INT) is introduced to improve segmentation accuracy using a single task-generic prompt. The main research question is how to generate accurate instance-specific prompts for image segmentation from a single task-generic prompt without per-instance supervision. The key methodology involves instance-specific prompt generation using negative mining on Vision-Language Model (VLM) outputs and semantic mask generation using GroundingDINO and SAM, refined iteratively. The primary results show that INT achieves a mean Intersection over Union (mIoU) of 0.808 on the CHAMELEON dataset for camouflaged object detection, outperforming existing methods. The principal implication for AI practitioners is that INT provides a method to enhance the accuracy of promptable segmentation models by effectively leveraging a single task-generic prompt across diverse images without requiring instance-specific annotations, thereby simplifying the segmentation process and potentially broadening its application in scenarios with limited labeled data. |
| Unraveling the Capabilities of Language Models in News Summarization (Read more on arXiv or HuggingFace) | Göksel Biricik, odabashi | This research paper benchmarks 20 language models for news summarization across three datasets using zero-shot and few-shot learning. The main research question is how effectively smaller-scale language models handle news summarization compared to larger models, balancing efficiency and performance. The key methodology involves a multifaceted evaluation approach including automatic metrics (ROUGE, METEOR, BERTScore), human evaluation, and AI-based evaluation using GPT-3.5-Turbo and GPT-4 as a judge. Primary results indicate that GPT-3.5-Turbo achieved the highest scores in automated metrics on the CNN/DM dataset in the zero-shot setting, with a ROUGE-L score of 0.2077, but including demonstration examples in the few-shot setting did not enhance the performance of the models, and in some cases, led to worse quality of the generated summaries. The principal implication for AI practitioners is that while large models like GPT-3.5-Turbo and GPT-4 dominate in news summarization tasks, smaller models such as Qwen1.5-7B, SOLAR-10.7B-Instruct-v1.0, Meta-Llama-3-8B and Zephyr-7B-Beta show promising results, offering competitive alternatives. |
| Fast Encoder-Based 3D from Casual Videos via Point Track Processing (Read more on arXiv or HuggingFace) | Haggai Maron, Wuyue Lu, Yoni Kasten | Here is a concise summary of the research paper “Fast Encoder-Based 3D from Casual Videos via Point Track Processing”: TRACKSTO4D, a learning-based approach, reconstructs 3D structures and camera positions from 2D point tracks extracted from casual videos in a single feed-forward pass. The main research question is how to efficiently infer 3D structure and camera positions from dynamic content in casual videos without relying on lengthy optimization processes. The key methodology involves a novel encoder architecture that processes 2D point track tensors as input, incorporating symmetry-aware attention mechanisms and a low-rank assumption for movement patterns to predict 3D point clouds and camera poses. The primary results show that TRACKSTO4D achieves comparable accuracy to state-of-the-art methods while reducing runtime by up to 95%, with a specific finding that it reduces inference time by 95% compared to the baseline. The principal implication for AI practitioners is that they can leverage TRACKSTO4D for significantly faster 3D reconstruction from casual videos, enabling more efficient development of applications in areas like robot navigation and autonomous driving without sacrificing accuracy. |
| DINO-WM: World Models on Pre-trained Visual Features enable Zero-shot Planning (Read more on arXiv or HuggingFace) | Lerrel Pinto, Yann LeCun, Hengkai Pan, Gaoyue Zhou | DINO-WM is a method for training visual world models using pretrained DINOv2 embeddings for task-agnostic behavior planning. The main research question is whether a world model can be trained offline on pre-collected trajectories to support test-time behavior optimization and task-agnostic reasoning using only passive data. The key methodology involves using DINOv2 patch features to model visual dynamics without reconstructing the visual world, predicting future patch features from offline behavioral trajectories. The primary result is that DINO-WM achieves a 90% success rate on the Push-T task, compared to 4% for DreamerV3. For AI practitioners, DINO-WM demonstrates that pretrained visual features can be leveraged to create world models capable of zero-shot planning across diverse tasks without task-specific data, enabling more generalizable and efficient robot learning. |
Papers for 2025-01-31
| Title | Authors | Summary |
|---|---|---|
| GuardReasoner: Towards Reasoning-based LLM Safeguards (Read more on arXiv or HuggingFace) | lakxtxue, JunXia97, zsf, HongchengGao, yueliu1998 | GuardReasoner is a reasoning-based safeguard for large language models (LLMs) that improves performance, explainability, and generalizability. The main research objective is to develop a guard model that can effectively moderate LLM inputs and outputs by incorporating reasoning capabilities. The key methodology involves creating a new dataset, GuardReasonerTrain, with 127K samples and 460K reasoning steps, and using reasoning supervised fine-tuning (R-SFT) and hard sample direct preference optimization (HS-DPO) to train the model. The primary result is that GuardReasoner 8B surpasses GPT-40+CoT by 5.74% and LLaMA Guard 3 8B by 20.84% F1 score on average across 13 benchmarks. The principal implication for AI practitioners is that incorporating explicit reasoning steps into guard models can significantly enhance their ability to detect and mitigate harmful content, offering a more robust and explainable safeguard mechanism for LLMs. |
| MedXpertQA: Benchmarking Expert-Level Medical Reasoning and Understanding (Read more on arXiv or HuggingFace) | Zhangren Chen, Yifei Li, Yuxin Zuo, stingning, lindsay-qu | MedXpertQA is introduced, a new benchmark for evaluating expert-level medical reasoning and understanding in AI systems. i) MedXpertQA, a challenging and comprehensive medical benchmark, is introduced to evaluate expert-level medical knowledge and advanced reasoning in AI. ii) The main research objective is to create a benchmark, MedXpertQA, that addresses limitations of existing medical AI benchmarks by incorporating specialty board questions, improving clinical relevance, and mitigating data leakage. iii) The key methodology involves curating a large-scale question bank from professional medical exams and textbooks, filtering questions using AI and human expert evaluation, augmenting data via model-based rewriting, and conducting multiple rounds of expert reviews to ensure quality. iv) The primary results show that leading AI models, such as GPT-4o, achieve limited performance on MedXpertQA, with GPT-4o achieving 35.96% average accuracy, indicating the benchmark’s difficulty. v) The principal implication for AI practitioners is that MedXpertQA provides a rigorous tool for evaluating and improving medical AI systems, particularly in complex reasoning tasks, driving advancements towards more reliable and clinically applicable AI in healthcare. |
| Thoughts Are All Over the Place: On the Underthinking of o1-Like LLMs (Read more on arXiv or HuggingFace) | yudian, freesunshine0316, zwhe99, Jiahao004, Dennis364 | Large language models (LLMs) termed “o1-like” exhibit a tendency to switch reasoning strategies prematurely, leading to a phenomenon called “underthinking.” The main research question is whether o1-like LLMs are thinking deeply enough when solving complex reasoning tasks. The key methodology involved analyzing thought-switching patterns in model responses and introducing a decoding strategy with thought-switching penalties. Primary results showed that incorrect answers from o1-like models had 418% more frequent thought-switching behaviors than correct answers. The principal implication for AI practitioners is that addressing underthinking through techniques like the proposed thought-switching penalty can improve the accuracy of o1-like LLMs on challenging datasets without requiring model fine-tuning. |
| PhysBench: Benchmarking and Enhancing Vision-Language Models for Physical World Understanding (Read more on arXiv or HuggingFace) | Vitor Guizilini, Daniel Seita, Jiageng Mao, Boyiliee, WeiChow | PhysBench is a benchmark for evaluating vision-language models’ (VLMs) understanding of the physical world through analysis of video, image, and text data. The main research question is whether existing VLMs possess an understanding of the physical world and how this understanding can be enhanced to improve embodied agent performance. The key methodology used involves the development of the PhysBench dataset, comprising 10,002 video-image-text entries across four physical domains, and a novel framework called PhysAgent that integrates vision foundation models and a physics knowledge memory to enhance VLMs. Primary results show that while state-of-the-art VLMs like GPT-4o achieve an average accuracy of 49.49% on PhysBench, the proposed PhysAgent framework improves GPT-4o’s performance by 18.4%. The principal implication for AI practitioners is that enhancing VLMs with specialized vision models and physics knowledge can significantly improve their physical world understanding, thereby facilitating the development of more capable embodied agents. |
| Streaming DiLoCo with overlapping communication: Towards a Distributed Free Lunch (Read more on arXiv or HuggingFace) | Zachary Charles, Satyen Kale, Keith Rush, Yanislav Donchev, Arthur Douillard | Training large language models (LLMs) can be distributed across non-colocated devices with reduced communication bandwidth using Streaming DiLoCo. The main research question is how to minimize peak bandwidth requirements and mitigate worker-blocking during distributed training of LLMs without compromising learning efficiency. The key methodology involves synchronizing subsets of model parameters in sequence, overlapping communication with computation, and quantizing the exchanged data. The primary results show that Streaming DiLoCo achieves similar performance to data-parallel training while reducing the required bandwidth by two orders of magnitude; for instance, a 1 billion parameter model achieved an evaluation loss of 2.50 with Streaming DiLoCo versus 2.49 with Data-Parallel. The principal implication for AI practitioners is that they can train LLMs across distributed devices with significantly lower bandwidth requirements, enabling more geographically distributed training setups and potentially reducing infrastructure costs. |
| WILDCHAT-50M: A Deep Dive Into the Role of Synthetic Data in Post-Training (Read more on arXiv or HuggingFace) | Chinmay Hegde, penfever | WILDCHAT-50M is a large-scale dataset of synthetic chat transcripts for improving language model post-training. The main research question is how the choice of data-generating model (DGM) impacts the synthetic data quality (SDQ) and downstream performance of language models (LLMs) after supervised fine-tuning (SFT). The key methodology involves generating chat transcripts using 50 different open-weight models ranging from 0.5B to 104B parameters and evaluating the performance of LLMs fine-tuned on these synthetic datasets using a mix of ground-truth and LLM-judge benchmarks. The primary results show that the choice of DGM significantly affects downstream benchmark performance, with fine-tuning on the RE-WILD data mix outperforming the Tulu-3 SFT mix by an average of 0.039 points across nine benchmarks. The principal implication for AI practitioners is that carefully selecting a high-quality DGM for generating synthetic data can compensate for a smaller dataset size and improve the performance of LLMs on generalist chat and instruction-following tasks. |
| o3-mini vs DeepSeek-R1: Which One is Safer? (Read more on arXiv or HuggingFace) | Miriam Ugarte, ssegura, japarejo, pablovalle, aitorarrieta | Here is a concise summary of the research paper “o3-mini vs DeepSeek-R1: Which One is Safer?”: i) This paper presents a comparative analysis of the safety alignment of two large language models, OpenAI’s o3-mini and DeepSeek-R1, using the automated safety testing tool ASTRAL. ii) The main research objective was to determine which of the two models exhibits a higher level of safety when responding to unsafe prompts. iii) The key methodology involved generating 1260 unsafe test inputs using ASTRAL and evaluating the safety of the models’ responses through automated and manual assessment. iv) Primary results indicate that DeepSeek-R1 responded unsafely to 11.98% of the prompts, while o3-mini responded unsafely to only 1.19%. v) The principal implication for AI practitioners is that DeepSeek-R1 may require further refinement to improve its safety alignment, and practitioners should be aware of the potential for unsafe responses when deploying this model. |
| Large Language Models Think Too Fast To Explore Effectively (Read more on arXiv or HuggingFace) | Robert C. Wilson, xhb120633, louanna | Summary of the research paper is: The study investigates exploration capabilities of Large Language Models (LLMs) in an open-ended task, revealing that most LLMs underperform compared to humans due to a tendency to make premature decisions. The main research question is whether LLMs can explore effectively in an open-ended task, comparable to humans. The key methodology involves using the game Little Alchemy 2 as a paradigm, applying regression models to analyze exploration strategies, and using Sparse Autoencoders (SAE) to probe latent representations of exploration-related values. The primary results show that o1 significantly outperformed humans (t = 9.71, p < 0.001), while other LLMs performed worse, with most models relying primarily on uncertainty-driven strategies. The principal implication for AI practitioners is that the current architecture of traditional LLMs may hinder effective exploration in open-ended tasks due to their tendency to process uncertainty and choices much earlier than empowerment values. |
Papers for 2025-01-30
| Title | Authors | Summary |
|---|---|---|
| Critique Fine-Tuning: Learning to Critique is More Effective than Learning to Imitate (Read more on arXiv or HuggingFace) | Xiang Yue, wenhu, ubowang | Critique Fine-Tuning (CFT) is more effective than Supervised Fine-Tuning (SFT) for enhancing mathematical reasoning in language models. The main research question is whether training language models to critique noisy responses is more effective than traditional imitation learning for improving mathematical reasoning. The key methodology involves constructing a 50K-sample dataset from WebInstruct and training models to provide critiques on query-response pairs using GPT-4o as a teacher. The primary result is that the Qwen2.5-Math-7B-CFT model achieved 56.0% average accuracy on mathematical reasoning benchmarks, outperforming the best SFT-trained model by 5.7%. The principal implication for AI practitioners is that CFT offers a more data-efficient and effective alternative to SFT for enhancing reasoning capabilities in large language models, as evidenced by the model trained on just 50K samples outperforming others trained on over 2M samples. |
| Exploring the sustainable scaling of AI dilemma: A projective study of corporations’ AI environmental impacts (Read more on arXiv or HuggingFace) | Simon Gosset, Caroline Vateau, Louis Ladan, Neyri56, clementdesroches | This paper proposes a methodology to estimate the environmental impact of a company’s AI portfolio, focusing on Generative AI’s increasing energy consumption. The main research objective is to develop a simplified yet exhaustive methodology for estimating the operational and embodied environmental impacts of AI solutions at a company level. The key methodology involves four interconnected models: life cycle impacts of primary components, life cycle impacts of AI use cases, an AI company portfolio model, and 2030 AI Landscape projections. The primary results indicate that large generative AI models consume up to 4600 times more energy than traditional models, and under a high adoption scenario, AI electricity use is projected to rise by a factor of 24.4 by 2030. The principal implication for AI practitioners is the need to adopt standardized environmental assessment frameworks and the “Return on Environment” metric to align AI development with net-zero goals due to the significant environmental impact of generative AI. |
| Atla Selene Mini: A General Purpose Evaluation Model (Read more on arXiv or HuggingFace) | Kyle Dai, Jackson Golden, Henry Broomfield, Andrei Alexandru, NinaCalvi | Atla Selene Mini is a state-of-the-art small language model fine-tuned for general-purpose evaluation. The main research objective was to develop a small language model-as-a-judge (SLMJ) that outperforms existing SLMJs and GPT-40-mini on diverse evaluation tasks. The key methodology involved curating a training dataset of 577k data points from 16 public datasets, augmented with synthetically generated critiques, filtered for quality, and fine-tuning a Llama 3.1 8B Instruct model using a combined direct preference optimization (DPO) and supervised fine-tuning (SFT) loss. The primary results showed that Selene Mini achieved an overall task-average performance of 0.756, outperforming other SLMJs and GPT-40-mini. The principal implication for AI practitioners is that Selene Mini provides a high-performing, promptable, and efficient model for automated evaluation, demonstrating strong performance in real-world scenarios and robustness to prompt variations. |
| Early External Safety Testing of OpenAI’s o3-mini: Insights from the Pre-Deployment Evaluation (Read more on arXiv or HuggingFace) | Miriam Ugarte, ssegura, japarejo, pablovalle, aitorarrieta | Here is a concise summary of the AI research paper: The paper presents an external safety evaluation of OpenAI’s o3-mini large language model (LLM) using the automated testing tool ASTRAL. The main research objective is to assess the safety of the o3-mini model by generating and executing a large number of unsafe test inputs. The key methodology involved using ASTRAL to automatically generate 10,080 unsafe test inputs (prompts) across 14 safety categories, with variations in writing style and persuasion techniques, and then evaluating the model’s responses. The primary results showed that ASTRAL identified 87 unsafe LLM outcomes after manual verification, with the most unsafe outcomes found in the “controversial topics and politics” category. The principal implication for AI practitioners is that automated tools like ASTRAL can effectively identify safety issues in LLMs, but the effectiveness of safety measures may vary across different categories, highlighting the importance of comprehensive testing. |
| Virus: Harmful Fine-tuning Attack for Large Language Models Bypassing Guardrail Moderation (Read more on arXiv or HuggingFace) | ling1119, sftekin25, tawreos, SihaoHu, TianshengHuang | This paper introduces a novel attack method called Virus that bypasses guardrail moderation in fine-tuning large language models (LLMs). The main research question is whether a harmful fine-tuning attack can bypass guardrail moderation and degrade the safety alignment of victim LLMs. The key methodology is a dual-goal data optimization scheme that optimizes harmful data to simultaneously bypass the guardrail and maintain attack effectiveness. The primary result is that Virus achieves up to a 100% leakage ratio through the guardrail and increases the victim model’s harmful score by up to 21.8%. The principal implication for AI practitioners is that relying solely on guardrail moderation for filtering harmful data during fine-tuning is insufficient to maintain the safety alignment of LLMs, and other robust defenses are needed. |
Papers for 2025-01-29
| Title | Authors | Summary |
|---|---|---|
| SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-training (Read more on arXiv or HuggingFace) | Saining Xie, Shengbang Tong, Jihan Yang, Yuexiang Zhai, Tianzhe Chu | Summary of the research paper is the following: The paper investigates the effects of Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL) on foundation model generalization and memorization in textual and visual domains. The main research question is whether SFT or RL leads to better generalization in foundation models when applied to unseen variants of learned tasks. The key methodology involves training language and vision-language models with SFT and RL on two tasks, GeneralPoints and V-IRL, and evaluating their performance on in-distribution and out-of-distribution variations of these tasks. The primary results show that RL, especially with an outcome-based reward, leads to better generalization than SFT across both tasks; for example, RL improves out-of-distribution performance on the V-IRL-L task by +11.0% (80.8% to 91.8%). The principal implication for AI practitioners is that RL should be favored over SFT when the goal is to enhance the generalization capability of foundation models to new, unseen task variants, particularly in complex, multi-modal tasks. |
| Optimizing Large Language Model Training Using FP4 Quantization (Read more on arXiv or HuggingFace) | Guoshuai Zhao, Xiao Liu, Yeyun Gong, Ruizhe Wang, cp5555 | This paper introduces an FP4 quantization framework for training large language models (LLMs). The main research question is whether it is feasible to train LLMs using 4-bit floating-point (FP4) quantization while maintaining accuracy comparable to higher-precision formats. The key methodology involves a differentiable quantization estimator for weight updates, an outlier clamping and compensation strategy for activations, mixed-precision training, and vector-wise quantization. The primary results demonstrate that the FP4 framework achieves accuracy comparable to BF16 and FP8, with training losses of 2.55 (FP4) vs. 2.49 (BF16) for a 1.3B parameter LLaMA model trained on 100B tokens. The principal implication for AI practitioners is that the proposed FP4 quantization method enables more efficient training of LLMs, potentially reducing computational costs and accelerating development, although the current lack of hardware support for FP4 limits direct measurement of speedup and energy efficiency gains. |
| Over-Tokenized Transformer: Vocabulary is Generally Worth Scaling (Read more on arXiv or HuggingFace) | Ya Wang, Yutao Zeng, Banggu Wu, Defa Zhu, Hongzhi Huang | Here is a concise summary of the research paper: The paper introduces Over-Tokenized Transformers, a framework that decouples input and output vocabularies to improve language modeling by scaling up input vocabularies with multi-gram tokens. The main research question is how scaling input and output vocabularies separately impacts the performance of large language models. The key methodology involves using hierarchical n-gram input vocabularies and analyzing the relationship between vocabulary size and training loss through experiments on context-free grammar and natural language modeling. A primary result is a log-linear relationship between input vocabulary size and training loss, with a 400M parameter model with an input vocabulary size of 12.8 million matching the training loss of a 1B parameter baseline model. The principal implication for AI practitioners is that scaling input vocabulary size, independent of output vocabulary size, can significantly enhance model scalability and performance without increasing training costs. |
| Open Problems in Mechanistic Interpretability (Read more on arXiv or HuggingFace) | Jeff Wu, Jack Lindsey, Joshua Batson, Lee Sharkey, bilalchughtai | Here is a summary of the paper “Open Problems in Mechanistic Interpretability”: This paper reviews the current state and future directions of mechanistic interpretability research for neural networks. The main research objective is to identify open problems in mechanistic interpretability methods, applications, and socio-technical aspects that need to be addressed to achieve the field’s scientific and engineering goals. The key methodology used is a synthesis of perspectives from various authors, combining literature review with forward-looking analysis to identify gaps and challenges. The primary results indicate that current decomposition methods, such as sparse dictionary learning, have high reconstruction errors, with one experiment showing that using sparse dictionary reconstructions in GPT-2 reduced performance by 40% when trained on the full distribution. The principal implication for AI practitioners is that significant advancements in decomposition, description, and validation methods are needed to enable reliable monitoring, control, and prediction of AI systems, particularly for safety-critical applications. |
| DiffSplat: Repurposing Image Diffusion Models for Scalable Gaussian Splat Generation (Read more on arXiv or HuggingFace) | Yadong Mu, Zeming Li, Bangbang Yang, Panwang Pan, Chenguo Lin | Here is a concise summary of the research paper “DiffSplat: Repurposing Image Diffusion Models for Scalable Gaussian Splat Generation”: DiffSplat is a novel 3D generative framework that leverages pretrained image diffusion models to generate 3D Gaussian Splats. The main research objective is to develop a 3D generative model that can effectively utilize web-scale 2D image priors while maintaining 3D consistency. The key methodology involves fine-tuning image diffusion models to directly generate structured Gaussian splat grids, utilizing a lightweight reconstruction model for scalable 3D dataset curation and a 3D rendering loss for multi-view consistency. The primary result is that DiffSplat achieves a CLIP similarity score of 30.95% on single object text-conditioned generation, outperforming other methods. For AI practitioners, DiffSplat provides an efficient way to generate high-quality 3D content by repurposing existing 2D image diffusion models, establishing a bridge between 3D content creation and the image generation community. |
| IndicMMLU-Pro: Benchmarking Indic Large Language Models on Multi-Task Language Understanding (Read more on arXiv or HuggingFace) | Nikunj Kotecha, Ashutosh Kumar, Sankalp KJ, amanchadha, laxmaanb | IndicMMLU-Pro is a benchmark for evaluating large language models (LLMs) on nine major Indic languages across various tasks. The main research objective is to establish a comprehensive benchmark for evaluating the performance of multilingual LLMs in understanding and generating text in Indic languages. The key methodology involved translating the English MMLU-Pro dataset into nine Indic languages using IndicTrans2 and validating the translations through back-translation, multiple evaluation metrics, and expert review. The primary results show that GPT-40 consistently outperformed other models, achieving the highest accuracy of 44.80% in Hindi. The principal implication for AI practitioners is that this benchmark can guide the development of more accurate and culturally sensitive multilingual LLMs for Indic languages, although there is a pressing need for higher-quality, diverse datasets across all Indic languages. |
| Low-Rank Adapters Meet Neural Architecture Search for LLM Compression (Read more on arXiv or HuggingFace) | Nilesh Jain, Jinjie Yuan, J. Pablo Muñoz | This paper explores synergistic methods combining low-rank adapters with neural architecture search (NAS) to compress large language models (LLMs). The research objective is to develop robust solutions for compressing and efficiently fine-tuning large pre-trained LLMs. The key methodology integrates low-rank representations, particularly elastic LoRA adapters, with weight-sharing super-networks from NAS techniques. One primary result demonstrates an inference speedup of up to 1.4x while reducing model parameters by approximately 80% in some experiments. The principal implication is that these combined strategies offer efficient LLM compression and fine-tuning, making LLMs more accessible for deployment in resource-constrained environments. |
| Histoires Morales: A French Dataset for Assessing Moral Alignment (Read more on arXiv or HuggingFace) | Charlotte Laclau, Julien Velcin, Antoine Gourru, Irina Proskurina, Thibaud Leteno | HISTOIRESMORALES, a French dataset derived from MORALSTORIES, is introduced for evaluating moral alignment in large language models (LLMs). The main research objective is to assess how well LLMs handle moral reasoning in French and compare it to English. The key methodology involves translating the MORALSTORIES dataset into French using a refined prompting strategy with GPT-3.5-turbo-16k, followed by manual annotation and validation, and evaluating LLMs using perplexity and action selection with declarative prompts. The primary results show that LLMs align better with moral norms in English than in French, with Mistral selecting the moral action 93.78% of the time in English versus 83.59% in French when prompted with the norm. For AI practitioners, the principal implication is that the HISTOIRESMORALES dataset can be used to evaluate and improve the moral alignment of LLMs in French, highlighting the importance of language-specific datasets for nuanced evaluations of model behavior. |
Papers for 2025-01-28
| Title | Authors | Summary |
|---|---|---|
| Baichuan-Omni-1.5 Technical Report (Read more on arXiv or HuggingFace) | Song Chen, Tao Zhang, Tao Zhang, Jun Liu, AdamLee1 | Baichuan-Omni-1.5 is a unified omni-modal large language model designed to process text, image, audio, and video inputs, achieving seamless cross-modal interactions. The research objective was to develop an omni-modal model with fluent and high-quality cross-modal interaction capabilities, particularly including end-to-end audio generation. The methodology involved a multi-stage training strategy using a high-quality 500B multimodal dataset, an audio-tokenizer, and progressive multimodal alignment. Results showed Baichuan-Omni-1.5 outperforming leading omni-modal models like VITA-1.5 and MiniCPM-0 2.6 on various benchmarks, including an average score of 73.3 across ten image understanding benchmarks. This work provides AI practitioners with a state-of-the-art open-source omni-modal model exhibiting superior performance across multiple modalities, particularly in medical image understanding. The details of some training hyperparameters are not explicitly stated in the provided excerpt, therefore a complete evaluation is difficult. |
| Qwen2.5-1M Technical Report (Read more on arXiv or HuggingFace) | Fei Huang, Dayiheng Liu, Chengyuan Li, Bowen Yu, An Yang | Qwen2.5-1M is a series of models that extend the context length to 1 million tokens, enhancing long-context capabilities. The main research objective is to develop and optimize models that can effectively process and understand sequences up to 1 million tokens long. Key methodologies include long data synthesis, progressive pre-training, multi-stage supervised fine-tuning, a training-free length extrapolation method, and a sparse attention mechanism. The Qwen2.5-14B-Instruct-1M model achieved 92.2 accuracy on 128k sequences in the RULER benchmark. For AI practitioners, the principal implication is that the provided inference framework and models, particularly Qwen2.5-14B-Instruct-1M, offer a robust solution for developing applications requiring long-context processing, with a remarkable 3x to 7x prefill speedup in scenarios with 1 million tokens of context. |
| Towards General-Purpose Model-Free Reinforcement Learning (Read more on arXiv or HuggingFace) | Michael Rabbat, Yuandong Tian, Amy Zhang, Pierluca D’Oro, Scott Fujimoto | This paper investigates the development of a unified model-free deep reinforcement learning algorithm applicable across diverse environments. The research objective is to identify a single model-free deep RL algorithm that performs well across multiple benchmarks without requiring hyperparameter tuning for each task. The methodology involves leveraging model-based representations to approximately linearize the value function, using a single set of hyperparameters across four benchmarks and 118 environments. Results demonstrate competitive performance against domain-specific and general baselines, with MR.Q achieving competitive performance on the DMC benchmarks. The principal implication is that a single, well-designed model-free algorithm can achieve competitive performance on diverse tasks, reducing the need for extensive hyperparameter tuning and potentially speeding up AI development cycles. Certain aspects of the ablation study results are unclear or lack sufficient detail for complete summarization. |
| ARWKV: Pretrain is not what we need, an RNN-Attention-Based Language Model Born from Transformer (Read more on arXiv or HuggingFace) | Peter Yue, Li Zhiyuan, Lin Yueyu, xiaol | ARWKV introduces an RNN-based language model derived from a Transformer via knowledge distillation, aiming to enhance expressiveness and efficiency. Main research question or objective: How to effectively transform a Transformer-based language model into an RNN-based model while preserving performance and improving efficiency. Key methodology used: A three-stage process involving aligning the hidden state output of the Transformer with an RWKV-7 time mixing module, followed by word-level KL-Divergence knowledge distillation, and concluding with supervised fine-tuning (SFT) and Direct Preference Optimization (DPO). Primary results: The ARWKV model achieved a score of 62.41 on the MMLU benchmark after stage-2 training, demonstrating the feasibility of the transformation. The paper does not clarify whether the ARWKV model outperformed the teacher model on the MMLU benchmark. Principal implication for AI practitioners: Knowledge distillation can be used to transform Transformer models into RNN-based architectures, potentially offering a pathway to developing more efficient language models without extensive pretraining. |
| Emilia: A Large-Scale, Extensive, Multilingual, and Diverse Dataset for Speech Generation (Read more on arXiv or HuggingFace) | Yicheng Gu, Xuyuan Li, Chaoren Wang, Zengqiang Shang, Haorui He | Here is a concise summary of the research paper: The paper introduces Emilia-Pipe, an open-source pipeline for creating speech generation datasets, and Emilia/Emilia-Large, large-scale multilingual datasets derived from in-the-wild speech data. The main research objective is to address the limitations of existing speech generation models trained on audiobook datasets by developing a diverse, spontaneous, and human-like speech dataset. The key methodology involves a six-step preprocessing pipeline (Emilia-Pipe) including standardization, source separation, speaker diarization, fine-grained segmentation, automated speech recognition, and filtering to process raw in-the-wild multilingual speech data. The primary results show that the Emilia dataset, comprising 101k hours of speech across six languages, significantly outperforms traditional audiobook datasets in generating spontaneous and human-like speech, with the Emilia-Test set achieving a DNSMOS score of 3.26. The principal implication for AI practitioners is that the Emilia dataset and Emilia-Pipe provide valuable resources for training speech generation models capable of producing more natural and human-like speech, particularly in diverse real-world contexts. |
| iFormer: Integrating ConvNet and Transformer for Mobile Application (Read more on arXiv or HuggingFace) | Chuanyang Zheng | iFormer is a new family of mobile hybrid vision networks designed for optimized latency and accuracy in mobile applications. The main research objective is to develop a lightweight network that effectively integrates the local representation capacity of convolution and the global modeling ability of self-attention for mobile devices. The key methodology involves transforming a standard convolutional network (ConvNeXt) into a lightweight mobile network and introducing a novel mobile modulation attention mechanism that removes memory-intensive operations in multi-head attention (MHA). The primary result is that iFormer achieves a Top-1 accuracy of 80.4% on ImageNet-1k with a latency of only 1.10 ms on an iPhone 13. The principal implication for AI practitioners is that they can deploy the iFormer architecture to achieve state-of-the-art balance between latency and accuracy in vision tasks on resource-constrained mobile devices. |
| Mixture-of-Mamba: Enhancing Multi-Modal State-Space Models with Modality-Aware Sparsity (Read more on arXiv or HuggingFace) | Luke Zettlemoyer, Ning Dong, Genghan Zhang, Junhong Shen, Weixin Liang | This paper introduces Mixture-of-Mamba, a novel state-space model architecture that enhances multi-modal learning through modality-aware sparsity. The main research question is how to improve the performance and efficiency of multi-modal state-space models (SSMs) by incorporating modality-specific parameterization. The key methodology involves extending the Mixture-of-Transformers approach to SSMs by selectively decoupling projection components in the Mamba block based on input modality, creating a sparse architecture. Primary results show that in the Transfusion setting, Mixture-of-Mamba achieves equivalent image loss using only 34.76% of the training FLOPs at the 1.4B parameter scale compared to dense Mamba models. For AI practitioners, Mixture-of-Mamba offers a more computationally efficient architecture for multi-modal pretraining, allowing for significant reductions in training costs while maintaining or improving performance compared to existing dense models. |
| Feasible Learning (Read more on arXiv or HuggingFace) | Meraj Hashemizadeh, Jose Gallego-Posada, Juan Elenter, Ignacio Hounie, Juan Ramirez | Feasible Learning (FL) is a novel learning paradigm that formulates training machine learning models as a feasibility problem where the loss for each training sample is bounded. The main research question is whether deep networks trained via FL can achieve comparable average performance to Empirical Risk Minimization (ERM) while providing improved tail behavior. The key methodology is a primal-dual approach that dynamically re-weights the importance of each sample during training, and a relaxation called Resilient Feasible Learning (RFL) is introduced to handle potential infeasibility. Primary results show that on CIFAR10, models trained with FL achieved a test accuracy of 0.932 ± 0.002, comparable to ERM’s 0.932 ± 0.002, with FL achieving a minimum Conditional Value at Risk (CVaR) across all loss percentiles, implying better performance on outlier samples. The principal implication is that AI practitioners can use FL as an alternative to ERM to achieve more consistent model performance across all data points, particularly when robustness to outliers is important, without significantly sacrificing average performance. |
Papers for 2025-01-27
| Title | Authors | Summary |
|---|---|---|
| Humanity’s Last Exam (Read more on arXiv or HuggingFace) | Josephina Hu, Nathaniel Li, Ziwen Han, Alice Gatti, Long Phan | Humanity’s Last Exam introduces a new multi-modal benchmark to evaluate large language model capabilities at the forefront of human knowledge. The research objective was to create a challenging, closed-ended benchmark resistant to simple internet retrieval, exceeding the accuracy of state-of-the-art LLMs on existing benchmarks. A multi-stage review process, involving LLM difficulty checks and expert review, was employed to curate 3,000 questions across various subjects. Results showed that all state-of-the-art models achieved less than 10% accuracy, highlighting a significant gap between current LLM capabilities and human expert performance. This benchmark’s creation provides a critical tool for evaluating and guiding future LLM development, demonstrating the limitations of current models on complex academic questions. |
| Redundancy Principles for MLLMs Benchmarks (Read more on arXiv or HuggingFace) | Chunyi Li, Xiangyu Zhao, Zicheng Zhang, KennyUTC, nebulae09 | This paper introduces a framework for evaluating and addressing redundancy in multi-modal large language model (MLLM) benchmarks. The main research question is how to quantify and mitigate redundancy across dimensions, instances, and benchmarks in MLLM evaluation. The key methodology involves calculating the correlation between MLLM performance rankings across different dimensions, instances, and benchmarks using metrics like SRCC, PLCC, and R2. The primary results show that a majority of existing MLLM benchmarks exhibit significant instance redundancy, with over 50% of instances being redundant in many cases, and that the widely used MathVista benchmark displays lower redundancy compared to other math-focused benchmarks. The principal implication for AI practitioners is that they should carefully evaluate and address redundancy in benchmarks to ensure efficient and accurate MLLM evaluation, particularly by checking dimension, instance, and cross-benchmark redundancy. |
| Chain-of-Retrieval Augmented Generation (Read more on arXiv or HuggingFace) | Zhicheng Dou, Xiaolong Huang, Nan Yang, Haonan Chen, Liang Wang | This paper introduces Chain-of-Retrieval Augmented Generation (CoRAG), a novel framework for training large language models (LLMs) to retrieve and reason over information step-by-step. The main research question is whether explicitly training LLMs to iteratively retrieve information can improve their performance on complex, multi-hop reasoning tasks compared to traditional single-step retrieval-augmented generation (RAG) methods. The key methodology involves using rejection sampling to automatically generate intermediate retrieval chains for training and employing various decoding strategies, including greedy decoding, best-of-N sampling, and tree search, to control test-time compute. The primary result is that CoRAG substantially outperforms strong baselines on multi-hop question-answering tasks, achieving more than a 10-point improvement in EM score on the MuSiQue dataset. The principal implication for AI practitioners is that CoRAG offers a more effective approach to retrieval-augmented generation, particularly for complex queries, by enabling dynamic query reformulation and iterative information retrieval. |
| RealCritic: Towards Effectiveness-Driven Evaluation of Language Model Critiques (Read more on arXiv or HuggingFace) | Ruoyu Sun, Tian Ding, Zhenyang Xiao, Ziniu Li, Zhengyang Tang | RealCritic is a new benchmark for evaluating the effectiveness of large language models’ (LLMs) critiques by measuring their impact on solution refinement. The main research question is how to effectively measure the quality of critiques generated by LLMs. The key methodology is a closed-loop approach that evaluates the quality of corrections generated from the critiques, including self-critique, cross-critique, and iterative critique scenarios. The primary results show that the o1-mini model outperforms others in self-critique, with a +3.3% average improvement over direct solutions, while other models show varying or negative performance changes. The principal implication for AI practitioners is that evaluating critique effectiveness through solution improvement provides a more accurate measure of critique quality compared to existing open-loop methods, which is crucial for developing LLMs with robust self-reflection capabilities. |
| Relightable Full-Body Gaussian Codec Avatars (Read more on arXiv or HuggingFace) | Timur Bagautdinov, Igor Santesteban, Tomas Simon, Shaofei Wang, psyth | This paper introduces Relightable Full-Body Gaussian Codec Avatars, a novel approach for modeling and rendering relightable, animatable full-body human avatars with high-fidelity details. The main research question is how to accurately model the relightable appearance of articulated full-body avatars, including body, face, and hands, under various lighting conditions and poses. The key methodology combines 3D Gaussian Splatting with learnable, orientation-dependent zonal harmonics for diffuse radiance transfer, a shadow network to predict non-local shadowing, and deferred shading for specular radiance transfer. The primary results show that the proposed method outperforms existing physically-based rendering approaches, achieving a PSNR of 29.48 dB and an SSIM of 0.8046 on held-out test data, demonstrating superior rendering quality and generalization. For AI practitioners, the principal implication is that this method provides a more accurate and efficient way to create and animate relightable full-body avatars, which can be instrumental for applications in virtual reality, telepresence, and digital human creation. |
Papers for 2025-01-24
| Title | Authors | Summary |
|---|---|---|
| SRMT: Shared Memory for Multi-agent Lifelong Pathfinding (Read more on arXiv or HuggingFace) | Yuri Kuratov, mbur, alsu-sagirova | The research introduces a Shared Recurrent Memory Transformer (SRMT) to enhance coordination in multi-agent systems by enabling implicit information exchange. The main research question is whether a shared recurrent memory mechanism can improve coordination and performance in multi-agent pathfinding tasks. The key methodology involves extending memory transformers to a multi-agent setting by pooling and broadcasting individual working memories, allowing agents to implicitly coordinate actions. Primary results show that SRMT consistently outperforms baselines in a bottleneck navigation task with sparse rewards, achieving a Cooperative Success Rate (CSR) of 1.0 on corridor lengths up to 400 cells. For AI practitioners, SRMT provides a decentralized method to improve coordination in multi-agent systems without relying on explicit communication protocols or centralized control, particularly useful in tasks requiring efficient pathfinding and cooperation. |
| Improving Video Generation with Human Feedback (Read more on arXiv or HuggingFace) | Ziyang Yuan, Jiajun Liang, Gongye Liu, Xintao, jieliu | This paper introduces a framework for aligning video generation models with human preferences using feedback. Main research question or objective: How to improve video generation models by incorporating multi-dimensional human feedback into the training process. Key methodology used: A large-scale human preference dataset was constructed, a multi-dimensional video reward model (VideoReward) was developed, and three alignment algorithms for flow-based models were introduced, including Flow-DPO, Flow-RWR, and Flow-NRG. Primary results: VideoReward significantly outperforms existing reward models, with a 72.89% overall accuracy on GenAI-Bench and 73.59% on VideoGen-RewardBench, and Flow-DPO demonstrates superior performance compared to other methods when a fixed beta is used. Principal implication for AI practitioners: AI practitioners can leverage VideoReward and the Flow-DPO alignment algorithm to enhance the quality and alignment of video generation models with human preferences, particularly by employing a constant beta in Flow-DPO, leading to improved visual quality, motion quality, and text alignment in generated videos. |
| Sigma: Differential Rescaling of Query, Key and Value for Efficient Language Models (Read more on arXiv or HuggingFace) | hanglics, yegong, lx865712528, tzh94588, Lin0 | SIGMA is a large language model specialized for the system domain, featuring a novel DiffQKV attention mechanism for improved inference efficiency. The main research objective is to optimize the Query, Key, and Value components of the attention mechanism in large language models to enhance inference efficiency without significantly compromising performance. The key methodology involves differentially compressing Key and Value components based on their varying impacts on model performance and augmenting the Query component to enhance representation capacity. The primary results show that SIGMA achieves up to a 33.36% improvement in inference speed over the conventional grouped-query attention (GQA) in long-context scenarios, and outperforms GPT-4 with an absolute improvement of up to 52.5% on the AIMICIUS system domain benchmark. The principal implication for AI practitioners is that they can leverage the DiffQKV attention mechanism to develop more efficient large language models, particularly for applications in the system domain, achieving substantial speed improvements and performance gains with strategically optimized attention components. |
| Temporal Preference Optimization for Long-Form Video Understanding (Read more on arXiv or HuggingFace) | Zeyu Wang, yeunglevy, yuhuizhang, nicholswang, ruili0 | Temporal Preference Optimization (TPO) is a post-training framework that enhances the temporal grounding capabilities of video-LMMs through preference learning. The main research question is how to improve the temporal grounding capabilities of video-LMMs for long-form video understanding without relying on extensive manually annotated data. The key methodology is a self-training approach using preference learning with a dataset curated at two granularities (localized and comprehensive temporal grounding) optimized via Direct Preference Optimization (DPO). Primary results show that TPO significantly improves performance on long-form video understanding benchmarks, with LLaVA-Video-TPO achieving a 2.5% performance boost on the Video-MME benchmark. The principal implication for AI practitioners is that TPO offers a scalable and efficient solution for advancing temporal reasoning in long-form video understanding, reducing reliance on manually annotated data. |
| DiffuEraser: A Diffusion Model for Video Inpainting (Read more on arXiv or HuggingFace) | Haolan Xue, Liefeng, lyraestar, asLKHFksasak | DiffuEraser is a diffusion model designed for video inpainting that improves both content completeness and temporal consistency. The main research question is how to enhance video inpainting to generate more detailed textures and maintain temporal consistency across long video sequences. The key methodology involves integrating a motion module into a stable diffusion-based image inpainting model (BrushNet), incorporating priors for initialization and weak conditioning, and expanding the temporal receptive fields during inference. The primary results demonstrate that DiffuEraser outperforms the state-of-the-art video inpainting method, Propainter, in generating content with greater detail and maintaining superior temporal consistency, although specific quantitative metrics are not explicitly provided in the text. For AI practitioners, DiffuEraser provides a new approach to video inpainting that leverages the generative power of diffusion models to fill in missing video content, offering a more robust solution compared to existing transformer-based methods, particularly for long videos with large masks. |
| IMAGINE-E: Image Generation Intelligence Evaluation of State-of-the-art Text-to-Image Models (Read more on arXiv or HuggingFace) | lzyhha, JackyZhuo, RuoyiDu, Afeng-x, jyjyjyjy | IMAGINE-E evaluates the intelligence of six text-to-image (T2I) models across various domains. The main research objective is to benchmark the performance of state-of-the-art T2I models like FLUX.1, Ideogram2.0, Dall-E3, Midjourney, Stable Diffusion 3, and Jimeng across a wide array of tasks. The key methodology involves qualitative and quantitative evaluations using metrics like CLIPScore, HPSv2, Aesthetic Score, and GPT-4o scores across five domains: structured output generation, realism and physical consistency, specific domain generation, challenging scenario generation, and multi-style creation. Primary results indicate that FLUX.1 and Ideogram2.0 generally perform the best, particularly in structured output and specific domain tasks, with FLUX.1 achieving a human evaluation score of 8.89 in the code2table task. The principal implication for AI practitioners is that while current T2I models show promise in specialized tasks, they still face significant challenges in code generation, 3D generation, and producing outputs with Chinese text, highlighting areas for future development. |
| Can We Generate Images with CoT? Let’s Verify and Reinforce Image Generation Step by Step (Read more on arXiv or HuggingFace) | Renrui Zhang, hsli-cuhk, gaopenghigh, zhizhengzhao, ZiyuG | Summary: This paper investigates the application of Chain-of-Thought (CoT) reasoning strategies to autoregressive image generation, proposing methods to verify and reinforce image generation step-by-step. Main research question or objective: Can CoT reasoning strategies, previously explored in large language models (LLMs) and large multimodal models (LMMs), be effectively applied to enhance autoregressive image generation? Key methodology used: The authors systematically investigate three techniques: scaling test-time computation for verification using Outcome/Process Reward Models (ORMs/PRMs), aligning model preferences with Direct Preference Optimization (DPO), and integrating these techniques. They also propose two new reward models, Potential Assessment Reward Model (PARM) and PARM++, tailored for autoregressive image generation. Primary results: Integrating the proposed PARM with iterative DPO improved the baseline model (Show-o) by +24% on the GenEval benchmark, surpassing Stable Diffusion 3 by +15%. Principal implication for AI practitioners: The proposed techniques, particularly the use of PARM and PARM++ for step-wise verification and refinement, offer a novel and effective approach for improving the quality and accuracy of autoregressive image generation models. |
| EchoVideo: Identity-Preserving Human Video Generation by Multimodal Feature Fusion (Read more on arXiv or HuggingFace) | Renjie Chen, Boyuan Liu, Shiyue Yan, Jiangchuan Wei, linwf | EchoVideo is a text-to-video generation model that produces videos of human subjects while preserving their identity from an input image. The main research objective is to generate identity-preserving videos that avoid “copy-paste” artifacts and low similarity issues found in existing methods. The key methodology used is a two-stage training strategy incorporating an Identity Image-Text Fusion Module (IITF) that integrates high-level semantic features from text and a stochastic method to randomly utilize shallow facial information. Primary results show that EchoVideo achieved a dynamic degree score of 0.771 and an aesthetic quality score of 0.601, outperforming the ID-Animator model. The principal implication for AI practitioners is that EchoVideo provides a method for generating high-quality, controllable, and high-fidelity videos, effectively preserving facial identities and maintaining full-body integrity, which is valuable for identity-preserving video generation applications. |
| Step-KTO: Optimizing Mathematical Reasoning through Stepwise Binary Feedback (Read more on arXiv or HuggingFace) | spermwhale, yunhe, sainbar, jindi, yentinglin | Step-KTO is a training framework that improves the mathematical reasoning of large language models (LLMs) using binary feedback on both intermediate steps and final answers. The main research question is whether integrating stepwise process feedback with outcome-level feedback can improve the accuracy and coherence of LLM reasoning in mathematical problem-solving. The key methodology is Stepwise Kahneman-Tversky-inspired Optimization (STEP-KTO), which combines process-level and outcome-level binary feedback using a Kahneman-Tversky-inspired value function to guide model training iteratively. The primary results show that on the MATH-500 dataset, STEP-KTO improves the Pass@1 accuracy of the Llama-3.1-8B-Instruct model from 53.4% to 63.2%. The principal implication for AI practitioners is that incorporating stepwise feedback into the training process can enhance both the final answer accuracy and the intermediate reasoning quality of LLMs, leading to more reliable and interpretable mathematical reasoning systems. |
| Debate Helps Weak-to-Strong Generalization (Read more on arXiv or HuggingFace) | Yongbin-Li, hzhwcmhf, langnick | This paper explores using debate between AI models to improve weak-to-strong generalization in AI alignment. The main research question is whether a strong AI model can be used to improve a weak model’s supervision capabilities, and then use this enhanced supervision to train the strong model. The key methodology involves finetuning a small “weak” model with help from a large “strong” model via debate, and then finetuning the strong model on labels generated by the weak model ensemble. The primary results show that debate ensembles lead to significant improvements in weak-to-strong generalization, with the approach achieving a 76.5% performance gap recovered (PGR) on the SciQ dataset, compared to 41.2% for a baseline. The principal implication for AI practitioners is that using debate to enhance weak model supervision can be a viable strategy for aligning more powerful AI models, especially when direct human supervision becomes infeasible. |
| Evolution and The Knightian Blindspot of Machine Learning (Read more on arXiv or HuggingFace) | Tarin Ziyaee, Kenneth O. Stanley, Tarek El-Gaaly, ekmeyerson, jal278 | Machine learning (ML) overlooks the critical aspect of robustness to qualitative unknowns in open-world environments, termed Knightian uncertainty (KU). The main research question is how ML, particularly reinforcement learning (RL), is limited by its formalisms in addressing Knightian uncertainty, and how biological evolution manages this challenge. The key methodology involves a comparative analysis between RL formalisms, specifically Markov Decision Processes (MDPs), and the principles of biological evolution, highlighting mechanisms like open-ended search, diversification, and persistence. The primary results indicate that RL’s standard objective, maximizing expected return with a discount factor approaching 0 with increasing time steps, leads to indifference to catastrophic events beyond a fixed time horizon. The principal implication for AI practitioners is the need to integrate mechanisms inspired by biological evolution, such as open-endedness and diversification, into ML algorithms to enhance robustness to unforeseen situations, as current formalisms limit this capability. |
| Video-MMMU: Evaluating Knowledge Acquisition from Multi-Discipline Professional Videos (Read more on arXiv or HuggingFace) | ZhangYuanhan, wangxiao1208, pufanyi, craigwu, KairuiHu | Video-MMMU is a benchmark for assessing knowledge acquisition in large multimodal models (LMMs) from educational videos. The main research question is how effectively LMMs can acquire and utilize knowledge from multi-discipline professional videos across three cognitive stages: perception, comprehension, and adaptation. The key methodology involves curating a dataset of 300 expert-level videos and 900 human-annotated questions across six disciplines, evaluating LMMs through stage-aligned question-answer pairs, and proposing a knowledge gain metric (∆knowledge) to quantify performance improvement after video viewing. The primary result is that the best-performing model, GPT-4o, achieved a knowledge gain (∆knowledge) of 15.6% after watching the videos, compared to a human expert’s 33.1%, and model performance declines as cognitive demands increase. The principal implication for AI practitioners is that current LMMs struggle to effectively learn and apply knowledge from videos in a manner comparable to humans, highlighting a critical area for further development to enhance video-based learning capabilities. |
| GSTAR: Gaussian Surface Tracking and Reconstruction (Read more on arXiv or HuggingFace) | Jie Song, Juan Zarate, Chengwei Zheng, lxxue | GSTAR is a novel method for tracking and reconstructing dynamic 3D surfaces with changing topologies using Gaussian Splatting. The main research question is how to achieve photo-realistic rendering, accurate surface reconstruction, and reliable 3D tracking for dynamic scenes where the topology of surfaces changes over time. The key methodology involves binding 3D Gaussians to mesh faces to create “Gaussian Surfaces,” using scene flow warping for frame-to-frame initialization, optimizing Gaussian parameters with fixed topology, then unbinding Gaussians and re-meshing to adapt to topological changes. The primary results show that GSTAR achieves a PSNR of 31.87, SSIM of 0.952, and LPIPS of 0.102 in appearance reconstruction, outperforming comparison methods. For AI practitioners, GSTAR provides a method to generate high-quality appearance and geometry reconstruction with consistent tracking for dynamic scenes, enabling advancements in areas like VR/XR, robotic interactions, and other applications requiring precise 3D representations. |
Papers for 2025-01-23
| Title | Authors | Summary |
|---|---|---|
| DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning (Read more on arXiv or HuggingFace) | AS-7, haha-point, freesky, DejianYang, guoday | DeepSeek-R1 is a series of reasoning models developed using reinforcement learning. Main research question or objective: How to enhance the reasoning capabilities of large language models (LLMs) using reinforcement learning (RL) without supervised fine-tuning (SFT). Key methodology used: A multi-stage training pipeline involving initial fine-tuning on a small amount of cold-start data, followed by reasoning-oriented RL, rejection sampling with supervised fine-tuning, and finally, reinforcement learning for all scenarios, alongside distillation to smaller models. Primary results: DeepSeek-R1 achieved 79.8% Pass@1 on AIME 2024, surpassing OpenAI-o1-1217, and attained an impressive score of 97.3% on MATH-500. Principal implication for AI practitioners: The findings suggest that the distillation of reasoning patterns from larger models into smaller models is highly effective, offering a practical approach for enhancing reasoning abilities in resource-constrained applications. |
| FilmAgent: A Multi-Agent Framework for End-to-End Film Automation in Virtual 3D Spaces (Read more on arXiv or HuggingFace) | Senbao Shi, Li-Zhouyi, PigCatchingExpert, longyuewang, imryanxu | FILMAGENT is an LLM-based multi-agent framework for automated film production in 3D virtual spaces. The main research objective is to automate virtual film production using a collaborative multi-agent approach. The key methodology involves simulating film crew roles (director, screenwriter, actors, cinematographer) with LLM-based agents, using a three-stage workflow (idea development, scriptwriting, cinematography) with Critique-Correct-Verify and Debate-Judge collaboration algorithms. Primary results show that FILMAGENT achieved an average human evaluation score of 3.98 out of 5, outperforming single-agent baselines. The principal implication for AI practitioners is that multi-agent collaboration can significantly enhance the quality of automated film production, offering a viable approach for end-to-end film automation. |
| Test-Time Preference Optimization: On-the-Fly Alignment via Iterative Textual Feedback (Read more on arXiv or HuggingFace) | Yu Cheng, linjieli222, Xiaoye08, huxy912, yaful | Test-time preference optimization (TPO) aligns large language model (LLM) outputs with human preferences during inference without retraining. The research objective was to determine if LLMs could be aligned with human preferences during inference using iterative textual feedback rather than purely numerical rewards. TPO iteratively refines LLM outputs based on textual critiques derived from a reward model’s numerical scores. Evaluation across multiple benchmarks showed TPO progressively improved alignment; for example, the unaligned Llama-3.1-70B-SFT model surpassed its aligned counterpart, Llama-3.1-70B-Instruct, on several metrics after only a few iterations. This work demonstrates a practical, lightweight method for test-time preference optimization, enabling rapid adaptation of LLMs to evolving preferences without retraining, directly impacting AI practitioners by offering a computationally efficient alignment technique. |
| VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understanding (Read more on arXiv or HuggingFace) | Sicong, Guanzheng, Zhiqiang007, ClownRat, CausalLi | VideoLLaMA3 is an advanced multimodal foundation model designed for image and video understanding, emphasizing a vision-centric approach. The main research objective is to develop a more capable model for both image and video understanding by leveraging high-quality image-text data. The key methodology involves a four-stage training paradigm: vision-centric alignment, vision-language pretraining, multi-task fine-tuning, and video-centric fine-tuning, coupled with a vision encoder adapted for dynamic resolution inputs and video token compression. Primary results show that VideoLLaMA3 achieves state-of-the-art performance on several benchmarks, including a 67.1% accuracy on the MathVista testmini dataset. The principal implication for AI practitioners is that focusing on high-quality image-text data and vision-centric training can significantly enhance both image and video understanding capabilities in multimodal models, as demonstrated by VideoLLaMA3’s performance improvements. |
| Kimi k1.5: Scaling Reinforcement Learning with LLMs (Read more on arXiv or HuggingFace) | ChonghuaLiao, DuChenZhuang, shelowize, xingbowei, KbsdJames | Kimi k1.5 is a multi-modal large language model trained with reinforcement learning, featuring enhanced reasoning and long-context processing. The main research objective is to explore scaling reinforcement learning (RL) with large language models (LLMs) to improve performance beyond the limitations of traditional supervised fine-tuning. The key methodology involves long-context scaling up to 128k tokens, improved policy optimization via a variant of online mirror descent, a simplistic RL framework, and multi-modal training on text and vision data. A primary result is that the long-context-of-thought (long-CoT) version achieved 96.2 on the MATH 500 benchmark. The principal implication for AI practitioners is that scaling context length in RL with LLMs, combined with refined optimization techniques, can significantly improve model performance on complex reasoning tasks, offering a viable path for continued advancements in AI capabilities. |
| Autonomy-of-Experts Models (Read more on arXiv or HuggingFace) | Yining Qian, kangzhanhui, shwu, Ruobing-Xie, AngLv | This paper introduces Autonomy-of-Experts (AoE), a novel Mixture-of-Experts (MoE) paradigm where experts autonomously select inputs based on their internal activation norms. The main research question is whether allowing experts to autonomously select inputs based on their internal activation norms can improve upon the traditional MoE model’s expert selection and training effectiveness. The key methodology involves removing routers and having experts pre-compute internal activations for inputs, ranking them by their activation norms, and only forwarding the top-ranking experts for processing. Primary results show that AoE models outperform traditional MoE models in downstream tasks, with a specific finding that a 4B parameter AoE model achieved an average accuracy of 49.80 across various tasks, compared to 48.06 for a comparable traditional MoE model. For AI practitioners, the principal implication is that AoE offers a more efficient and effective approach to training MoE models by eliminating the need for routers and improving expert specialization, directly enhancing downstream performance. |
| Pairwise RM: Perform Best-of-N Sampling with Knockout Tournament (Read more on arXiv or HuggingFace) | Yixin Cao, Rui Min, Zijun Yao, Yantao Liu, juanli | Pairwise Reward Model (Pairwise RM) is introduced to improve Best-of-N (BoN) sampling for Large Language Models (LLMs) through a knockout tournament framework. The main research question is how to effectively select the best candidate solution from multiple LLM-generated outputs without relying on arbitrary and inconsistent reward scores. The key methodology involves training a Pairwise RM to perform pairwise comparisons of candidate solutions’ correctness and using a knockout tournament to iteratively eliminate incorrect solutions. Primary results show that Pairwise RM achieves a 6.7% average improvement on MATH-500 over the strongest baseline. The principal implication for AI practitioners is that Pairwise RM with knockout tournaments offers a more robust mechanism for selecting the best solution in BoN sampling, especially for challenging math problems. |
| O1-Pruner: Length-Harmonizing Fine-Tuning for O1-Like Reasoning Pruning (Read more on arXiv or HuggingFace) | Yibo Wang, Haiying He, Li Shen, cxc361461518, iNk233 | O1-Pruner is a fine-tuning method designed to reduce the inference overhead of long-thought reasoning models while maintaining accuracy. The main research question is how to minimize the reasoning overhead of long-thought Large Language Models (LLMs) without compromising their accuracy. The key methodology is Length-Harmonizing Fine-Tuning (O1-Pruner), which uses pre-sampling and RL-style fine-tuning to encourage shorter reasoning processes under accuracy constraints. The primary results show that O1-Pruner reduces solution length by 40.5% while achieving an average accuracy of 76.8% on the Marco-01-7B model. The principal implication for AI practitioners is that O1-Pruner offers an effective method to optimize long-thought reasoning models, achieving a balance between computational efficiency and high accuracy. |
| IntellAgent: A Multi-Agent Framework for Evaluating Conversational AI Systems (Read more on arXiv or HuggingFace) | Ilankad23, Eladlev | IntellAgent is a multi-agent framework for evaluating conversational AI systems by generating synthetic benchmarks. The main research objective is to develop a scalable, open-source framework that addresses the limitations of manually curated benchmarks for evaluating conversational AI. The key methodology involves a multi-agent pipeline that combines policy-driven graph modeling, realistic event generation, and interactive user-agent simulations. Primary results show a strong correlation (0.98 for Airline, 0.92 for Retail) between model performance on IntellAgent and the T-bench benchmark, despite IntellAgent using only synthetic data. The principal implication for AI practitioners is that IntellAgent provides a robust and detailed evaluation tool for conversational AI, enabling targeted optimization of models across diverse scenarios and policies. |
Papers for 2025-01-22
| Title | Authors | Summary |
|---|---|---|
| Agent-R: Training Language Model Agents to Reflect via Iterative Self-Training (Read more on arXiv or HuggingFace) | Zhengyin Du, Zhiheng Xi, Junjie-Ye, lovesnowbest, siyuyuan | Agent-R is an iterative self-training framework that enables language agents to reflect on and correct their actions in interactive environments. The main research question is whether language model agents can be trained to reflect on their behavior and improve performance via iterative self-training without relying on human or expert model supervision. The key methodology involves using Monte Carlo Tree Search (MCTS) to construct training samples that recover correct trajectories from erroneous ones and a model-guided critique mechanism for timely error revision. The primary result is that agents trained with Agent-R achieved a 70.71% average success rate across three interactive environments, outperforming baseline methods by 5.59%. The principal implication for AI practitioners is that Agent-R offers a method to develop language agents with enhanced self-reflection and error correction capabilities, enabling more robust performance in interactive and agentic environments. |
| MMVU: Measuring Expert-Level Multi-Discipline Video Understanding (Read more on arXiv or HuggingFace) | Lujing Xie, Yilun Zhao, Phil-01, entropyhu, freesky | MMVU is a benchmark for evaluating the expert-level, multi-discipline video understanding capabilities of foundation models. The main research question is how well current multimodal foundation models can understand and reason about specialized-domain videos requiring expert knowledge across multiple disciplines. The key methodology involves creating a dataset of 3,000 expert-annotated examples from 1,529 specialized-domain videos, spanning 27 subjects across four core disciplines, with each example including expert-annotated reasoning rationales and relevant domain knowledge. The primary results show that the best performing model, ol, achieved an accuracy of 77.0% on the test set, significantly below the human expert performance of 86.8% in an open-book setting. The principal implication for AI practitioners is that while current models show promise in expert-level video understanding, there remains a substantial gap compared to human expertise, indicating a need for further development in integrating domain-specific knowledge and reasoning into multimodal models for specialized domains. |
| Demons in the Detail: On Implementing Load Balancing Loss for Training Specialized Mixture-of-Expert Models (Read more on arXiv or HuggingFace) | Kaiyue Wen, Bo Zheng, Zeyu Huang, Zihan Qiu, Losin94 | This paper revisits the implementation of Load-balancing Loss (LBL) in Mixture-of-Experts (MoEs) models. The main research question is how the calculation scope of LBL (micro-batch vs. global-batch) affects the performance and expert specialization of MoE-based large language models (LLMs). The key methodology involves synchronizing expert selection frequency across parallel groups to calculate LBL at the global-batch level and comparing it with the traditional micro-batch approach. The primary results show that global-batch LBL significantly improves model performance, for example by 0.1 in pre-training perplexity in the MoE-3.4A0.6B model, and enhances domain specialization of experts. The principal implication for AI practitioners is that using global-batch LBL can lead to more performant and specialized MoE models during training. |
| UI-TARS: Pioneering Automated GUI Interaction with Native Agents (Read more on arXiv or HuggingFace) | Shihao Liang, Haoming Wang, Junjie Fang, Yining Ye, Yujia Qin | UI-TARS introduces a native GUI agent model that solely uses screenshots as input to perform human-like GUI interactions. The research objective was to develop an end-to-end GUI agent model surpassing existing framework-based models. UI-TARS employed enhanced perception, unified action modeling, system-2 reasoning, and iterative training with reflective online traces. Results showed UI-TARS achieving state-of-the-art performance on multiple benchmarks, including a score of 24.6 on the OSWorld benchmark with 50 steps. This work demonstrates the potential of native GUI agents, suggesting that data-driven approaches can outperform framework-based methods for GUI interaction. |
| Mobile-Agent-E: Self-Evolving Mobile Assistant for Complex Tasks (Read more on arXiv or HuggingFace) | Ming Yan, Xi Zhang, Junyang Wang, xhyandwyy, mikewang | Mobile-Agent-E is a hierarchical multi-agent mobile assistant framework with a self-evolution module that improves task performance and efficiency on complex real-world mobile tasks. The research objective was to address limitations of existing mobile agents, namely their struggles with reasoning-intensive tasks and lack of learning from experience. Mobile-Agent-E employs a hierarchical architecture separating high-level planning from low-level action execution and a self-evolution module learning reusable shortcuts and general tips. Results showed a 22% absolute improvement in satisfaction score over previous state-of-the-art approaches using GPT-40. The most impactful finding, a substantial performance gain, directly suggests the efficacy of hierarchical multi-agent frameworks and self-evolution mechanisms for improving mobile agent capabilities. |
| TokenVerse: Versatile Multi-concept Personalization in Token Modulation Space (Read more on arXiv or HuggingFace) | Shiran Zada, Omer Tov, Roni Paiss, Shahar Yadin, Daniel Garibi | TokenVerse is a method for multi-concept personalization in text-to-image diffusion models, enabling disentangled control over diverse visual elements extracted from single or multiple images. The main research question is how to achieve versatile and disentangled multi-concept personalization and composition in diffusion transformers. The key methodology involves optimizing per-token directions in the modulation space of a Diffusion Transformer (DiT) model to learn and compose visual concepts described by text tokens. Primary results show that TokenVerse outperforms existing methods, achieving a Concept Preservation score of 0.470108 and Prompt Fidelity score of 0.688061 in the composition task, while other methods score lower on at least one of these metrics. The principal implication for AI practitioners is that TokenVerse provides a more effective way to personalize and control the generation of complex images with multiple concepts, offering advantages in creative control and content customization compared to existing methods, especially for those working with DiT-based text-to-image models. |
| Video Depth Anything: Consistent Depth Estimation for Super-Long Videos (Read more on arXiv or HuggingFace) | Zilong Huang, Feihu Zhang, Shengnan Zhu, Hengkai Guo, Sili Chen | Video Depth Anything is a new method for producing temporally consistent depth estimations for arbitrarily long videos. The main research question is whether it is possible to achieve temporal stability in depth estimation for arbitrarily long videos while inheriting the capabilities of existing depth foundation models. The key methodology involves replacing the head of the Depth Anything V2 model with a spatial-temporal head and using a temporal gradient matching loss during training, along with a key-frame-based strategy for inference. The primary results show that the proposed model, Video Depth Anything, achieves state-of-the-art zero-shot video depth estimation, outperforming all baselines on temporal consistency across five datasets and achieving a Temporal Alignment Error (TAE) of 0.570 on the NYUv2 dataset. The principal implication for AI practitioners is that this model offers a new state-of-the-art approach for video depth estimation that maintains quality, consistency, and generalization ability without sacrificing efficiency, even for videos of several minutes in length. |
| Hunyuan3D 2.0: Scaling Diffusion Models for High Resolution Textured 3D Assets Generation (Read more on arXiv or HuggingFace) | Haolin Liu, Yunfei Zhao, Qingxiang Lin, Zeqiang Lai, Zibo Zhao | Hunyuan3D 2.0 is an open-source system for generating high-resolution textured 3D assets from images using diffusion models. The main research objective is to develop a scalable 3D asset creation system that outperforms existing models in geometry details, condition alignment, and texture quality. The key methodology involves a two-stage pipeline: first, a shape generation model (Hunyuan3D-DiT) based on a flow-based diffusion transformer creates a bare mesh from an input image; second, a texture synthesis model (Hunyuan3D-Paint) generates a high-resolution texture map for the mesh. Primary results show that Hunyuan3D-ShapeVAE achieved a 93.6% volume Intersection of Union (V-IoU) in shape reconstruction, surpassing other models. The principal implication for AI practitioners is that Hunyuan3D 2.0 provides a strong foundation for large-scale 3D generative models, offering pre-trained weights and code for practical application in generating high-fidelity 3D assets. |
| Learn-by-interact: A Data-Centric Framework for Self-Adaptive Agents in Realistic Environments (Read more on arXiv or HuggingFace) | Tao Yu, Pengcheng Yin, Jinsung Yoon, Ruoxi Sun, Hongjin Su | Learn-by-interact is a data-centric framework for training LLM-based agents without human annotations. The main research question is how to adapt large language models (LLMs) to new environments without human annotations. The key methodology used is “backward construction,” which synthesizes agent-environment interaction trajectories from documentation and constructs instructions by summarizing interaction histories. Primary results show that using this method, the baseline results are improved by up to 12.2% for in-context learning (ICL) with Claude-3.5-sonnet and 19.5% for training with Codestral-22B. The principal implication for AI practitioners is that they can use this framework to adapt LLMs to new environments efficiently, significantly reducing the reliance on manually annotated data. |
| Reasoning Language Models: A Blueprint (Read more on arXiv or HuggingFace) | Afonso Catarino, Ales Kubicek, Eric Schreiber, Julia Barth, Maciej Besta | Reasoning Language Models (RLMs) integrate large language models (LLMs) with reasoning mechanisms to enhance AI problem-solving. The main research question is: What is the detailed design of an RLM, and how can it achieve effectiveness, low cost, and scalability? The key methodology is a modular blueprint organizing RLM components, including reasoning structures (chains, trees, graphs), strategies (e.g., Monte Carlo Tree Search), reinforcement learning concepts, and supervision schemes, along with mathematical formulations and algorithmic specifications. A primary result is that the blueprint can model various existing RLMs, such as LLaMA-Berry and QwQ, as special cases, although specific quantitative performance metrics are not provided in the summary. The principal implication for AI practitioners is that the blueprint and the x1 framework provide tools for RLM development, experimentation, and analysis, potentially democratizing advanced reasoning capabilities. |
| Condor: Enhance LLM Alignment with Knowledge-Driven Data Synthesis and Refinement (Read more on arXiv or HuggingFace) | Chuyu Zhang, Mo Li, Taolin Zhang, Maosong Cao, zsytony | Condor is a two-stage framework for generating synthetic data to enhance the conversational capabilities of large language models (LLMs). The main research question is whether a novel knowledge-driven data synthesis and refinement framework can improve LLM alignment and performance on human-preference benchmarks. The key methodology involves constructing a World Knowledge Tree to generate diverse prompts, synthesizing question-answer pairs, and using Self-Reflection Refinement to improve response quality. The primary results show that a model fine-tuned on 20K Condor-generated samples achieved an average human-preference score of 61.29, judged by GPT4o-0806, surpassing the official model’s score of 58.02. The principal implication for AI practitioners is that leveraging the Condor framework to generate high-quality synthetic data can significantly enhance LLM performance in subjective chat evaluations, even with relatively small datasets. |
| EMO2: End-Effector Guided Audio-Driven Avatar Video Generation (Read more on arXiv or HuggingFace) | Liefeng Bo, Bang Zhang, Qi Wang, Siqi Hu, Linrui Tian | EMO2 proposes a novel two-stage audio-driven talking head video generation method focusing on co-speech gesture generation. The research objective was to address the weak correspondence between audio and full-body gestures by generating hand poses directly from audio in the first stage, followed by video frame synthesis using a diffusion model in the second stage. The proposed method outperformed state-of-the-art approaches, such as CyberHost and Vlogger, in terms of visual quality and synchronization accuracy, with specific quantitative results showing an improvement in Diversity (DIV) scores. This work provides a robust framework for creating expressive and natural talking head animations, particularly relevant for AI practitioners working on audio-visual synchronization and diffusion model applications. The paper does not provide a clear description of the specific quantitative improvement in all metrics across all datasets. |
| GPS as a Control Signal for Image Generation (Read more on arXiv or HuggingFace) | Andrew Owens, Alexei A. Efros, Aleksander Holynski, Ziyang Chen, chfeng | The paper introduces GPS conditioning as a novel control signal for image generation and 3D reconstruction using diffusion models. The main research question is whether GPS tags in photo metadata can be used to generate images that accurately reflect location-specific visual characteristics and to extract 3D models from 2D images. The key methodology involves training diffusion models conditioned on GPS coordinates and text prompts, and using GPS-guided score distillation sampling for 3D reconstruction. The primary results show that the method achieves an average CLIP score and GPS score of 18.02, outperforming baseline methods, and that angle-to-image diffusion models achieve 22.36% accuracy in generating images with the correct azimuth. The principal implication for AI practitioners is that GPS conditioning offers a new and effective way to control image generation and perform 3D reconstruction, leveraging the readily available geospatial information in photo metadata. |
| MSTS: A Multimodal Safety Test Suite for Vision-Language Models (Read more on arXiv or HuggingFace) | Alicia Parrish, Janis Goldzycher, Felix Friedrich, Giuseppe Attanasio, Paul Röttger | This paper introduces MSTS, a Multimodal Safety Test Suite for evaluating the safety of Vision-Language Models (VLMs). The main research question is how to assess the novel safety risks posed by VLMs due to their multimodal inputs. The key methodology is the creation of 400 multimodal test prompts across 40 hazard categories, where each prompt’s unsafe meaning is only evident when both image and text are combined. A primary result is that commercial VLMs were found to be very safe with less than 0.5% unsafe responses on average, whereas the least safe open VLM, xGen-MM, responded unsafely to 14.0% of test prompts. The principal implication for AI practitioners is that MSTS can be used to identify safety issues in VLMs, particularly highlighting safety disparities between open and commercial models and across different languages. |
| InternLM-XComposer2.5-Reward: A Simple Yet Effective Multi-Modal Reward Model (Read more on arXiv or HuggingFace) | Ziyu Liu, Yuhang Cao, Pan Zhang, Xiaoyi Dong, Yuhang Zang | InternLM-XComposer2.5-Reward is a multi-modal reward model designed to align large vision-language models (LVLMs) with human preferences. The main research question is how to create an effective multi-modal reward model for LVLMs that can handle diverse modalities and domains. The key methodology involves constructing a multi-modal preference dataset and training the model on this data by augmenting an existing LVLM (InternLM-XComposer2.5) with a scoring head. A primary result is that InternLM-XComposer2.5-Reward achieved a 70.0% Macro Accuracy on the VL-RewardBench benchmark. The principal implication for AI practitioners is that they can use this model to improve the quality of multi-modal chat, follow user instructions, and filter noisy or low-quality samples from pre-training and post-training datasets. |
Papers for 2025-01-21
| Title | Authors | Summary |
|---|---|---|
| GameFactory: Creating New Games with Generative Interactive Videos (Read more on arXiv or HuggingFace) | Yiran Qin, XihuiLiu, di-zhang-fdu, Xintao, VictorYuki | GameFactory is a framework for generating new, open-domain game videos with action controllability using pre-trained video diffusion models. The main research objective is to achieve scene generalization in game video generation, enabling the creation of entirely new game environments beyond existing game styles. The key methodology involves a multi-phase training strategy that decouples game style learning from action control, utilizing a new action-annotated dataset (GF-Minecraft) derived from Minecraft. Primary results show that the model can generate diverse, action-controllable game videos in open domains, with a Flow-MSE of 54.13 for open-domain video generation using multi-phase training. The principal implication for AI practitioners is that this framework enables the development of generative game engines capable of creating new games with diverse scenes, leveraging pre-trained video models and a relatively small amount of action-annotated game data. |
| VideoWorld: Exploring Knowledge Learning from Unlabeled Videos (Read more on arXiv or HuggingFace) | Bingyi Kang, Yao Zhao, Xun Guo, Yunchao Wei, maverickrzw | VideoWorld is an autoregressive video generation model that learns complex knowledge from unlabeled video data. The main research question is whether a deep generative model can learn complex knowledge, including rules, reasoning, and planning, solely from visual input. The key methodology involves training a transformer-based model on unlabeled videos of Go games and robotic manipulation tasks, using a Latent Dynamics Model (LDM) to represent visual changes compactly. The primary results show that VideoWorld achieves a 5-dan professional level in Go with a 300-million-parameter model and generalizes across environments in robotic control tasks, achieving 88.1 action accuracy. The principal implication for AI practitioners is that training video generation models on unlabeled visual data can be a viable approach for acquiring complex knowledge and control policies, demonstrating strong performance and generalization capabilities without relying on text-based training or reward mechanisms. |
Papers for 2025-01-20
| Title | Authors | Summary |
|---|---|---|
| Evolving Deeper LLM Thinking (Read more on arXiv or HuggingFace) | Shumeet Baluja, Dave Marwood, Yueh-Hua Wu, Ian Fischer, Kuang-Huei Lee | Mind Evolution, an evolutionary search strategy, improves large language model (LLM) problem-solving. The research aimed to enhance LLM problem-solving abilities by leveraging inference time compute. Mind Evolution uses an LLM to generate, recombine, and refine candidate solutions based on evaluator feedback, avoiding formal problem representation. Results show Gemini 1.5 Flash achieving a 95.6% success rate on the TravelPlanner benchmark using Mind Evolution, significantly outperforming other methods. This approach enables efficient exploration of the solution space in natural language tasks, offering a valuable strategy for LLM application development. |
| PaSa: An LLM Agent for Comprehensive Academic Paper Search (Read more on arXiv or HuggingFace) | Yuchen Zhang, Yuan Lin, Peiyuan Feng, Guanhua Huang, Yichen He | PaSa is a large language model (LLM) based agent designed for comprehensive academic paper search. The main research question is whether an LLM agent can autonomously conduct comprehensive and accurate academic paper searches, mimicking human-like behavior. The key methodology involves using two LLM agents, a “Crawler” and a “Selector,” optimized with reinforcement learning on a synthetic dataset, AutoScholarQuery, containing 35k fine-grained academic queries. The primary results show that PaSa-7B surpasses the Google with GPT-40 baseline by 37.78% in recall@20 and 39.90% in recall@50 on the RealScholarQuery benchmark. The principal implication for AI practitioners is that PaSa provides a more effective tool for academic literature search, significantly improving search accuracy and recall compared to existing search engines and other LLM-based approaches. |
| Textoon: Generating Vivid 2D Cartoon Characters from Text Descriptions (Read more on arXiv or HuggingFace) | Liefeng Bo, Jianqiang Ren, Chao He | Textoon generates diverse, animatable 2D cartoon characters from text descriptions using a novel Live2D-based framework. The research objective is to develop a method for generating high-quality, interactive 2D cartoon characters from text prompts, overcoming the limitations of existing Live2D creation methods. The methodology combines a fine-tuned large language model (LLM) for accurate text parsing, a text-to-image diffusion model (Stable Diffusion) for controllable appearance generation, an image editing technique for re-editing, and a component completion and repair module. ARKit’s face blendshapes are integrated for improved animation. The primary result is achieving >90% accuracy in parsing component categories from complex input text at millisecond speeds using 4GB of memory (RTX 4090). The system can generate a new character within one minute. The most impactful finding is the creation of a method for generating Live2D characters from text prompts in under one minute, enhancing efficiency in 2D character creation and potentially impacting workflows for game developers, animators, and other creative professionals. |
| Multiple Choice Questions: Reasoning Makes Large Language Models (LLMs) More Self-Confident Even When They Are Wrong (Read more on arXiv or HuggingFace) | Pedro Reviriego, Gonzalo Martínez, Javier Conde, Tairan Fu, mariagrandury | This paper investigates how prompting techniques affect LLM confidence in multiple-choice question responses. The research objective was to determine if LLMs exhibit altered confidence levels when prompted to provide reasoning before selecting an answer, compared to directly answering. The study employed two prompting methods: direct answer and chain-of-thought (CoT), evaluating seven different LLMs on the MMLU benchmark. Results indicated that LLMs demonstrated higher confidence (average probability of selected option increased) with CoT prompts, regardless of answer correctness. For example, the increase in average confidence was larger for incorrect answers than for correct answers. The principal implication is that LLM-estimated probabilities may have intrinsic limitations, impacting their use in evaluation procedures and highlighting a potential mismatch between confidence and accuracy. Further research is needed to clarify how to leverage LLM confidence estimates effectively. |
| HiFi-SR: A Unified Generative Transformer-Convolutional Adversarial Network for High-Fidelity Speech Super-Resolution (Read more on arXiv or HuggingFace) | Chong Zhang, Yukun Ma, Zexu Pan, Kun Zhou, Shengkui Zhao | HiFi-SR proposes a unified generative adversarial network for high-fidelity speech super-resolution. The research objective was to improve speech super-resolution (SR) by addressing limitations of existing methods that use independently trained networks. The methodology involved a unified transformer-convolutional generator trained end-to-end, incorporating a multi-band, multi-scale time-frequency discriminator and mel-reconstruction loss. Results showed HiFi-SR significantly outperformed existing methods, achieving an average log-spectral distance (LSD) of 0.82 on the VCTK test set, improving upon the baseline NVSR model’s LSD of 0.85. This demonstrates the effectiveness of a unified network architecture for high-fidelity speech SR, providing a more robust and generalizable approach for AI practitioners developing speech enhancement technologies. |
| X-Dyna: Expressive Dynamic Human Image Animation (Read more on arXiv or HuggingFace) | Zhengfei Kuang, Yipeng Gao, You Xie, Hongyi Xu, Boese0601 | X-Dyna introduces a zero-shot, diffusion-based pipeline for animating a single human image using facial expressions and body movements from a driving video. The research objective was to create a method for realistic, context-aware dynamic human image animation addressing shortcomings in existing approaches. The methodology employed a diffusion UNet backbone with a novel Dynamics-Adapter module integrating reference appearance context into spatial attentions, coupled with a local face control module for expression transfer. Quantitative results demonstrated that X-Dyna outperforms state-of-the-art methods, achieving a 0.900 FG-DTFVD score compared to scores ranging from 1.753 to 2.639 for other methods. This research significantly advances the field of human image animation offering a more efficient and effective method for realistic video generation which directly improves the quality and realism of animated videos. |
| GaussianAvatar-Editor: Photorealistic Animatable Gaussian Head Avatar Editor (Read more on arXiv or HuggingFace) | Yuan Liu, Qi Zhang, Heng Li, Kunming Luo, Xiangyue Liu | GaussianAvatar-Editor introduces a novel framework for text-driven editing of animatable 3D Gaussian head avatars. The research objective was to develop a method for fully controllable text-driven editing of animatable Gaussian head avatars, addressing challenges of motion occlusion and spatiotemporal inconsistency. The methodology employed a Weighted Alpha Blending Equation (WABE) for anti-occlusion and conditional adversarial learning to ensure 4D consistency. Quantitative results demonstrated that the proposed method achieved superior CLIP-S scores (0.275) compared to baselines (e.g., INSTA+I-N2N, 0.181) in novel view rendering. This work provides AI practitioners with a novel approach to high-quality, consistent 4D Gaussian head avatar editing, directly applicable to applications such as virtual and augmented reality. |
| ComplexFuncBench: Exploring Multi-Step and Constrained Function Calling under Long-Context Scenario (Read more on arXiv or HuggingFace) | Jie Tang, Haiyi Hu, Xiaohan Zhang, Zhengxiao Du, Lucen Zhong | ComplexFuncBench is a benchmark for evaluating large language models’ (LLMs) complex function-calling capabilities. The research aimed to evaluate LLMs’ ability to handle multi-step, constrained function calls within a long-context (128k tokens) setting. The authors developed ComplexEval, an automated evaluation framework using a multi-dimensional matching approach to assess function call correctness. Results showed that even leading closed-source models achieved only a 61% success rate on complex function calls. This highlights a significant deficiency in current LLMs’ ability to manage complex real-world API interactions, emphasizing the need for further research into robust and efficient LLM function-calling capabilities for production-level applications. |
| Bridging Language Barriers in Healthcare: A Study on Arabic LLMs (Read more on arXiv or HuggingFace) | Ronnie Rajan, Marco AF Pimentel, Clément Christophe, Tathagata Raha, Nada Saadi | This paper investigates the challenges of developing effective Arabic LLMs for clinical tasks. The main objective was to determine optimal strategies for training LLMs proficient in both multilingual understanding and medical knowledge, focusing on Arabic. The researchers employed a methodology combining translation of existing English medical datasets into Arabic, synthetic data generation, and fine-tuning Llama 3.1 with varying ratios of Arabic and English data. Results showed that Llama 3.1 achieved significantly lower accuracy on Arabic medical benchmarks (29.5% on MedQA) compared to English (62.0% on MedQA); optimal language ratios varied across tasks. For AI practitioners, the study highlights the limitations of solely relying on translation and fine-tuning for low-resource languages in specialized domains; more computationally intensive pretraining techniques may be necessary for optimal multilingual medical LLM performance. |
Papers for 2025-01-17
| Title | Authors | Summary |
|---|---|---|
| OmniThink: Expanding Knowledge Boundaries in Machine Writing through Thinking (Read more on arXiv or HuggingFace) | Ningyu, Runnaning, callanwu, JizhanFang, ZekunXi | OmniThink is a novel machine writing framework that emulates human-like iterative expansion and reflection to enhance the quality of generated long-form articles. The main research question is whether simulating the cognitive behavior of learners through continuous reflection and exploration can improve the knowledge density and quality of machine-generated articles. The key methodology involves an iterative process of expansion, using search engines to retrieve information and construct an information tree, and reflection, refining retrieved information and updating a conceptual pool to guide further expansion. Primary results show that OmniThink achieved a knowledge density of 22.31 when using GPT-4o as a backbone, surpassing the Co-STORM model’s knowledge density of 19.53. The principal implication for AI practitioners is that incorporating iterative expansion and reflection processes in machine writing can enhance the information density and novelty of generated content without compromising coherence or depth. |
| Inference-Time Scaling for Diffusion Models beyond Scaling Denoising Steps (Read more on arXiv or HuggingFace) | mingdazhang, ycsu, hexianghu, S8T, willllis | This paper explores inference-time scaling for diffusion models by optimizing the sampling process through noise search. The main research question is how to improve the generation performance of diffusion models by increasing computation during inference beyond simply increasing denoising steps. The key methodology involves formulating the search for optimal initial noise as a search problem, using verifiers to evaluate candidates and algorithms to refine noise candidates iteratively. The primary results show that increasing inference-time compute via search significantly improves sample quality, with a 3.6% relative improvement in the LLM Grader metric when using the Verifier Ensemble on the DrawBench dataset with 3840 NFEs allocated to search. The principal implication for AI practitioners is that allocating computational resources to noise search during inference can substantially enhance the performance of diffusion models across various tasks, offering a new avenue for scaling beyond training-time optimization. |
| Exploring the Inquiry-Diagnosis Relationship with Advanced Patient Simulators (Read more on arXiv or HuggingFace) | Quan Tu, hsaest, ShizhengLi, sdujq, zhaocheng | This paper investigates the relationship between inquiry and diagnosis in online medical consultations using AI patient simulators. The main research question is how the quality of inquiries generated by different doctor models impacts diagnostic accuracy in a simulated online medical consultation setting. The key methodology involved training a patient simulator on synthesized doctor-patient dialogues, then using it to evaluate the inquiry-diagnosis relationship by interacting with various doctor models and assessing subsequent diagnostic accuracy. A primary result was that inquiries generated by the Claude model had consistently lower diagnostic accuracy compared to other models such as GPT-40, with Claude achieving 43.9% accuracy after 5 inquiry rounds compared to GPT-40’s 48.1% when diagnosed by the 01-preview model. The principal implication for AI practitioners is that the quality of inquiries significantly affects diagnostic accuracy, suggesting that developing models with robust inquiry capabilities is crucial for effective AI-driven medical diagnosis. |
| SynthLight: Portrait Relighting with Diffusion Model by Learning to Re-render Synthetic Faces (Read more on arXiv or HuggingFace) | Jingyuan Liu, Yannick Hold-Geoffroy, Sumit Chaturvedi, zhixinshu, mengweir | SynthLight is a diffusion model for portrait relighting that learns to re-render synthetic faces based on changes in environmental lighting conditions. The main research question is how to effectively model portrait relighting as a re-rendering problem using synthetic data and a diffusion model, while bridging the domain gap between synthetic and real images. The key methodology involves training a diffusion model on synthetic portrait pairs generated with a physically-based rendering engine, employing multi-task training with real human portraits, and using an inference-time diffusion sampling procedure based on classifier-free guidance. The primary results show that SynthLight achieves comparable or superior quantitative results to state-of-the-art methods on Light Stage data, with a LPIPS score of 0.165 on the Light Stage test set, and user studies indicate superior visual quality, lighting, and identity preservation. The principal implication for AI practitioners is that SynthLight demonstrates the feasibility of using synthetic data to train a diffusion model for high-quality portrait relighting, offering a viable alternative to methods relying on real-world labeled data, such as Light Stage data. |
| FAST: Efficient Action Tokenization for Vision-Language-Action Models (Read more on arXiv or HuggingFace) | oier-mees, dannydriess, brianichter, kylestach, KarlP | This paper introduces FAST, a new action tokenization method for training vision-language-action (VLA) models based on the discrete cosine transform (DCT). The main research objective is to develop an action tokenization scheme that enables efficient training of autoregressive VLA policies on high-frequency and highly dexterous robot action data. The key methodology involves applying DCT to action sequences, quantizing the resulting coefficients, and compressing them using byte-pair encoding (BPE). The primary results show that VLA models trained with FAST achieve comparable performance to state-of-the-art diffusion-based models while reducing training time by up to 5x. The principal implication is that AI practitioners can use FAST as an efficient and effective action tokenizer to train high-performing autoregressive VLA models for robotic control, especially for tasks requiring high-frequency actions. |
| Learnings from Scaling Visual Tokenizers for Reconstruction and Generation (Read more on arXiv or HuggingFace) | David Yan, Philippe Hansen-Estruch, endernewton, Tingbo, orrzohar | Here is a concise summary of the research paper: i) The paper explores scaling properties of Transformer-based auto-encoders, termed ViTok, for visual tokenization in image and video reconstruction and generation tasks. ii) The main research objective is to investigate how design choices and scaling of auto-encoder components influence reconstruction and downstream generative performance. iii) The key methodology involves replacing convolutional backbones with a Vision Transformer (ViT) architecture enhanced with Llama, training on large-scale image and video datasets, and systematically scaling the bottleneck size, encoder, and decoder to analyze their impacts. iv) A primary result is that scaling the bottleneck size E to 8192 for ViTok S-B/16 achieves a rFID score of 0.8 on 256p image reconstruction, but increasing E beyond an optimal point degrades generative performance. v) For AI practitioners, the principal implication is that scaling the decoder while optimizing the bottleneck size E enhances reconstruction performance, but scaling the encoder does not consistently improve reconstruction or generation, which indicates the importance of focusing scaling efforts on the decoder and bottleneck. |
| RLHS: Mitigating Misalignment in RLHF with Hindsight Simulation (Read more on arXiv or HuggingFace) | Jaime Fernández Fisac, Thomas L. Griffiths, Ryan Liu, Haimin Hu, kaiquliang | Generative AI systems can be aligned with human values by using Reinforcement Learning from Hindsight Simulation (RLHS), a novel method introduced to improve upon Reinforcement Learning from Human Feedback (RLHF). The main research question is whether decoupling human feedback from the prediction of downstream outcomes can mitigate misalignment in RLHF. The key methodology used is hindsight simulation, where evaluators are shown simulated downstream outcomes of an interaction before providing feedback on model behavior. The primary result is that RLHS consistently outperforms RLHF in human user studies, with models trained using RLHS achieving a higher true utility score (0.43) compared to RLHF models (-0.16). The principal implication for AI practitioners is that using hindsight simulation during training can significantly reduce model misalignment with human values, leading to more truthful and helpful AI assistants. |
| Towards Large Reasoning Models: A Survey of Reinforced Reasoning with Large Language Models (Read more on arXiv or HuggingFace) | Ouyangtj, zhazhahui7, berserkerko, zzfoutofspace, haohao11 | Large language models (LLMs) are being enhanced through reinforcement learning to improve their reasoning capabilities for complex tasks. The main research objective is to develop methods for training and deploying LLMs as “Large Reasoning Models” capable of advanced, human-like reasoning. Key methodologies include automated data construction via process reward models (PRMs), reinforcement learning from AI feedback (RLAIF), and test-time scaling with PRM-guided search. Primary results show that the “01” model series achieves 83.3% success in competitive programming through structured analytical approach and knowledge integration, demonstrating significant improvements in reasoning tasks. The principal implication for AI practitioners is that integrating “thought” sequences and scaling computation during both training and test times can substantially enhance LLMs’ reasoning abilities, paving the way for more powerful reasoning AI systems. |
| AnyStory: Towards Unified Single and Multiple Subject Personalization in Text-to-Image Generation (Read more on arXiv or HuggingFace) | Junjie He, Liefeng, gengyifeng, ashui, tuoyuxiang | Here is a concise summary of the research paper “AnyStory: Towards Unified Single and Multiple Subject Personalization in Text-to-Image Generation”: i) AnyStory is a unified framework for generating personalized images of single or multiple subjects from text prompts while preserving subject fidelity and alignment with descriptions. ii) The main research objective is to develop a method for high-fidelity personalized text-to-image generation that can handle both single and multiple subjects without blending or sacrificing details. iii) The key methodology involves an “encode-then-route” approach, using a simplified ReferenceNet combined with a CLIP vision encoder for subject encoding and a decoupled instance-aware subject router for guiding subject condition injection during the denoising process. iv) The primary results show that AnyStory effectively preserves subject details, aligns with text descriptions, and personalizes multiple subjects; the simplified ReferenceNet achieves a speed of 53.2 ms/img with 2.02 billion parameters. v) For AI practitioners, AnyStory offers a method to generate high-fidelity personalized images with multiple subjects, directly improving the development of applications requiring precise control over subject representation in text-to-image generation. |
| CaPa: Carve-n-Paint Synthesis for Efficient 4K Textured Mesh Generation (Read more on arXiv or HuggingFace) | Junyoung Choi, Jeong A Wi, Seongyeong Lee, Hwan Heo, longshiine | CaPa: Carve-n-Paint Synthesis for Efficient 4K Textured Mesh Generation is a framework for generating high-fidelity 3D assets from textual or visual inputs. The main research objective is to develop a method for generating high-quality 3D assets that overcomes challenges like multi-view inconsistency, slow generation times, low fidelity, and surface reconstruction problems. The key methodology involves a two-stage process: (1) a 3D latent diffusion model guided by multi-view inputs to generate geometry and (2) a model-agnostic Spatially Decoupled Attention framework to synthesize high-resolution textures, followed by a 3D-aware occlusion inpainting algorithm. The primary results demonstrate that CaPa generates high-quality 3D assets in under 30 seconds, achieving a CLIP score of 86.34 and an FID score of 47.56, outperforming existing methods. For AI practitioners, CaPa provides an efficient pipeline to generate high-quality textured 3D meshes ready for commercial applications, representing a significant advancement in practical, scalable 3D asset generation. |
| Do generative video models learn physical principles from watching videos? (Read more on arXiv or HuggingFace) | Priyank Jaini, Laura Culp, rgeirhos, kswersky, sam-motamed | This research investigates whether generative video models acquire an understanding of physical principles from video data. The main research question is: Do generative video models learn the physical principles that underpin reality from passively “watching” videos? The key methodology involves creating a benchmark dataset, Physics-IQ, to test models’ ability to predict video continuations that require understanding physics, such as solid mechanics, fluid dynamics, and optics. The primary results show that current video models, including Sora and Runway Gen 3, exhibit limited physical understanding, with the best model achieving only a 24.1% Physics-IQ score, where 100% represents the upper bound based on physical variance in real-world videos. The principal implication for AI practitioners is that generating visually realistic videos does not equate to understanding the underlying physical principles, suggesting a need for new methods to incorporate physics into video generation models. |
Papers for 2025-01-16
| Title | Authors | Summary |
|---|---|---|
| MMDocIR: Benchmarking Multi-Modal Retrieval for Long Documents (Read more on arXiv or HuggingFace) | Ruiming Tang, Dexun Li, Xin Deik Goh, Yujing Chang, daviddongdong | MMDocIR introduces a new benchmark for multi-modal document retrieval focusing on long documents. The research objective was to create a robust benchmark dataset for evaluating multi-modal document retrieval systems, addressing shortcomings in existing benchmarks. The methodology involved creating a dataset (MMDocIR) with two tasks: page-level and layout-level retrieval, and using expertly-annotated labels for 1,685 questions. Results showed that visual retrievers significantly outperformed text-based counterparts, with visual methods achieving a Recall@k of 86.0 vs. 72.3 for DPR-Phi3ours and Colbert respectively in page-level retrieval at k=5. This highlights the importance of incorporating visual information for enhanced multi-modal document retrieval, providing a valuable benchmark for AI practitioners developing and evaluating such systems. |
| CityDreamer4D: Compositional Generative Model of Unbounded 4D Cities (Read more on arXiv or HuggingFace) | liuziwei7, hongfz16, FrozenBurning, hzxie | CityDreamer4D is a compositional generative model for unbounded 4D city generation. The research objective was to develop a model capable of generating realistic and temporally consistent 4D city scenes with diverse objects and unbounded extents. The methodology employed a compositional approach, separating dynamic (vehicles) and static (buildings, roads) scene elements, using distinct neural fields for each object type. Results showed CityDreamer4D achieved a Fréchet Inception Distance (FID) of 96.83 and a Kernel Inception Distance (KID) of 0.096 on the Google Earth dataset, significantly outperforming existing methods. This research provides AI practitioners with a novel architecture for generating high-fidelity 4D scenes, potentially impacting applications in urban planning, game development, and metaverse creation. |
| RepVideo: Rethinking Cross-Layer Representation for Video Generation (Read more on arXiv or HuggingFace) | liuziwei7, Ziqi, cszy98, weepiess2383, ChenyangSi | RepVideo investigates the impact of cross-layer representations on video generation using diffusion models. The research aims to understand how intermediate layer representations affect spatial appearance and temporal coherence in video generation. The study employs a feature cache module that aggregates features from multiple adjacent transformer layers and integrates these into the model via a gating mechanism. RepVideo improves the total score on the VBench benchmark by 0.4% in motion smoothness and 4.46% in object class compared to the baseline. The findings highlight the importance of optimizing intermediate representations for improved video generation quality, suggesting that this methodology could improve other transformer-based generative models. |
| Towards Best Practices for Open Datasets for LLM Training (Read more on arXiv or HuggingFace) | jending12, ayahbdeir, avi-skowron, stellaathena, stefan-baack | Summary of the AI research paper “Towards Best Practices for Open Datasets for LLM Training”: i) The paper outlines best practices for creating openly licensed datasets for large language model (LLM) training, based on a convening of scholars and practitioners. ii) The main objective is to define normative principles and technical guidelines for developing open access and openly licensed datasets that foster a competitive and transparent LLM ecosystem. iii) The methodology involved analyzing case studies of leading open datasets (Common Pile, Common Corpus, and YouTube-Commons) and convening experts to discuss challenges and opportunities in creating open LLM training datasets. iv) The paper highlights that approximately 480,000 books published between 1929 and 1989 in the U.S. are estimated to be in the public domain but lack specific title identification. v) For AI practitioners, the principal implication is the need to adopt the outlined best practices for data sourcing, processing, governance, and release to ensure the creation of high-quality, transparent, and ethically sound open datasets for LLM training. The paper emphasizes the importance of openly licensed datasets for promoting transparency and accountability in AI, particularly concerning training data. The document lacks specific examples of quantitative findings beyond the stated estimation of public domain books, focusing more on qualitative principles and practices. |
| XMusic: Towards a Generalized and Controllable Symbolic Music Generation Framework (Read more on arXiv or HuggingFace) | Wenjie Zhu, Wei Tan, Wei Yuan, Can Zhang, Sida Tian | XMusic is a framework for generating symbolic music using multi-modal prompts. The main research question is how to build a generalized, controllable, and high-quality framework for symbolic music generation that can handle diverse input prompts. The key methodology involves a multi-modal prompt parsing method (XProjector) that translates various prompts into symbolic music elements, and a music composer (XComposer) with a Generator and a Selector that creates and filters music based on the parsed elements. The primary results show that XMusic outperforms state-of-the-art methods, achieving an average ranking of 1.3077 in video-conditioned subjective evaluations, compared to 1.6923 for the next best method (CMT). Principal implication for AI practitioners is that XMusic provides a novel framework for multi-modal symbolic music generation, demonstrating superior performance in controllability and quality compared to existing methods, as evidenced by the objective and subjective evaluations. |
| Trusted Machine Learning Models Unlock Private Inference for Problems Currently Infeasible with Cryptography (Read more on arXiv or HuggingFace) | Sarah Meiklejohn, Ilia Shumailov, bballe, fhartmann, danrama | Trusted Capable Model Environments (TCMEs) are proposed as a new paradigm for secure computation, enabling private inference for problems currently infeasible with classical cryptography. The main research question is whether capable machine learning models can act as trusted third parties to facilitate secure computations while preserving privacy. The key methodology involves using a machine learning model within a constrained environment (TCME) that ensures statelessness, explicit information flow control, and model trustworthiness. The primary result is that models struggle with structured tasks like graph coloring, achieving only 35% accuracy in identifying correct coloring, but show higher precision (83%) in identifying correct solutions, indicating potential when combined with classical computing methods. The principal implication for AI practitioners is that TCMEs could enable privacy-preserving solutions for complex, unstructured problems where traditional cryptographic methods are impractical, but current model capabilities suggest a need for hybrid approaches combining TCMEs with classical computing techniques. |
| Parameter-Inverted Image Pyramid Networks for Visual Perception and Multimodal Understanding (Read more on arXiv or HuggingFace) | douwh, Changyao, favor123, Einsiedler, wzk1015 | Parameter-Inverted Image Pyramid Networks (PIIP) improve efficiency in visual perception and multimodal understanding tasks. The main research objective is to reduce the computational cost of processing multi-scale images in image pyramids while maintaining high performance. The key methodology used is a novel network architecture, PIIP, which processes higher-resolution images with smaller network branches and integrates information across scales via a cross-branch feature interaction mechanism. When applied to InternViT-6B, PIIP improves detection and segmentation performance by 1%-2% while using only 40%-60% of the original computation, achieving a 60.0 box AP on MS COCO. For AI practitioners, PIIP offers a more efficient way to build high-performance, multi-scale image processing models, significantly reducing computational overhead without sacrificing accuracy. |
| Multimodal LLMs Can Reason about Aesthetics in Zero-Shot (Read more on arXiv or HuggingFace) | Vincentchang, Ruixiang | Multimodal large language models (MLLMs) can be prompted to reason about the aesthetic quality of artwork in a zero-shot setting. The main research question is whether MLLMs can reason about the aesthetic quality of artistic images in a manner aligned with human preferences. The key methodology involves constructing a dataset called MM-StyleBench for benchmarking artistic stylization, modeling human aesthetic preferences, and performing a correlation analysis between MLLM responses and human preferences using various prompting strategies, including the proposed ArtCoT method. The primary results show that ArtCoT significantly enhances aesthetic alignment, achieving an average improvement of 56% in the per-method alignment compared to the baseline. The principal implication is that AI practitioners should utilize task decomposition and concrete language, as demonstrated by ArtCoT, to reduce hallucinations and improve the aesthetic reasoning capabilities of MLLMs when applying them to art evaluation tasks. |
| Ouroboros-Diffusion: Exploring Consistent Content Generation in Tuning-free Long Video Diffusion (Read more on arXiv or HuggingFace) | Jie An, GiantBision, qiudavy, FireCRT, jchensteve | Ouroboros-Diffusion is a novel framework for generating consistent long videos using a pre-trained diffusion model without additional tuning. The main research objective is to address content inconsistency, specifically structural and subject consistency, in tuning-free long video generation using diffusion models. The key methodology involves coherent tail latent sampling to improve structural consistency, a Subject-Aware Cross-Frame Attention (SACFA) mechanism to enhance subject consistency, and self-recurrent guidance using a subject feature bank for long-range coherence. The primary results show that Ouroboros-Diffusion achieves a Temporal Flickering score of 96.12% in single-scene video generation, outperforming the FIFO-Diffusion baseline by 2.74%. For AI practitioners, particularly those working with generative video models, Ouroboros-Diffusion provides a method to significantly enhance the temporal and subject consistency of generated videos without requiring model re-training or fine-tuning, improving the quality and applicability of long video generation. |
Papers for 2025-01-15
| Title | Authors | Summary |
|---|---|---|
| MiniMax-01: Scaling Foundation Models with Lightning Attention (Read more on arXiv or HuggingFace) | Bangwei Gong, Aonian Li, MiniMax, Hannnnnxd, enochzhang | MiniMax-01 introduces a series of large language models featuring efficient scaling via lightning attention and Mixture of Experts, achieving comparable performance to top-tier models with significantly longer context windows. The main research objective is to develop models that match the performance of leading commercial models while offering context windows longer by an order of magnitude using an optimized architecture and training framework. The key methodology involves a hybrid architecture employing lightning attention, a variant of linear attention, combined with softmax attention and a Mixture of Experts (MoE) model, alongside optimized parallel strategies and computation-communication overlap techniques. Primary results show that MiniMax-Text-01, with 456 billion parameters, achieves an 88.5% accuracy on the MMLU benchmark, comparable to leading models, while supporting context windows up to 4 million tokens during inference. The principal implication for AI practitioners is that the model’s architecture and training framework enable efficient training and inference on models with large context windows, which could facilitate the development of more sophisticated AI agents. |
| Padding Tone: A Mechanistic Analysis of Padding Tokens in T2I Models (Read more on arXiv or HuggingFace) | Yoad Tewel, Rinon Gal, Hadas Orgad, Ido Galil, Michael Toker | This paper investigates the role of padding tokens in text-to-image (T2I) models. The main research question is how padding tokens, typically used to standardize input prompt lengths, affect the image generation process in T2I models. The key methodology involves two causal intervention techniques, ITE and IDP, to analyze the impact of padding tokens on model components by selectively replacing prompt or padding tokens with “clean” pads and observing the changes in generated images. The primary results show that in models like LDM and LLaMA-UNet, padding tokens encode significant semantic information, achieving a CLIP score of 0.30 when only the first 20% of pad tokens are used, and contribute to image generation, whereas, in models with frozen text encoders, they are largely ignored. The principal implication for AI practitioners is that the choice to include or exclude padding tokens during training and inference can significantly impact model behavior, particularly in models with trainable text encoders or those employing multi-modal attention mechanisms. |
| MangaNinja: Line Art Colorization with Precise Reference Following (Read more on arXiv or HuggingFace) | Hao Ouyang, Jie Xiao, Xi Chen, Ka Leong Cheng, Zhiheng Liu | MangaNinja is a reference-based line art colorization method that leverages diffusion models to accurately transfer colors from a reference image to a target line art. The main research question is how to achieve precise and controllable line art colorization that preserves character identity and details from a reference image, even with significant variations between the reference and line art. The key methodology involves a dual-branch architecture with a patch shuffling module for correspondence learning between the reference image and line art, and a point-driven control scheme using PointNet for fine-grained color matching. The primary results show that MangaNinja achieves a DINO score of 69.91 and a CLIP score of 90.02, outperforming existing methods on a newly collected benchmark. For AI practitioners, MangaNinja offers a robust method for automating line art colorization, potentially accelerating the animation and comics production workflow. |
| A Multi-Modal AI Copilot for Single-Cell Analysis with Instruction Following (Read more on arXiv or HuggingFace) | Jingyang Qian, Kangwei Liu, Xinle Deng, Ningyu, Fangyinfff | A multi-modal AI copilot, INSTRUCTCELL, is introduced for single-cell analysis using natural language instructions. Main research question or objective: How can a multi-modal AI copilot be developed to effectively integrate natural language instructions with single-cell RNA sequencing (scRNA-seq) data to perform various analytical tasks? Key methodology used: A multi-modal instruction dataset was constructed, pairing text-based instructions with scRNA-seq profiles, and a multi-modal cell language model was developed, featuring a Q-Former module, a pre-trained language model (LM), and a cell reconstruction block, tuned via instruction tuning. Primary results: INSTRUCTCELL achieved an accuracy exceeding 99.97% in answer extraction using the xFinder tool and demonstrated robust performance in cell type annotation, conditional pseudo-cell generation, and drug sensitivity prediction, outperforming existing single-cell foundation models in several benchmarks. Principal implication for AI practitioners: AI practitioners can leverage INSTRUCTCELL’s architecture and training methodology to develop multi-modal AI tools that integrate diverse data types and natural language processing, enhancing the interpretability and accessibility of complex biological data analysis. |
| Diffusion Adversarial Post-Training for One-Step Video Generation (Read more on arXiv or HuggingFace) | Xuefeng Xiao, Ceyuan Yang, Yuxi Ren, Xin Xia, PeterL1n | Diffusion Adversarial Post-Training (APT) accelerates one-step video generation using diffusion models. The research objective was to develop a method for high-quality, real-time one-step video generation, overcoming limitations of existing diffusion distillation techniques. The methodology employed adversarial post-training against real data, following diffusion pre-training, incorporating several architectural and training improvements, and an approximated R1 regularization objective. The model, Seaweed-APT, generated 2-second, 1280x720, 24fps videos in real time using a single forward pass; it achieved image generation quality comparable to state-of-the-art methods. This research directly impacts AI practitioners by providing a method for generating high-resolution videos in real-time with a single forward pass, potentially improving efficiency and application across various domains; however, text alignment quality was lower than the original 25-step diffusion model. |
| PokerBench: Training Large Language Models to become Professional Poker Players (Read more on arXiv or HuggingFace) | Zhengyu Li, Aniket Rahane, Richard Yang, Richard Zhuang, akshat57 | POKERBENCH is a new benchmark for evaluating large language models’ (LLMs) ability to play poker. The main research objective is to assess how well LLMs can learn and apply game theory optimal poker strategies. The key methodology involves creating a dataset (POKERBENCH) of 11,000 poker scenarios, evaluating various LLMs on this dataset, and fine-tuning them using a subset of this data. The primary results show that GPT-4 achieved the highest accuracy of 53.55% among pre-trained models, but fine-tuned models like Llama-3-8B surpassed it, reaching 80.64% accuracy. For AI practitioners, POKERBENCH provides a valuable benchmark for training and evaluating LLMs on complex decision-making tasks, with the most impactful finding being that supervised fine-tuning can significantly improve LLM performance in strategic game environments like poker, but may have limitations. |
| Democratizing Text-to-Image Masked Generative Models with Compact Text-Aware One-Dimensional Tokens (Read more on arXiv or HuggingFace) | Xiaohui Shen, Chenglin Yang, Qihang Yu, Dongwon Kim, turkeyju | This paper introduces TA-TiTok, a text-aware one-dimensional image tokenizer, and MaskGen, a text-to-image masked generative model, designed for efficient and accessible text-to-image generation. The main research question is: Can an efficient and effective text-to-image generative model be developed using only open data, enabling reproducibility? The key methodology involves a novel text-aware 1D tokenizer (TA-TiTok) that integrates textual information during de-tokenization and a simplified one-stage training process for masked generative models. Primary results show that MaskGen-XL achieves a generation FID of 7.51 on the MJHQ-30K benchmark using discrete tokens, surpassing several recent models while using only open-source datasets. The principal implication for AI practitioners is that high-quality text-to-image generation can be achieved with reduced computational resources and publicly available data, facilitating broader access and research in this area. |
| Omni-RGPT: Unifying Image and Video Region-level Understanding via Token Marks (Read more on arXiv or HuggingFace) | Subhashree Radhakrishnan, Sifei Liu, De-An Huang, Min-Hung Chen, Miran Heo | Omni-RGPT unifies image and video region-level understanding using token marks for consistent spatio-temporal comprehension. The main research question is how to achieve consistent region representation across spatio-temporal dimensions in images and videos for multimodal large language models (MLLMs). The key methodology involves introducing Token Mark, a set of tokens highlighting target regions within the visual feature space, and an auxiliary task that guides Token Mark by leveraging the consistency of the tokens for stable region interpretation across video frames. Primary results show that Omni-RGPT achieves 88.5% accuracy on the Visual Commonsense Reasoning (VCR) validation set, demonstrating state-of-the-art performance in image-based commonsense reasoning. The principal implication for AI practitioners is that using Token Mark for region-level understanding enhances the performance of MLLMs on tasks requiring detailed visual comprehension, offering a more robust method for integrating region-specific information in both image and video domains. |
| OpenCSG Chinese Corpus: A Series of High-quality Chinese Datasets for LLM Training (Read more on arXiv or HuggingFace) | Ran Chen, Wei Wang, Zekun Wang, Ziyun Dai, yuyijiong | OpenCSG Chinese Corpus introduces four high-quality Chinese datasets for LLM training. The research objective was to address the scarcity of high-quality Chinese datasets for LLM training by creating a series of datasets with diverse characteristics. The methodology involved combining automated filtering techniques with synthetic data generation and domain-focused curation. Results demonstrated significant performance improvements using a 2B parameter model trained on Fineweb-Edu-Chinese (achieving an accuracy increase of approximately 0.08 over the baseline on the CMMLU benchmark). This work provides publicly available high-quality datasets that are directly applicable to improving the performance of Chinese LLMs, particularly in educational contexts. |
| Tarsier2: Advancing Large Vision-Language Models from Detailed Video Description to Comprehensive Video Understanding (Read more on arXiv or HuggingFace) | Yuan Lin, Yuchen Zhang, Haomiao Sun, Jiawei Wang, Liping Yuan | Tarsier2 is a state-of-the-art large vision-language model for video understanding, especially detailed video description. The main research objective is to develop a model that can generate detailed and accurate video descriptions and exhibit superior general video understanding capabilities. The key methodology involves scaling pre-training data to 40 million video-text pairs, performing fine-grained temporal alignment during supervised fine-tuning, and using model-based sampling with Direct Preference Optimization (DPO). The primary results show that Tarsier2-7B outperforms GPT-40 by 2.8% in F1 score on the DREAM-1K benchmark for detailed video description. The principal implication for AI practitioners is that scaling training data and incorporating fine-grained temporal alignment, along with DPO, significantly enhances the performance of vision-language models on video understanding tasks, particularly in generating detailed and accurate video descriptions. |
| Enhancing Automated Interpretability with Output-Centric Feature Descriptions (Read more on arXiv or HuggingFace) | Mor Geva, Chen Agassy, Roy Mayan, Yoav Gur-Arieh, atticusg | This paper introduces output-centric methods for automatically generating feature descriptions in large language models (LLMs). The research objective was to improve automated interpretability pipelines by addressing the limitations of input-centric approaches. Two output-centric methods, VocabProj and TokenChange, were developed and compared to the existing input-centric MaxAct method using input- and output-based evaluations. Results showed that ensemble methods combining input and output-centric approaches consistently outperformed MaxAct on both evaluations, with a significant improvement of 6-10% observed in Gemma-2. This work provides AI practitioners with improved methods for generating feature descriptions, leading to more effective model interpretability and steering capabilities, particularly by enabling efficient discovery of previously “dead” features. |
| Potential and Perils of Large Language Models as Judges of Unstructured Textual Data (Read more on arXiv or HuggingFace) | Satya Kapoor, Sreyoshi Bhaduri, Natalie Perez, Rewina Bedemariam, amanchadha | This research investigates the effectiveness of LLMs as judge models for evaluating thematic alignment in summaries generated by other LLMs using open-ended survey data. The main objective was to determine if LLMs could replicate human judgment in thematic alignment evaluations and the implications of higher inter-model agreement compared to human-model agreement. A three-stage methodology was used, employing human evaluation as a baseline, followed by LLM evaluation using several models (Claude, Titan Express, Nova Pro, and Llama) and statistical analysis (Cohen’s kappa, Spearman’s rho, Krippendorff’s alpha). Results showed that while LLMs offered a scalable alternative to human raters, achieving moderate agreement (Cohen’s kappa = 0.44) with human ratings, humans demonstrated superior ability in detecting subtle nuances. This highlights the need for cautious consideration when generalizing LLM judge models across various contexts and reinforces the importance of human oversight in ensuring fair and accurate AI-assisted text analysis. |
| HALoGEN: Fantastic LLM Hallucinations and Where to Find Them (Read more on arXiv or HuggingFace) | Yejin Choi, David Wadden, Shrusti Ghela, Abhilasha Ravichander | HALOGEN is a benchmark for evaluating hallucinations in long-form text generated by large language models (LLMs). Main research question or objective: To construct a comprehensive benchmark for measuring and analyzing hallucination behavior in long-form generations of LLMs across diverse domains. Key methodology used: Development of the HALOGEN benchmark, comprising 10,923 prompts across nine domains and automatic high-precision verifiers that decompose LLM generations into atomic units and verify them against external knowledge sources. Primary results: Evaluation of 14 LLMs revealed that even the best-performing models produce hallucinations in 4% to 86% of generated atomic facts, depending on the task, with GPT-4 demonstrating better refusal behavior than other models. Principal implication for AI practitioners: AI practitioners should leverage diverse, multi-domain benchmarks like HALOGEN to evaluate and mitigate LLM hallucinations, as no single domain is highly predictive of hallucination behavior in others, highlighting the complexity of this issue. |
| AfriHate: A Multilingual Collection of Hate Speech and Abusive Language Datasets for African Languages (Read more on arXiv or HuggingFace) | Ibrahim Said Ahmad, David Ifeoluwa Adelani, Abinew Ali Ayele, Idris Abdulmumin, Shamsuddeen Hassan Muhammad | AfriHate is a new dataset for hate speech and abusive language detection in 15 African languages. The main research objective is to address the lack of high-quality data for hate speech and abusive language in African languages and evaluate the effectiveness of current models. The key methodology involves collecting tweets, crowdsourcing keywords, manually annotating data for hate speech, abusive language, or neutral content, and conducting experiments with various pre-trained language models (PLMs), few-shot learning, and prompting large language models (LLMs). The primary results show that fine-tuning multilingual models yields the best performance, with AfroXLMR-76L achieving an average macro F1-score of 78.16 across all languages. The principal implication for AI practitioners is that multilingual fine-tuning on AfriHate is currently the most effective approach for hate speech detection in the studied African languages, emphasizing the importance of multilingual and context-specific models for low-resource settings. |
Papers for 2025-01-14
| Title | Authors | Summary |
|---|---|---|
| The Lessons of Developing Process Reward Models in Mathematical Reasoning (Read more on arXiv or HuggingFace) | RunjiLin, BeichenZhang, wuyangzhen, chujiezheng, Zhenru | This paper investigates the development of Process Reward Models (PRMs) for mathematical reasoning in large language models (LLMs). The main research question is how to effectively construct and evaluate PRMs to improve the process supervision in mathematical reasoning. The key methodology involves a consensus filtering mechanism that integrates Monte Carlo (MC) estimation with LLM-as-a-judge for data annotation and a combination of response-level and step-level metrics for evaluation. The primary results show that the consensus filtering mechanism improves PRM performance, with Qwen2.5-Math-PRM-7B achieving a 67.6% average accuracy on the Best-of-8 evaluation, outperforming other 7B PRMs. The principal implication for AI practitioners is that combining MC estimation with LLM-as-a-judge and using comprehensive evaluation strategies can lead to more robust and reliable PRMs for enhancing mathematical reasoning in LLMs. |
| Tensor Product Attention Is All You Need (Read more on arXiv or HuggingFace) | Huizhuo Yuan, Yifeng Liu, thughost, zhenqincn, yifAI | Tensor Product Attention (TPA) is a novel attention mechanism that improves memory efficiency during inference in language models. The main research question is how to reduce the memory overhead of key-value (KV) caches in language models while maintaining or improving performance. The key methodology is using tensor decompositions to represent queries, keys, and values compactly, integrating with Rotary Positional Embedding (RoPE). Primary results show that TPA reduces KV cache size by up to 10x or more during inference and achieves lower validation perplexity than baselines like Multi-Head Attention (MHA), as evidenced by TPA achieving an average of 51.41% in zero-shot mode versus MHA’s 50.11% on medium-size models. The principal implication for AI practitioners is that TPA offers a more memory-efficient way to deploy large language models, enabling the processing of significantly longer sequences under fixed resource constraints. |
| $\text{Transformer}^2$: Self-adaptive LLMs (Read more on arXiv or HuggingFace) | tyj2022, edoarc, lfsm | Transformer², a self-adaptation framework for large language models (LLMs), enhances LLMs’ performance on unseen tasks in real-time. The main research objective is to develop a framework that enables LLMs to adapt to diverse tasks dynamically without extensive fine-tuning. The key methodology involves a two-pass mechanism during inference, employing task-specific “expert” vectors trained using reinforcement learning, and a novel parameter-efficient fine-tuning method called Singular Value Fine-tuning (SVF). A primary result is that SVF fine-tuning of LLAMA3-8B-INSTRUCT boosted performance on the GSM8K task from a baseline score of 75.89 to 79.15. The principal implication for AI practitioners is that Transformer² provides a scalable and efficient solution for enhancing LLM adaptability and task-specific performance, particularly valuable for dynamic, self-organizing AI systems. |
| VideoAuteur: Towards Long Narrative Video Generation (Read more on arXiv or HuggingFace) | Jiepeng Cen, Liangke Gui, Lu Qi, Feng Cheng, lambertxiao | VideoAuteur introduces a new method for long-form narrative video generation in the cooking domain. The main research objective is to generate coherent and informative long-form videos that convey clear narratives. The key methodology involves curating a large-scale cooking video dataset (CookGen) and developing an interleaved auto-regressive model, “VideoAuteur,” which sequentially generates actions, captions, and keyframes, conditioning a video generation model. The primary result is that the proposed method achieves substantial improvements in generating visually detailed and semantically aligned keyframes, with human evaluations showing an 82.0 rating for their caption quality compared to 79.3 for Qwen2-VL-72B. The principal implication for AI practitioners is that the VideoAuteur model and CookGen dataset can be used to enhance long-form narrative video generation, offering a framework for creating more coherent and contextually rich videos. |
| WebWalker: Benchmarking LLMs in Web Traversal (Read more on arXiv or HuggingFace) | zhoudeyu, Runnaning, ZekunXi, wzl0228, callanwu | WebWalkerQA is a new benchmark for evaluating large language models (LLMs) on web traversal tasks. The main research question is how well LLMs can navigate and extract information from websites to answer complex, multi-step queries. The key methodology is a multi-agent framework called WebWalker, which uses explorer and critic agents to simulate human-like web navigation, combined with a dataset of 680 queries across 1373 webpages. A primary result is that the best-performing model achieved only 37.50% accuracy on the WebWalkerQA benchmark. The principal implication for AI practitioners is that current LLMs struggle with deep web traversal tasks, and WebWalker can be integrated with retrieval-augmented generation (RAG) systems to enhance their ability to navigate and utilize information from websites. |
| O1 Replication Journey – Part 3: Inference-time Scaling for Medical Reasoning (Read more on arXiv or HuggingFace) | Gui Geng, Pengfei, alanyoung058, ZhenHuang, zongzi | The paper explores inference-time scaling in large language models (LLMs) for medical reasoning tasks, demonstrating improved performance through extended reasoning processes. The main research question is whether increasing inference time can enhance the performance of LLMs on medical reasoning benchmarks of varying complexity. The key methodology involves fine-tuning LLMs on synthesized datasets that demonstrate extended reasoning (LongStep and LongMonolog) and evaluating their performance on MedQA, Medbullets, and JAMA Clinical Challenges using metrics like accuracy and average output token length. The primary results show that increasing inference time leads to improved performance, with models trained on extended reasoning data achieving accuracy improvements of 6-11% using a training set of only 500 samples. For AI practitioners, the principal implication is that scaling inference time by incorporating structured thought processes can significantly enhance LLMs’ ability to address complex medical reasoning tasks, even with limited training data. |
| MinMo: A Multimodal Large Language Model for Seamless Voice Interaction (Read more on arXiv or HuggingFace) | langgz, gaoruize, zhihaodu, Yingda, chenmengzhe | MinMo is an 8-billion-parameter multimodal large language model designed for seamless voice interactions. The main research objective is to develop a model that addresses limitations of prior aligned multimodal models, specifically in maintaining text-LLM capabilities while achieving state-of-the-art voice comprehension and generation. The key methodology involves multi-stage training on 1.4 million hours of diverse speech data, aligning speech-to-text, text-to-speech, speech-to-speech, and duplex interactions. The primary result is that MinMo achieves state-of-the-art performance across various benchmarks, including spoken dialogue and multilingual speech recognition, with a speech-to-text latency of approximately 100ms. The principal implication for AI practitioners is that MinMo provides a robust framework for developing voice interaction systems, demonstrating strong performance in full-duplex conversations and nuanced speech generation. |
| SPAM: Spike-Aware Adam with Momentum Reset for Stable LLM Training (Read more on arXiv or HuggingFace) | Zhangyang Wang, Lu Liu, Gaojie Jin, Ziquan Zhu, Tianjin Huang | This paper introduces Spike-Aware Adam with Momentum Reset (SPAM), a novel optimizer to address gradient and loss spikes in large language model (LLM) training. The main research question is how to mitigate the negative impact of gradient spikes on LLM training stability and performance. The key methodology involves integrating momentum reset and spike-aware gradient clipping into the Adam optimizer, along with a sparse momentum technique for memory efficiency. Primary results show that SPAM outperforms Adam and its variants across various tasks; for example, SPAM achieved a perplexity of 30.46 on the C4 dataset with the LLaMA-60M model, compared to 34.09 for Adam. The principal implication for AI practitioners is that SPAM provides a more stable and resource-efficient optimizer for training LLMs, directly addressing a known issue that affects model performance and training cost. |
| BIOMEDICA: An Open Biomedical Image-Caption Archive, Dataset, and Vision-Language Models Derived from Scientific Literature (Read more on arXiv or HuggingFace) | yeunglevy, yuhuizhang, jnirschl, minwoosun, lozanoe | Here is a concise summary of the research paper “BIOMEDICA: An Open Biomedical Image-Caption Archive, Dataset, and Vision-Language Models Derived from Scientific Literature”: The paper introduces BIOMEDICA, a framework for curating a large-scale biomedical image-caption dataset from open-access scientific literature and using it to train vision-language models. The main research objective is to address the scarcity of publicly available, diverse biomedical image-caption datasets for training generalist biomedical vision-language models. The key methodology involves an ETL pipeline to extract and serialize image-caption pairs from PubMed Central Open Access articles, followed by expert-guided annotation of image clusters and continual pre-training of CLIP-style models on the resulting dataset. The primary result is that the best model (BMCA-CLIP) achieved a 6.56% average improvement in zero-shot classification across 40 biomedical tasks compared to prior state-of-the-art models. The principal implication for AI practitioners is that BIOMEDICA provides a valuable resource for training and evaluating vision-language models for diverse biomedical applications, demonstrated by the strong zero-shot performance of BMCA-CLIP, even with 10x less compute. |
| ChemAgent: Self-updating Library in Large Language Models Improves Chemical Reasoning (Read more on arXiv or HuggingFace) | Wangchunshu, siruo2, super-dainiu, CamelH, RTT1 | ChemAgent: A novel framework improving chemical reasoning in large language models through a dynamic, self-updating library. Main research question or objective: To address the challenges of large language models (LLMs) in handling domain-specific formulas, executing accurate reasoning, and integrating code effectively in chemical reasoning tasks. Key methodology used: Development of a dynamic, self-updating library that decomposes chemical tasks into sub-tasks, compiles them into a structured collection, and retrieves/refines pertinent information for future queries, alongside three types of memory (planning, execution, knowledge) and a library-enhanced reasoning component. Primary results: ChemAgent achieved performance gains of up to 46% (using GPT-4) on four chemical reasoning datasets from SciBench, significantly outperforming existing methods. Principal implication for AI practitioners: AI practitioners can leverage ChemAgent’s self-updating library and memory components to enhance LLMs’ performance on complex, multi-step reasoning tasks, particularly in specialized domains like chemistry. |
| UnCommon Objects in 3D (Read more on arXiv or HuggingFace) | EarlGr, Jiali, zarzarj, JianyuanWang, wenchang05 | This paper introduces UnCommon Objects in 3D (uCO3D), a new object-centric 3D dataset for deep learning and generative AI. The main research objective is to address the scarcity of high-quality, diverse real-world 3D object datasets for training AI models. The key methodology involves collecting 360° videos of over 1,000 object categories, annotated with 3D camera poses, point clouds, captions, and 3D Gaussian Splat reconstructions, validated through extensive quality checks. The primary result is that uCO3D contains 170,000 scenes, and models trained on uCO3D outperform those trained on MVImgNet and CO3Dv2 in few-view 3D reconstruction and novel-view synthesis tasks. For AI practitioners, uCO3D provides a higher-quality dataset for training 3D deep learning models, directly improving the performance of models in tasks such as 3D object reconstruction and generation. |
Papers for 2025-01-13
| Title | Authors | Summary |
|---|---|---|
| OmniManip: Towards General Robotic Manipulation via Object-Centric Interaction Primitives as Spatial Constraints (Read more on arXiv or HuggingFace) | Wenlong Gao, Tianshu Wu, Ergogogogo, JiyaoZhang, pmj110119 | Here is a concise summary of the paper “OmniManip: Towards General Robotic Manipulation via Object-Centric Interaction Primitives as Spatial Constraints”: i) The paper introduces OmniManip, a novel system for open-vocabulary robotic manipulation that uses object-centric interaction primitives as spatial constraints to bridge the gap between vision-language models (VLMs) and low-level precision. ii) The main research objective is to develop a more efficient and generalizable representation that bridges VLM high-level reasoning with precise, low-level robotic manipulation. iii) The key methodology involves using a dual closed-loop system: one loop for high-level planning through primitive resampling, interaction rendering, and VLM checking, and another for low-level execution via 6D pose tracking, along with representing object interactions within a canonical space to define actionable 3D spatial constraints. iv) Primary results show that OmniManip achieved a 68.3% success rate in closed-loop, zero-shot generalization across diverse robotic manipulation tasks, outperforming the best baseline (ReKep) which achieved 45.0%. v) The principal implication for AI practitioners is that OmniManip provides a framework for automating large-scale simulation data generation and developing robotic systems capable of robust, real-time control without requiring VLM fine-tuning. |
| VideoRAG: Retrieval-Augmented Generation over Video Corpus (Read more on arXiv or HuggingFace) | Sung Ju Hwang, jinheon, KangsanKim71, starsuzi | VideoRAG introduces a novel framework for retrieval-augmented generation using video corpora. The research objective was to improve factual accuracy in large language models by dynamically retrieving and incorporating relevant video content into the generation process. The methodology involved leveraging large video language models (LVLMs) to process both visual and textual information from videos for retrieval and generation. Results showed VideoRAG-VT (using both visual and textual video features) achieved a ROUGE-L score of 0.252, significantly outperforming text-only baselines. This demonstrates the efficacy of incorporating video data into RAG, suggesting that incorporating multimodal data, particularly video, enhances the accuracy and quality of generated responses. |
| OVO-Bench: How Far is Your Video-LLMs from Real-World Online Video Understanding? (Read more on arXiv or HuggingFace) | qiaozc, zyh, HelloJiang, Niujunbo2002, JoeLeelyf | OVO-Bench is a new benchmark for evaluating online video understanding capabilities of Video Large Language Models (Video-LLMs). The main research question is: How effective are current Video-LLMs at understanding video content in an online, real-world setting where questions are posed at specific timestamps? The key methodology involves creating a dataset (OVO-Bench) of 644 videos with 2,814 human-curated meta-annotations, and evaluating nine Video-LLMs using a pipeline that queries models along the video timeline under three scenarios (Backward Tracing, Real-Time Understanding, Forward Active Responding). The primary results show that even the best-performing model, Gemini 1.5 Pro, achieved only 65.25% overall accuracy, significantly lower than human performance, and forward active responding accuracy was 57.15%. The principal implication for AI practitioners is that current Video-LLMs still struggle with online video understanding tasks that require temporal awareness, highlighting a need for model development focusing on real-time processing and continuous adaptation to incoming video streams. |
| LlamaV-o1: Rethinking Step-by-step Visual Reasoning in LLMs (Read more on arXiv or HuggingFace) | Dinura Dissanayake, hishamcholakkal, ahmedheakl, Ritesh-hf, omkarthawakar | LlamaV-01 introduces a framework for advancing step-by-step visual reasoning in large language models (LLMs). The main research objective is to develop a comprehensive framework for evaluating and enhancing step-by-step visual reasoning in LLMs, addressing the limitations of current models that primarily focus on end-task accuracy. The key methodology includes the introduction of a new benchmark (VRC-Bench) for multi-step reasoning, a novel metric evaluating reasoning quality at the step level, and a new multimodal visual reasoning model (LlamaV-01) trained using a multi-step curriculum learning approach. The primary results show that LlamaV-01 achieves an average score of 67.3 across six benchmarks, with an absolute gain of 3.8% over the Llava-CoT model while being 5x faster during inference. The principal implication for AI practitioners is that using this framework, including the VRC-Bench and the LlamaV-01 model, can lead to more accurate, interpretable, and efficient visual reasoning systems. |
| Enabling Scalable Oversight via Self-Evolving Critic (Read more on arXiv or HuggingFace) | Losin94, Benyou, yeshoubaizi, ziniuli, tangzhy | This paper introduces SCRIT, a framework that enables the self-evolution of critique abilities in large language models (LLMs) for scalable oversight. The main research question is how to enhance the critique capabilities of LLMs without relying on external supervision from humans or stronger models. The key methodology used is a two-step process involving contrastive-based self-critic generation using reference solutions and a self-validation mechanism that ensures critique quality through correction outcomes, followed by self-training on the validated data. The primary results show that SCRIT, implemented with Qwen2.5-72B-Instruct, achieves up to a 10.3% improvement on critique-correction and error identification benchmarks, with the average F1 score on error identification tasks rising from 37.8% to 45.0%. The principal implication for AI practitioners is that SCRIT offers a method for improving LLMs’ abilities to critique and correct mathematical reasoning problems without the need for costly human annotations or access to more powerful models, demonstrating a path towards more autonomous model refinement. |
| ConceptMaster: Multi-Concept Video Customization on Diffusion Transformer Models Without Test-Time Tuning (Read more on arXiv or HuggingFace) | Ruimao, Xintao, Qiulin, ziyangy, Yuzhou914 | ConceptMaster is introduced as a novel framework for multi-concept video customization using diffusion transformer models without requiring test-time tuning. The main research question is how to achieve high-fidelity multi-concept video customization while effectively decoupling identities and maintaining concept fidelity. The key methodology involves learning decoupled multi-concept embeddings via a Decouple Attention Module (DAM) and injecting them into diffusion models using a standalone Multi-Concept Injector (MC-Injector), alongside a data construction pipeline for creating high-quality multi-concept video-entity pairs. The primary result is that ConceptMaster achieved a score of 22.378 on identity decoupling, outperforming other compared methods on the MC-Bench benchmark. The principal implication for AI practitioners is that ConceptMaster provides an effective method for generating personalized and semantically accurate videos across multiple concepts without the need for additional test-time tuning, enhancing the practicality of video customization in real-world applications. |
| Multi-subject Open-set Personalization in Video Generation (Read more on arXiv or HuggingFace) | universome, studyfang, willi-menapace, aliaksandr-siarohin, tschen | Video Alchemist is introduced, a video generation model capable of multi-subject, open-set personalization for foreground objects and backgrounds without test-time optimization. The main research objective is to develop a video personalization model that can incorporate multiple subjects and open-set entities into generated videos without requiring fine-tuning for new concepts. The key methodology involves a new Diffusion Transformer module that fuses conditional reference images and corresponding subject-level text prompts with cross-attention layers, along with a data construction pipeline featuring extensive image augmentations. The primary result is that Video Alchemist outperforms existing personalization methods, achieving a 23.2% higher subject similarity than VideoBooth in quantitative evaluations. For AI practitioners, Video Alchemist offers a new approach to video generation with enhanced personalization capabilities, directly applicable to creating customized videos with specific subjects and contexts. |
| ReFocus: Visual Editing as a Chain of Thought for Structured Image Understanding (Read more on arXiv or HuggingFace) | danielpaulroth, jw2yang, zyang39, mqliu, Fiaa | ReFocus is a framework that equips multimodal Large Language Models (LLMs) with the ability to generate “visual thoughts” by performing visual editing on structured images such as tables and charts. The main research question is how to improve multimodal LLMs’ selective attention and multi-hop visual reasoning capability on structured images. The key methodology involves prompting LLMs to generate Python code to call visual editing tools that modify the input image, sequentially drawing boxes, highlighting sections, and masking out areas to enhance visual reasoning. The primary results show that ReFocus improves performance on table and chart understanding tasks, yielding an average gain of 11.0% on table tasks and 6.8% on chart tasks over GPT-4o without visual editing. For AI practitioners, ReFocus offers a simple yet effective framework to enhance multimodal LLMs’ performance on structured image understanding by integrating visual reasoning as an intermediate step. |
| Multiagent Finetuning: Self Improvement with Diverse Reasoning Chains (Read more on arXiv or HuggingFace) | Shuang Li, Joshua B. Tenenbaum, Antoniotorralbaborruel, yilundu, vsub851 | This paper introduces a multiagent finetuning approach for improving large language models (LLMs) through self-generated synthetic data. The main research question is whether finetuning a multiagent society of LLMs, rather than a single model, can enhance reasoning performance and preserve diversity over multiple rounds of self-improvement. The key methodology involves specializing independent LLMs as generation or critic agents via finetuning on data generated through multiagent debate, followed by iterative finetuning of these agents on their own generated data. The primary result is that across five rounds of finetuning using the Phi-3 model, the accuracy of multiagent finetuning improved from 58.8% to 66.0% on the MATH dataset. The principal implication is that AI practitioners can leverage multiagent finetuning to enhance LLM performance beyond the limitations of single-agent self-improvement, particularly on complex reasoning tasks. |
| Infecting Generative AI With Viruses (Read more on arXiv or HuggingFace) | fgmckee, dnoever | Here is a concise summary of the research paper: i) This study examines the security of Vision-Language Models (VLMs) by embedding the EICAR test file in JPEG images and assessing the models’ ability to handle and potentially execute it. ii) The main research objective is to evaluate whether VLMs can be used as a vector to transport, manipulate, and potentially execute a surrogate malware (EICAR) embedded within image files. iii) The key methodology involved appending the EICAR string to JPEG images, uploading them to various LLMs, and using Python scripts within the LLMs’ environments to extract and manipulate the embedded string. iv) The primary results showed that the EICAR string could be consistently masked in image metadata, and successfully extracted using Python within the LLM environments; for example, 1 out of 55 virus detectors flagged the initial pixel file with the appended EICAR string. v) The principal implication for AI practitioners is the need to develop robust file inspection methods for VLMs to detect and prevent the manipulation of potentially malicious code embedded in image files. |
Papers for 2025-01-10
| Title | Authors | Summary |
|---|---|---|
| The GAN is dead; long live the GAN! A Modern GAN Baseline (Read more on arXiv or HuggingFace) | jamestompkin, kuleshov, Skylion007, Eva1209 | Here is a concise summary of the paper: i) The paper introduces R3GAN, a new baseline for Generative Adversarial Networks (GANs) that achieves state-of-the-art results without relying on ad-hoc tricks common in previous GAN architectures. ii) The main research objective is to develop a more principled and stable GAN baseline by addressing mode dropping and non-convergence issues in existing GAN training. iii) The key methodology involves proposing a novel regularized relativistic GAN loss (RpGAN + R1 + R2) and modernizing the network backbone using ResNet design principles and grouped convolutions. iv) The primary results show that R3GAN surpasses StyleGAN2 on FFHQ-256, achieving an FID score of 7.05 compared to StyleGAN2’s 7.52, and matches or exceeds state-of-the-art GANs and diffusion models on various datasets. v) The principal implication for AI practitioners is that R3GAN provides a robust and efficient baseline for image generation tasks, demonstrating that GANs remain competitive with modern architectures and can be trained reliably without complex, ad-hoc techniques. |
| An Empirical Study of Autoregressive Pre-training from Videos (Read more on arXiv or HuggingFace) | Ilija Radosavovic, jitendra1995, yossig, rravishankar, brjathu | This paper empirically studies autoregressive pre-training of transformer models on videos for visual representation learning. The main research question is how effective is autoregressive pre-training on videos for learning visual representations across various downstream tasks. The key methodology involves training a series of autoregressive video models, called Toto, to predict future tokens in videos and images, using a diverse dataset of over 1 trillion visual tokens and evaluating these models on downstream tasks. The primary result is that autoregressive pre-training leads to competitive performance across all benchmarks, with the Toto-1b model achieving 75.3% top-1 accuracy on ImageNet classification. The principal implication for AI practitioners is that autoregressive pre-training on videos is a viable method for learning visual representations, achieving strong performance on various tasks despite minimal inductive biases. |
| Are VLMs Ready for Autonomous Driving? An Empirical Study from the Reliability, Data, and Metric Perspectives (Read more on arXiv or HuggingFace) | ZwwWayne, Chonghao, THUdyh, ldkong, shaoyuanxie | DriveBench, a benchmark dataset, evaluates the reliability of Vision-Language Models (VLMs) in autonomous driving across various tasks and conditions. The main research question is: Are existing VLMs capable of providing reliable explanations grounded on visual cues for driving? The methodology involves evaluating 12 VLMs on a dataset with 19,200 frames and 20,498 QA pairs across 17 settings (clean, corrupted, and text-only inputs), using metrics like accuracy, traditional language metrics, and GPT scores. Primary results indicate that under clean image inputs, the GPT-4 model achieved a GPT score of 75.75 in the planning task, but VLMs often generated plausible yet fabricated responses under degraded or missing visual inputs. The principal implication for AI practitioners is that current VLMs are not yet reliable for autonomous driving applications due to their tendency to provide fabricated responses under degraded visual conditions, emphasizing the need for improved datasets and evaluation protocols. |
| On Computational Limits and Provably Efficient Criteria of Visual Autoregressive Models: A Fine-Grained Complexity Analysis (Read more on arXiv or HuggingFace) | Yingyu Liang, Xiaoyu Li, Zhenmei, JamesSand, keyekun | Visual Autoregressive (VAR) models’ computational complexity and efficiency for image generation are analyzed in this paper. The main research question is whether the computations of VAR models can be performed faster than O(n⁴) time. The key methodology involves analyzing the computation of VAR models under the Strong Exponential Time Hypothesis (SETH) and using low-rank approximations to develop efficient algorithms. A primary result is that when the hidden dimension d = O(log n) and the bound of the entries of the input matrices R = o(√log n), there is an algorithm that approximates the VAR model up to 1/poly(n) additive error in O(n²⁺⁰⁽¹⁾) time. The principal implication for AI practitioners is that VAR models can be computed in almost quadratic time under specific conditions, offering a more efficient approach to image generation than previous O(n⁴) methods. |
| Centurio: On Drivers of Multilingual Ability of Large Vision-Language Model (Read more on arXiv or HuggingFace) | Radu Timofte, Chris Biemann, Carolin Holtermann, Florian Schneider, Gregor Geigle | Centurio is a 100-language large vision-language model (LVLM) that offers state-of-the-art performance across 14 tasks and 56 languages. The main research question is what are the optimal training strategies for developing massively multilingual LVLMs, focusing on the number of training languages, data distribution across languages, and techniques for improving multilingual text-in-image understanding. The key methodology involves a series of multi-stage experiments spanning 13 downstream vision-language tasks and 43 languages, systematically varying the training data composition and evaluating performance. A primary result is that including up to 100 training languages simultaneously with as little as 25-50% of non-English data greatly improves multilingual performance while retaining strong English performance, with negligible performance degradation compared to fewer languages. The principal implication for AI practitioners is that massively multilingual LVLMs can be effectively trained with a balanced mix of English and multilingual data, even for low-resource languages, and incorporating synthetic OCR data can significantly enhance multilingual text-in-image understanding. |
| Building Foundations for Natural Language Processing of Historical Turkish: Resources and Models (Read more on arXiv or HuggingFace) | Ece Elif Adak, tcTHEBESTMAN, fatihburakkaragoz, temretiras, sbozates | The paper introduces new resources and models for natural language processing (NLP) of historical Turkish, a previously underexplored area. The main research objective is to develop foundational resources and models for NLP tasks in historical Turkish, including named entity recognition (NER), dependency parsing, and part-of-speech (POS) tagging. The key methodology involves creating and annotating datasets (HisTR, OTA-BOUN), compiling a clean text corpus (Ottoman Text Corpus - OTC), and fine-tuning transformer-based language models (BERTurk, mBERT, TURNA) on these resources. Primary results indicate that the BERTurk model fine-tuned on both MilliyetNER and HisTR achieved a 90.07 F1 score on the HisTR development set for NER. The principal implication for AI practitioners is that fine-tuning language-specific pre-trained models on domain-specific datasets is a viable approach for historical Turkish NLP, but challenges remain in adapting to out-of-domain data. |
| Entropy-Guided Attention for Private LLMs (Read more on arXiv or HuggingFace) | Brandon Reagen, nandan523 | This paper introduces an information-theoretic framework to optimize transformer architectures for privacy-preserving language model inference. The main research question is how the removal of nonlinearities in decoder-only language models impacts their training dynamics and expressiveness, particularly in the context of private inference (PI). The key methodology involves using Shannon’s entropy to analyze the dual role of nonlinearities in maintaining training stability and attention head diversity, and exploring PI-friendly alternatives like weight normalization and entropy regularization. A primary result is that the proposed entropy-guided attention mechanism with a Softmax-only model reduces communication overhead by 3.94x and improves end-to-end PI latency by 1.72x, compared to a baseline GPT-2 model with GELU and LayerNorm. The principal implication for AI practitioners is that entropy-guided attention can enable more efficient and scalable privacy-preserving inference for large language models by reducing reliance on computationally expensive nonlinear operations. |
Papers for 2025-01-09
| Title | Authors | Summary |
|---|---|---|
| rStar-Math: Small LLMs Can Master Math Reasoning with Self-Evolved Deep Thinking (Read more on arXiv or HuggingFace) | Youran Sun, Yifei Liu, Xinyu Guan, J-shang, lynazhang | rStar-Math demonstrates that small language models (SLMs) can achieve advanced math reasoning through self-evolved deep thinking. The main research question is whether SLMs can rival or surpass the mathematical reasoning capabilities of larger models like OpenAI’s models without distillation from superior models. The key methodology involves a novel code-augmented Chain-of-Thought data synthesis method, Monte Carlo Tree Search (MCTS) for test-time search guided by an SLM-based process reward model, and a four-round self-evolution recipe to iteratively improve the policy SLM and process preference model (PPM). The primary result is that rStar-Math improves the accuracy of the Qwen2.5-Math-7B model on the MATH benchmark from 58.8% to 90.0% with 64 search trajectories. The principal implication for AI practitioners is that they can leverage rStar-Math’s self-evolutionary framework to enhance the mathematical reasoning capabilities of SLMs without relying on larger, more resource-intensive models. |
| URSA: Understanding and Verifying Chain-of-thought Reasoning in Multimodal Mathematics (Read more on arXiv or HuggingFace) | Xinzhe Ni, Yiyao Yu, Yifan Wang, fun6668, AntimageTHU | URSA-7B is a new model for multimodal mathematical reasoning that uses chain-of-thought (CoT) supervision to improve performance. The main research question is how to enhance the CoT reasoning capabilities of Multimodal Large Language Models (MLLMs) in mathematical problem-solving using a new dataset and training method. The key methodology involves a three-module synthesis strategy that integrates CoT distillation, trajectory-format rewriting, and format unification to create a high-quality CoT reasoning instruction fine-tuning dataset, MMathCoT-1M, and a dual-view process supervision data synthesis to train a reward model, URSA-RM-7B. The primary results show that URSA-7B achieves state-of-the-art performance on multiple multimodal mathematical benchmarks, with a 97.1 pass@64 accuracy on the GPS task of MathVista. The principal implication for AI practitioners is that using high-quality CoT datasets and advanced process supervision can significantly enhance MLLMs’ mathematical reasoning capabilities, offering a pathway to improve performance in tasks requiring complex, multi-step reasoning. |
| Towards System 2 Reasoning in LLMs: Learning How to Think With Meta Chain-of-Though (Read more on arXiv or HuggingFace) | Kanishk Gandhi, Charlie Snell, Violet Xiang, nlile, Asap7772 | This paper introduces Meta Chain-of-Thought (Meta-CoT), a framework for enhancing reasoning in large language models (LLMs) by explicitly modeling the underlying thought processes involved in reaching a solution. The main research question is how to enable LLMs to perform complex reasoning analogous to System 2 cognitive processes by integrating search, verification, and iterative refinement into their operational framework. The key methodology involves process supervision, synthetic data generation via search algorithms (e.g. Monte Carlo Tree Search, A*), and reinforcement learning to train models on linearized search traces. Primary results indicate that models trained with Meta-CoT, specifically when using a backtracking strategy at a rate of 50% for incorrect steps, can achieve up to 94% accuracy on hard math problems, compared to 78% for standard Chain-of-Thought models. The principal implication for AI practitioners is that incorporating Meta-CoT into model training can significantly improve the ability of LLMs to solve complex reasoning tasks, suggesting that future model development should focus on integrating explicit search and verification mechanisms. |
| Agent Laboratory: Using LLM Agents as Research Assistants (Read more on arXiv or HuggingFace) | Jialian Wu, Ximeng Sun, Ze Wang, Yusheng Su, Samuel Schmidgall | Agent Laboratory is an autonomous LLM-based framework designed to conduct the entire research process, from literature review to experimentation and report writing, with optional human feedback. The main research question is whether this framework can accelerate scientific discovery, reduce research costs, and improve research quality. The key methodology involves a three-stage process: literature review using the arXiv API, experimentation using specialized agents and tools like mle-solver for code generation, and report writing with a module called paper-solver for iterative report generation and refinement. The primary results show that Agent Laboratory driven by o1-preview generates the best research outcomes, and human involvement at each stage improves the overall quality of research, with an 84% decrease in research expenses compared to previous autonomous research methods. The principal implication for AI practitioners is that Agent Laboratory can enable researchers to allocate more effort toward creative ideation rather than low-level coding and writing, potentially accelerating scientific discovery in machine learning. |
| LLM4SR: A Survey on Large Language Models for Scientific Research (Read more on arXiv or HuggingFace) | Xinya Du, Wei Yang, Ziming Luo, Ason-jay, ZonglinY | LLM4SR is a survey that systematically explores the application of large language models (LLMs) across the scientific research lifecycle. The main research question is how LLMs are being integrated into various stages of scientific research, including hypothesis discovery, experiment planning and implementation, scientific writing, and peer review. The key methodology used involves a comprehensive review and analysis of existing literature, focusing on task-specific methodologies, evaluation benchmarks, and the unique roles LLMs play in each research stage. The primary results indicate that LLMs have been used to generate novel hypotheses, with one study showing LLMs generating hypotheses in chemistry and material science published in high impact journals such as Nature or Science after the training cutoff date of the LLM; however, the paper does not explicitly state quantitative results across all stages. The principal implication for AI practitioners is that LLMs present significant opportunities for enhancing and automating various aspects of the scientific research process, but challenges remain in areas such as ensuring the validity of generated hypotheses and addressing ethical considerations. |
| InfiGUIAgent: A Multimodal Generalist GUI Agent with Native Reasoning and Reflection (Read more on arXiv or HuggingFace) | Xueyu Hu, Congkai Xie, Zishu Wei, Yuhang Liu, pengxiang | InfiGUIAgent is a multimodal GUI agent designed for task automation on computing devices, trained through a two-stage supervised fine-tuning pipeline. The main research objective is to develop a GUI agent with enhanced reasoning capabilities and reduced reliance on textual annotations. The key methodology involves two-stage supervised fine-tuning (SFT), with Stage 1 focusing on fundamental skills like GUI understanding and grounding using diverse datasets, and Stage 2 integrating hierarchical reasoning and expectation-reflection reasoning skills into synthesized data. Primary results show that InfiGUIAgent-2B achieves 76.3% accuracy on the ScreenSpot benchmark, surpassing several strong baselines. For AI practitioners, the principal implication is that a two-stage SFT approach incorporating hierarchical and expectation-reflection reasoning can significantly enhance GUI agents’ performance on benchmarks without reliance on additional GUI metadata, suggesting a path towards more robust and autonomous GUI automation. |
| GeAR: Generation Augmented Retrieval (Read more on arXiv or HuggingFace) | Hao Sun, Yuefeng Zhan, Jianfeng Liu, Shaohan Huang, noobimp | GeAR: Generation Augmented Retrieval introduces a novel method to enhance document retrieval with fine-grained information localization. The main research question is whether integrating information localization capabilities into existing retrievers is possible without sacrificing their retrieval capabilities. The key methodology involves constructing (query-document-information) triples and employing a text decoder to generate relevant fine-grained information from fused query and document representations, optimized with contrastive learning. The primary results show that GeAR achieves competitive performance on retrieval tasks, with a recall rate of 0.963 at rank 5 on the PAQ dataset, and effectively localizes information within documents. The principal implication for AI practitioners is that GeAR provides a flexible framework capable of handling both document retrieval and fine-grained unit localization simultaneously, offering new insights into the interpretation of retrieval results. |
| Chirpy3D: Continuous Part Latents for Creative 3D Bird Generation (Read more on arXiv or HuggingFace) | Chee Seng Chan, Jiankang Deng, Jia Wei Sii, Jing Yang, Kam Woh Ng | This paper introduces Chirpy3D, a novel framework for fine-grained, creative 3D bird generation using continuous part latents. The main research objective is to enable the generation of detailed and creative 3D objects by lifting 2D fine-grained understanding into 3D space and enabling part-level control. The key methodology involves fine-tuning a multi-view diffusion model (MVDream) with 2D images, modeling part latents as continuous Gaussian distributions, and introducing a self-supervised feature consistency loss. Primary results show that Chirpy3D effectively reconstructs 3D subjects, with a cosine similarity score of 0.724 for part composition, and generates novel species with diverse parts. The principal implication for AI practitioners is that Chirpy3D offers a new approach for generating high-quality, creative 3D assets with fine-grained control, which is directly applicable to improve creative freedom and output detail in 3D content creation. |
| SPAR3D: Stable Point-Aware Reconstruction of 3D Objects from Single Images (Read more on arXiv or HuggingFace) | Varun Jampani, James M. Rehg, Aaryaman Vasishta, Zixuan Huang, mboss | SPAR3D is a two-stage model for reconstructing 3D objects from single images. The main research question is how to combine the strengths of regression-based and diffusion-based methods for single-image 3D object reconstruction while avoiding their limitations. The key methodology involves a two-stage approach: first, a point diffusion model generates a sparse 3D point cloud, and second, a meshing stage uses the point cloud and the input image to create a detailed mesh. On the GSO dataset, SPAR3D achieves a Chamfer Distance (CD) of 0.120, outperforming prior methods. The principal implication for AI practitioners is that SPAR3D offers a computationally efficient approach to generate high-quality 3D meshes from single images, with an inference speed of 0.7 seconds per object, and enables interactive user edits. |
| DPO Kernels: A Semantically-Aware, Kernel-Enhanced, and Divergence-Rich Paradigm for Direct Preference Optimization (Read more on arXiv or HuggingFace) | Rajarshi Roy, Danush Khanna, Suranjana Trivedy, Amitava Das, amanchadha | Here is a concise summary of the AI research paper “DPO-Kernels: A Semantically-Aware, Kernel-Enhanced, and Divergence-Rich Paradigm for Direct Preference Optimization”: i) This paper introduces DPO-Kernels, an enhanced framework for direct preference optimization (DPO) that integrates kernel methods and alternative divergence measures to improve alignment of large language models with human preferences. ii) The main research objective is to address the limitations of standard DPO in aligning models with diverse human values and preferences by proposing a more expressive and adaptable framework. iii) The key methodology involves integrating kernelized representations (using polynomial, RBF, Mahalanobis, and spectral kernels), a hybrid loss function combining probability-based and embedding-based signals, and alternative divergence measures (Jensen-Shannon, Hellinger, Rényi, Bhattacharyya, Wasserstein, and f-divergences), along with data-driven selection of kernel-divergence pairs and a Hierarchical Mixture of Kernels (HMK). iv) Evaluations on 12 datasets show that DPO-Kernels, particularly HMK, achieve state-of-the-art generalization in factuality, safety, reasoning, and instruction-following tasks, with HMK demonstrating a performance improvement of up to 9.2% over baseline DPO. v) The principal implication for AI practitioners is that DPO-Kernels provide a more robust and flexible framework for preference alignment in large language models, but they must carefully consider the 3-4x higher computational costs associated with HMK. |
| EpiCoder: Encompassing Diversity and Complexity in Code Generation (Read more on arXiv or HuggingFace) | Xiao Liu, Jie Wu, Yaoxiang Wang, CharonBony, Ringo1110 | EpiCoder is a novel feature tree-based code synthesis framework designed to enhance the diversity and complexity of code generation. The main research question is how to generate more nuanced, diverse, and complex code instruction data that aligns with real-world programming scenarios. The key methodology involves a feature tree-based synthesis inspired by Abstract Syntax Trees (AST) that models semantic relationships between code elements, iteratively refined to enhance feature diversity. The primary results show that EpiCoder-Qwen-7B achieves state-of-the-art performance on function-level code generation benchmarks, with an 81.7% average pass rate on HumanEval and MBPP. The principal implication for AI practitioners is that using EpiCoder’s feature tree-based framework can significantly improve the quality and diversity of synthesized code data, leading to more robust and adaptable code generation models. |
Papers for 2025-01-08
| Title | Authors | Summary |
|---|---|---|
| REINFORCE++: A Simple and Efficient Approach for Aligning Large Language Models (Read more on arXiv or HuggingFace) | chuyi777 | REINFORCE++ is a novel variant of the REINFORCE algorithm designed to enhance the alignment of large language models (LLMs) with human preferences. The main research objective is to develop a more efficient and stable reinforcement learning from human feedback (RLHF) algorithm by simplifying the REINFORCE framework and removing the need for a critic network. Key methodologies include a token-level Kullback-Leibler (KL) penalty, Proximal Policy Optimization (PPO)-clip integration, mini-batch updates, and reward normalization. Primary results demonstrate that REINFORCE++ achieves comparable or superior performance to PPO and Group Relative Policy Optimization (GRPO), with a specific quantitative finding showing a reduction in training time from 60 hours (for PPO) to 42 hours on NVIDIA H100 with the LLaMA3 8b model. Principal implication for AI practitioners is that REINFORCE++ provides a simpler and more computationally efficient method for aligning LLMs, making it a valuable alternative to more complex RLHF approaches like PPO. |
| MotionBench: Benchmarking and Improving Fine-grained Video Motion Understanding for Vision Language Models (Read more on arXiv or HuggingFace) | Lefan Wang, Weihan Wang, Zhuoyi Yang, LiquidAmmonia, wenyi | MotionBench: A comprehensive benchmark for evaluating fine-grained video motion understanding in vision-language models (VLMs). The research objective was to assess the capability of VLMs in understanding fine-grained video motion and to improve VLM performance in this area. The key methodology involved creating a new benchmark, MotionBench, with diverse video sources and question types focusing on motion-level perception, along with proposing a novel Through-Encoder (TE) Fusion method for enhancing video feature representation. The primary results indicated that existing VLMs perform poorly in understanding fine-grained motions, achieving accuracies below 60% on MotionBench; TE Fusion yielded improvements in motion understanding. The paper does not clearly specify the improvement magnitude. The principal implication is that MotionBench provides a valuable resource for evaluating and improving video understanding VLMs, highlighting a significant deficiency in current models’ ability to handle fine-grained motion and offering a novel architectural approach to address this limitation. |
| Sa2VA: Marrying SAM2 with LLaVA for Dense Grounded Understanding of Images and Videos (Read more on arXiv or HuggingFace) | Shilin Xu, Zilong Huang, Tao Zhang, Xiangtai Li, HarborYuan | Sa2VA is a unified model for dense grounded understanding of images and videos, integrating SAM-2 and LLaVA-like models. The research objective was to create a model capable of handling a wide range of image and video tasks, including referring segmentation and conversation, within a single framework. The methodology involved a one-shot visual instruction tuning approach, unifying text, image, and video into a shared LLM token space. Sa2VA achieved state-of-the-art results on multiple benchmarks, exceeding GLaMM-7B by 2.1, 3.6, and 4.5 cIoU on RefCOCO, RefCOCO+, and RefCOCOg respectively. For AI practitioners, this work provides a unified, highly effective architecture and demonstrates that integrating powerful visual foundation models with LLMs is highly effective for a broad range of vision-language tasks, offering a superior approach to the design of multi-modal models. |
| Cosmos World Foundation Model Platform for Physical AI (Read more on arXiv or HuggingFace) | Yogesh Balaji, Maciej Bala, Arslan Ali, Niket Agarwal, NVIDIA | The Cosmos World Foundation Model Platform facilitates Physical AI development by providing pre-trained world models and tools for customization. The research objective was to create a platform for building and fine-tuning world foundation models (WFMs) for Physical AI applications. The methodology involved developing video data curation, pre-trained WFMs using diffusion and autoregressive models, video tokenizers, and post-training techniques. Results showed Cosmos Tokenizer achieved a 4dB PSNR improvement over existing tokenizers on the DAVIS dataset at 8× spatial compression. The platform’s open-source nature and model availability empower AI practitioners to build and deploy customized WFMs for their specific Physical AI systems, potentially accelerating development in various applications. |
| LLaVA-Mini: Efficient Image and Video Large Multimodal Models with One Vision Token (Read more on arXiv or HuggingFace) | Yang Feng, Zhe Yang, Qingkai Fang, Shaolei Zhang | LLaVA-Mini introduces an efficient large multimodal model using a single vision token to represent images and videos. The research objective was to develop efficient large multimodal models (LMMs) by minimizing the number of vision tokens while maintaining performance. The key methodology involved modality pre-fusion to fuse visual information into text tokens before feeding them into the LLM backbone, along with a compression module to reduce vision token quantity. Results show LLaVA-Mini outperforms LLaVA-v1.5 with only one vision token instead of 576, achieving a 77% reduction in FLOPs. This research demonstrates the feasibility of building highly efficient LMMs with significantly reduced computational costs, potentially leading to faster inference times and wider accessibility for real-time multimodal applications. |
| Diffusion as Shader: 3D-aware Video Diffusion for Versatile Video Generation Control (Read more on arXiv or HuggingFace) | Zhiyang Dou, Jiahao Lu, Rui Yan, Zekai Gu, pengHTYX | Diffusion as Shader (DaS) is a 3D-aware video diffusion model that enables versatile control over video generation by utilizing 3D tracking videos as conditional inputs. The main research objective is to develop a unified framework for video generation that supports multiple control tasks, such as mesh-to-video generation, camera control, motion transfer, and object manipulation. The key methodology involves using 3D tracking videos, which represent the motion trajectories of 3D points, as control inputs to a video diffusion model that acts as a shader to compute shaded appearances. The primary results demonstrate that DaS outperforms baseline methods on camera control, achieving a rotation error of 10.40 degrees and a translation error of 5.97 degrees on large camera movements, compared to 39.86 and 67.05 for MotionCtrl. For AI practitioners, the principal implication is that leveraging 3D tracking videos as control signals enables more precise and temporally consistent control over video generation compared to methods that rely solely on 2D control signals. |
| MoDec-GS: Global-to-Local Motion Decomposition and Temporal Interval Adjustment for Compact Dynamic 3D Gaussian Splatting (Read more on arXiv or HuggingFace) | Jihyong Oh, Won-Sik Cheong, Jun Young Jeong, Joonsoo Kim, Sangwoon Kwak | MoDec-GS is a memory-efficient 3D Gaussian splatting framework for reconstructing novel views from dynamic videos with complex motions. The research objective was to develop a method for efficiently representing and rendering dynamic scenes with complex motions, addressing limitations in existing methods regarding storage and representation of complex movements. MoDec-GS uses Global-to-Local Motion Decomposition (GLMD) and Temporal Interval Adjustment (TIA) to model complex motions effectively and efficiently. The results demonstrate a 70% average reduction in model size compared to state-of-the-art methods while maintaining or improving rendering quality; specifically, on the iPhone dataset, MoDec-GS achieved a 0.7dB PSNR gain and a 94% storage reduction compared to the second-best method. This work provides a highly compact and efficient approach for dynamic scene representation relevant to AI practitioners working on real-time video processing and novel view synthesis. |
| PPTAgent: Generating and Evaluating Presentations Beyond Text-to-Slides (Read more on arXiv or HuggingFace) | Hongyu Lin, Jia Zheng, Hao Kong, Xinyan Guan, Forceless | PPTAgent is a novel two-stage, edit-based framework for automatic presentation generation that leverages reference presentations and LLMs. The research aimed to improve presentation generation by addressing the limitations of existing text-to-slide methods. PPTAgent utilizes a two-stage process: presentation analysis (clustering slides and extracting schemas) and presentation generation (iterative editing of reference slides). Experiments showed that PPTAgent significantly outperformed baselines across three dimensions (Content, Design, Coherence), achieving an average score of 3.67 and a 97.8% success rate. This work provides a new approach for AI practitioners to generate high-quality presentations, improving efficiency and visual effectiveness in communication. |
| MagicFace: High-Fidelity Facial Expression Editing with Action-Unit Control (Read more on arXiv or HuggingFace) | Guoying Zhao, Huai-Qian Khor, Xingxun Jiang, Tuomas Varanka, Mengting Wei | MagicFace: High-fidelity facial expression editing using action unit (AU) variations as conditions within a Stable Diffusion framework. The research objective was to develop a method for high-fidelity facial expression editing that is both interpretable and controllable by adjusting AU variations. The methodology involved a diffusion model conditioned on AU variations, an ID encoder for identity preservation, and an Attribute Controller for maintaining background and pose consistency. The model was trained on a dataset of 30,000 image pairs. The primary result showed that MagicFace achieved a mean squared error (MSE) of 0.261 for AU intensity, outperforming other methods. The main implication for AI practitioners is the demonstration of precise and controllable facial expression editing using AU variations within a diffusion model framework; this offers improvements for generating photorealistic facial expressions for applications like virtual characters and avatars. |
| Magic Mirror: ID-Preserved Video Generation in Video Diffusion Transformers (Read more on arXiv or HuggingFace) | Zexin Yan, Bohao Peng, Bin Xia, Yaoyang Liu, julianjuaner | Magic Mirror: A novel framework for generating high-fidelity identity-preserved videos using video diffusion transformers. The research objective is to develop a method for generating high-quality, identity-preserved videos with dynamic motion, addressing the challenge of maintaining consistent identity while producing natural motion in existing text-to-video generation models. The methodology involves a dual-branch facial feature extractor, a lightweight cross-modal adapter with Conditioned Adaptive Normalization (CAN) for efficient identity integration, and a two-stage training strategy. The primary results demonstrate that Magic Mirror outperforms existing methods, achieving an average ID similarity of 0.911 while maintaining high video quality metrics and dynamic motion. The overall preference score from a user study was 7.315. The paper does not explicitly specify if the user study is statistically significant. The most impactful finding is the successful integration of identity preservation into a video diffusion transformer architecture without person-specific fine-tuning, offering a more efficient and scalable approach to personalized video generation. This has direct relevance for AI practitioners working with video diffusion models, as it provides a more efficient and effective method for identity-preserved video generation. |
| Dolphin: Closed-loop Open-ended Auto-research through Thinking, Practice, and Feedback (Read more on arXiv or HuggingFace) | Tao Chen, Botian Shi, Xiangchao Yan, Jiakang Yuan, BoZhang | DOLPHIN is a closed-loop open-ended auto-research framework automating the scientific research process. The research aims to create a fully automated scientific research system capable of generating research ideas, performing experiments, and iteratively refining ideas based on results. DOLPHIN employs LLMs for idea generation and code generation, incorporating an exception-traceback-guided debugging process. Experiments across three benchmark datasets demonstrated DOLPHIN generating methods comparable to state-of-the-art in some tasks; for example, a 2.9% improvement in ModelNet40 accuracy over the baseline. This work provides a significant advancement for AI practitioners in automating the scientific research process, though the paper lacks information regarding certain experimental setup details. |
Papers for 2025-01-07
| Title | Authors | Summary |
|---|---|---|
| STAR: Spatial-Temporal Augmentation with Text-to-Video Models for Real-World Video Super-Resolution (Read more on arXiv or HuggingFace) | yingtai, zhenheny, chenzhao, yinhongliu, SherryX | STAR introduces a novel approach for real-world video super-resolution using text-to-video models. The research objective was to enhance spatio-temporal quality in restored videos by addressing artifacts from complex degradations and mitigating fidelity loss from powerful generative models. The methodology involved a Local Information Enhancement Module (LIEM) and a Dynamic Frequency (DF) Loss. Results showed STAR outperforming state-of-the-art methods, achieving a 0.5422 DOVER score on the UDM10 dataset. This research highlights the significant potential of integrating text-to-video models and specifically designed loss functions for improving the fidelity and temporal consistency of real-world video super-resolution. |
| BoostStep: Boosting mathematical capability of Large Language Models via improved single-step reasoning (Read more on arXiv or HuggingFace) | lindahua, yhcao, KennyUTC, yuhangzang, BeichenZhang | Here’s a concise summary of the paper: i) BoostStep improves large language models’ mathematical reasoning by enhancing single-step reasoning through step-level in-context learning. ii) The main objective is to address the granularity mismatch and negative-effect noise in in-context learning examples to improve the reasoning quality within each step of a multi-step mathematical problem-solving process. iii) The key methodology is step-level in-context learning with a “first-try” strategy, which aligns the granularity between retrieving and reasoning on a step-by-step basis using an example problem bank constructed with step-level granularity. iv) Quantitatively, BoostStep improves GPT-4o’s performance on various mathematical benchmarks by 3.6% and Qwen2.5-Math-72B by 2.0%. v) For AI practitioners, BoostStep provides a method to enhance the mathematical reasoning ability of large language models without additional training, demonstrating the importance of fine-grained, step-level guidance in complex problem-solving. |
| Dispider: Enabling Video LLMs with Active Real-Time Interaction via Disentangled Perception, Decision, and Reaction (Read more on arXiv or HuggingFace) | myownskyW7, lindahua, yhcao, yuhangzang, Mar2Ding | Dispider is a novel system designed for active real-time interaction with streaming video using large language models (LLMs). The main research objective is to enable video LLMs to process and respond to streaming video input continuously and in real-time, unlike existing offline models. The key methodology is a disentangled architecture that separates perception, decision, and reaction into asynchronous modules operating in parallel, with a lightweight proactive streaming video processing module and an asynchronous interaction module. Primary results show that Dispider outperforms VideoLLM-online in the Proactive Output task with a score of 25.3, and achieves a leading performance of 55.6 on the EgoSchema benchmark. The principal implication for AI practitioners is that Dispider’s disentangled and asynchronous design enables more efficient and responsive real-time video interaction, making it ideal for long-duration video streams and maintaining strong performance in conventional video QA tasks. |
| Test-time Computing: from System-1 Thinking to System-2 Thinking (Read more on arXiv or HuggingFace) | Jia Xu, Kaixin Wu, Hai Ye, douvleplus, Yisam | This paper surveys test-time computing methods, focusing on their role in enabling the transition from System-1 to System-2 thinking in AI models. The main research question is how test-time computing can enhance the robustness, generalization, and reasoning ability of AI models, particularly large language models (LLMs). The methodology involves a comprehensive review and categorization of existing literature on test-time computing techniques, including test-time adaptation and test-time reasoning, applied to both System-1 and System-2 models. A primary result highlighted is that self-consistency Chain-of-Thought prompting can improve accuracy by 18% over vanilla Chain-of-Thought in math reasoning tasks. The principal implication for AI practitioners is that leveraging test-time computing strategies can significantly enhance model performance on downstream tasks, particularly in complex reasoning scenarios, without the need for retraining. |
| Personalized Graph-Based Retrieval for Large Language Models (Read more on arXiv or HuggingFace) | Franck-Dernoncourt, namyongp, Ojasmitha17, Tobilee, StevenAu | Personalized Graph-Based Retrieval for Large Language Models introduces a framework called PGraphRAG to enhance personalized text generation. The main research question is how to improve the performance of large language models (LLMs) in generating personalized text, especially in cold-start scenarios with sparse user data. The key methodology is PGraphRAG, a framework that leverages user-centric knowledge graphs to augment prompts with user-relevant context during the retrieval process. Primary results show that PGraphRAG significantly outperforms state-of-the-art personalization methods across diverse tasks, with a +32.1% improvement in ROUGE-1 for Hotel Experience Generation using the LLaMA-3.1-8B model. The principal implication for AI practitioners is that integrating structured user knowledge via PGraphRAG enhances the ability of LLMs to generate personalized and contextually appropriate text, particularly when user history is limited. |
| METAGENE-1: Metagenomic Foundation Model for Pandemic Monitoring (Read more on arXiv or HuggingFace) | willieneis, oliu-io, upup-ashton-wang, Johannes, oliu-io | METAGENE-1: A 7-billion parameter autoregressive transformer model is pretrained on a novel metagenomic dataset for pandemic monitoring. The research aimed to pretrain a foundation model on diverse metagenomic DNA and RNA sequences from human wastewater samples. Byte-pair encoding (BPE) tokenization was used for the dataset, and the model was pretrained using a decoder-style architecture. METAGENE-1 achieved state-of-the-art results on pathogen detection benchmarks, with a 92.96% average MCC score across four datasets. The successful pretraining of a large-scale metagenomic language model demonstrates the potential of this technology for applications in public health and opens up avenues for AI practitioners to develop and deploy similar models for diverse genomic tasks. |
| TransPixar: Advancing Text-to-Video Generation with Transparency (Read more on arXiv or HuggingFace) | Yijun Li, yingcongchen, HeZhang, zhifeichen097, wileewang | TransPixar introduces a method for generating RGBA videos from text prompts, addressing the challenge of producing transparent visual effects in text-to-video models. The research objective was to extend pretrained video models to generate RGBA videos while preserving original RGB capabilities. The methodology involved incorporating alpha-specific tokens and using LoRA-based fine-tuning within a diffusion transformer architecture, optimizing attention mechanisms to align RGB and alpha channels. A user study revealed a significant preference for TransPixar’s RGBA alignment (93.3%) over a comparable method (6.7%). This work demonstrates that high-quality RGBA video generation is achievable with limited training data using a modified DiT architecture, offering a practical advancement for creating realistic video effects with transparency for applications such as VFX. |
| Ingredients: Blending Custom Photos with Video Diffusion Transformers (Read more on arXiv or HuggingFace) | Di Qiu, MichaelFan, Changqian, Debang, onion | This paper introduces Ingredients, a framework for customizing video generation by incorporating multiple specific identity (ID) photos with video diffusion Transformers. The main research question is how to achieve multi-ID customization in video generation while preserving high-fidelity identity, enhancing content flexibility, and ensuring natural video generation. The key methodology involves a facial extractor for versatile facial feature capture, a multi-scale projector to map embeddings into the contextual space of image query in video diffusion Transformers, and an ID router for dynamically combining and allocating multiple ID embeddings to corresponding space-time regions, trained through a multi-stage protocol. The primary results show that the proposed Ingredients method achieved a face similarity score of 77.1% in multi-ID video generation, significantly outperforming baselines. The principal implication for AI practitioners is that Ingredients provides a training-free framework for multi-ID customization in video generation based on diffusion Transformers, enabling the preservation of multiple IDs while supporting precise textual control signals. |
| DepthMaster: Taming Diffusion Models for Monocular Depth Estimation (Read more on arXiv or HuggingFace) | Ruijie Zhu, Hao Zhang, Bo Li, Zerong Wang, Ziyang Song | DepthMaster is a single-step diffusion model designed for improved monocular depth estimation by adapting generative features to this discriminative task. The main research question is how to adapt generative features in diffusion models to enhance the performance of discriminative depth estimation while maintaining efficiency. The key methodology involves a Feature Alignment module to incorporate high-quality semantic features into the denoising network and a Fourier Enhancement module to balance low-frequency structure and high-frequency details in a single forward pass, using a two-stage training strategy. The primary results show that DepthMaster achieves state-of-the-art zero-shot performance, with an 8.2% AbsRel on the KITTI dataset. The principal implication for AI practitioners is that DepthMaster provides an effective way to leverage diffusion models for depth estimation with improved generalization and detail preservation, which is particularly beneficial for applications such as autonomous driving. |
| Through-The-Mask: Mask-based Motion Trajectories for Image-to-Video Generation (Read more on arXiv or HuggingFace) | Yaniv Taigman, Shelly Sheynin, Amit Zohar, Yuval Kirstain, GuyYariv | Through-The-Mask proposes a two-stage image-to-video generation framework using mask-based motion trajectories. The research objective was to improve the accuracy and consistency of object motion in generated videos, especially in multi-object scenarios. The methodology involved generating mask-based motion trajectories as an intermediate representation, conditioned on the input image, segmentation mask, and text prompt, followed by video generation conditioned on this representation. Results demonstrated state-of-the-art performance on several benchmarks, including a FVD score of 925.39 (U-Net) on the SA-V-128 benchmark. This work provides AI practitioners with a novel two-stage framework for I2V generation that significantly improves motion realism and consistency, particularly in complex scenes. |
| GS-DiT: Advancing Video Generation with Pseudo 4D Gaussian Fields through Efficient Dense 3D Point Tracking (Read more on arXiv or HuggingFace) | Yijin Li, Xiaoyu Shi, Zhaoyang Huang, Weikang Bian, wangfuyun | GS-DiT advances video generation by enabling 4D video control using pseudo 4D Gaussian fields and efficient dense 3D point tracking. The main research objective is to enable precise 4D control in video generation, such as multi-camera shooting and dolly zoom, without requiring expensive multi-view videos. The key methodology involves constructing a pseudo 4D Gaussian field with a novel dense 3D point tracking method (D3D-PT) and finetuning a pretrained Diffusion Transformer (DiT) to generate videos guided by the rendered videos from this field. The primary result is that D3D-PT outperforms SpatialTracker in accuracy and accelerates dense 3D point tracking by two orders of magnitude, achieving a 3D-AJ score of 9.0 on the TAPVid-3D minival split. The principal implication for AI practitioners is that GS-DiT enables 4D controllable video generation from monocular videos, broadening the applicability of advanced cinematic techniques in AI-driven video content creation. |
| Auto-RT: Automatic Jailbreak Strategy Exploration for Red-Teaming Large Language Models (Read more on arXiv or HuggingFace) | Weiqiang Wang, Huijia Zhu, Yaojie Lu, Shuhen Zhou, Yanjiang Liu | AUTO-RT is a reinforcement learning framework for automatically exploring and optimizing attack strategies to uncover security vulnerabilities in large language models (LLMs). The main research objective is to develop an automated red-teaming approach that can efficiently identify complex vulnerabilities in LLMs without relying on predefined safety flaws or fixed attack strategies. The key methodology involves two mechanisms: Early-terminated Exploration, which focuses on high-potential attack strategies, and a Progressive Reward Tracking algorithm that uses intermediate downgrade models to refine the search trajectory. The primary result is that AUTO-RT achieved a 16.63% higher success rate in detecting vulnerabilities compared to existing methods. The principal implication for AI practitioners is that they can use AUTO-RT to improve the efficiency of discovering vulnerabilities in LLMs, enabling more robust and secure language model development. |
| Samba-asr state-of-the-art speech recognition leveraging structured state-space models (Read more on arXiv or HuggingFace) | Kartik-angadi, kruthika, SyedAbdul | Samba-ASR is a novel speech recognition model utilizing state-space models (SSMs) for improved accuracy and efficiency. The main research objective is to develop an Automatic Speech Recognition (ASR) model that outperforms existing transformer-based models by leveraging the Mamba architecture. The key methodology involves replacing transformer encoders with Mamba’s state-space modeling in both the encoder and decoder, using a Mamba-cross-connection mechanism, and training on a combined dataset of LibriSpeech, GigaSpeech, and SPGISpeech. The primary result is that Samba-ASR achieved a Word Error Rate (WER) of 3.65% on average across multiple benchmark datasets, including a 1.17% WER on LibriSpeech Clean. For AI practitioners, Samba-ASR offers a new state-of-the-art model for speech recognition, demonstrating that SSMs can surpass transformers in accuracy and efficiency, particularly for long audio sequences. |
| ToolHop: A Query-Driven Benchmark for Evaluating Large Language Models in Multi-Hop Tool Use (Read more on arXiv or HuggingFace) | Yufei Xu, Xuesong Yao, Zhengyin Du, Junjie Ye, maverick1994 | Here is a concise summary of the research paper “ToolHop: A Query-Driven Benchmark for Evaluating Large Language Models in Multi-Hop Tool Use”: ToolHop is a new benchmark for evaluating large language models (LLMs) on multi-hop tool use, focusing on their ability to decompose complex queries and utilize multiple tools sequentially. The main research objective is to assess LLMs’ capabilities in understanding, reasoning, and function-calling within a multi-hop tool-use context. The key methodology involves a query-driven data construction process that includes tool creation, document refinement, and code generation, resulting in 995 multi-hop queries and 3,912 associated tools. The primary result is that the leading model, GPT-4o, achieved an accuracy of only 49.04% in the mandatory tool use scenario, highlighting significant limitations in current LLMs’ multi-hop tool-use abilities. The principal implication for AI practitioners is that there is substantial room for improvement in developing LLMs that can effectively handle complex multi-hop reasoning and tool-use tasks, as evidenced by the leading model’s relatively low performance. |
| Scaling Laws for Floating Point Quantization Training (Read more on arXiv or HuggingFace) | Kan Wu, Weidong Han, Ruobing Xie, Shuaipeng Li, Xingwu Sun | This paper explores scaling laws for floating-point quantization training in large language models (LLMs) to optimize low-precision training. The main research question is how do factors like data size, model size, exponent bits, mantissa bits, and block size of scaling factors affect the performance of LLMs under floating-point quantization training. The key methodology involves training 366 LLMs with various configurations and analyzing the relationships between these factors and model loss to formulate a unified scaling law. The primary result is a unified scaling law that accurately predicts LLM performance under different floating-point quantization settings, with the optimal floating-point quantization precision being directly proportional to computational power. The principal implication for AI practitioners is that they can use the derived scaling law to optimize the trade-off between computational cost and performance when training LLMs with floating-point quantization, particularly that the best cost-performance precision lies between 4-8 bits within a wide computational power range. |
Papers for 2025-01-06
| Title | Authors | Summary |
|---|---|---|
| EnerVerse: Envisioning Embodied Future Space for Robotics Manipulation (Read more on arXiv or HuggingFace) | jzzzzk, Shengcong, lyuukuu, pathcn, SiyuanH | Here’s a concise summary of the research paper: i) ENERVERSE is a comprehensive framework for embodied future space generation designed for robotic manipulation tasks, integrating a novel chunk-wise autoregressive diffusion model with a Free Anchor View (FAV) space and a 4D Gaussian Splatting (4DGS) data engine pipeline. ii) The main research objective is to develop a method for generating embodied future spaces that enhances a robot’s ability to perform long-range manipulation tasks by improving predictive capabilities and spatial understanding. iii) The key methodology involves a chunk-wise autoregressive diffusion model with a sparse contextual memory mechanism, a FAV-based 4D future space generation method, and a data flywheel pipeline integrating 4DGS optimization with multi-view video generation. iv) The proposed method achieved a state-of-the-art average success rate of 88.5 on the LIBERO benchmark with a Three Third View configuration. v) For AI practitioners, the principal implication is that integrating ENERVERSE’s future space generation prior into policy learning can significantly enhance the performance of robotic systems, particularly in complex, long-range manipulation tasks, by leveraging enhanced spatial understanding and a robust data generation pipeline. |
| VITA-1.5: Towards GPT-4o Level Real-Time Vision and Speech Interaction (Read more on arXiv or HuggingFace) | hertin, shenyunhang, yifanzhang114, xiongwang, linhaojia13 | VITA-1.5 is a multimodal large language model designed for real-time vision and speech interaction. The main research objective is to develop a model that integrates vision, language, and speech modalities without compromising performance due to modality differences. The key methodology involves a three-stage training process: vision-language training, audio input tuning, and audio output tuning, progressively incorporating each modality. The primary results show that VITA-1.5 achieves a Character Error Rate (CER) of 2.2 on the aishell-1 Mandarin speech recognition benchmark and maintains comparable performance to state-of-the-art models in vision tasks after audio training. The principal implication for AI practitioners is that VITA-1.5 provides an effective framework for building multimodal AI systems with near real-time vision and speech interaction capabilities, eliminating the need for separate ASR and TTS modules. |
| Virgo: A Preliminary Exploration on Reproducing o1-like MLLM (Read more on arXiv or HuggingFace) | jrwen, whenfra, yifanli, JohnCage, Richard1999 | Virgo is a multimodal slow-thinking system developed by fine-tuning a capable MLLM with a small amount of textual long-form thought data. The main research question is whether slow-thinking ability can be transferred across modalities through fine-tuning with text-based long-thought data and if this ability is comparable to that distilled from multimodal slow-thinking systems. The key methodology involves fine-tuning Qwen2-VL-72B-Instruct with textual and visual long-thought instruction datasets, including data distilled from other slow-thinking models. The primary result is that Virgo-72B, fine-tuned with 5K textual instructions, achieved 48.4% accuracy on MathVerse, which is comparable to or surpasses commercial reasoning systems. The principal implication for AI practitioners is that fine-tuning MLLMs with textual long-form thought data can effectively transfer slow-thinking capacities, suggesting a simpler approach to developing such systems. |
| VisionReward: Fine-Grained Multi-Dimensional Human Preference Learning for Image and Video Generation (Read more on arXiv or HuggingFace) | Jiajun Xu, Yuanming Yang, Jiale Cheng, Yu Huang, xujz0703 | Here is a concise summary of the research paper “VisionReward: Fine-Grained Multi-Dimensional Human Preference Learning for Image and Video Generation”: i) The paper introduces VisionReward, a fine-grained, multi-dimensional reward model for aligning visual generation models with human preferences, and a Multi-Objective Preference Optimization (MPO) algorithm for stable model tuning. ii) The main research objective is to develop a reward model that accurately and interpretably predicts human preferences in both image and video generation, addressing the limitations of existing reward models and optimization methods. iii) The key methodology involves decomposing human preferences into multiple dimensions, represented by a series of judgment questions, linearly weighted and summed to produce an interpretable score, and using a multi-objective preference learning algorithm to address confounding factors in preference data. iv) The primary results show that VisionReward surpasses existing methods in video preference prediction, outperforming VideoScore by 17.2% in accuracy. v) The principal implication for AI practitioners is that they can use VisionReward to better align image and video generation models with human preferences, leading to more satisfactory outputs in visual content creation. |
| Graph Generative Pre-trained Transformer (Read more on arXiv or HuggingFace) | XiaolinXu, y6q9, RArchered, Spony, xchen16 | 1. Summary: The paper introduces the Graph Generative Pre-trained Transformer (G2PT), an auto-regressive model that generates graphs as sequences of nodes and edges, utilizing a transformer decoder for next-token prediction, and explores fine-tuning for goal-oriented generation and property prediction. 2. Main research question or objective: The main objective is to develop an efficient graph generative model that leverages a novel sequence-based representation and auto-regressive transformer architecture. 3. Key methodology used: The key methodology involves representing graphs as sequences, training a transformer decoder on these sequences using next-token prediction, and applying fine-tuning strategies such as rejection sampling and reinforcement learning for downstream tasks. 4. Primary results: G2PT achieves superior performance on generic graph and molecule datasets; for instance, on the MOSES dataset, G2PT achieves a validity score of 97.2 and an FCD score of 1.02. 5. Principal implication for AI practitioners: AI practitioners can utilize G2PT as a versatile framework for graph generation and property prediction tasks, benefiting from its strong adaptability and superior performance demonstrated across multiple datasets. |
| LUSIFER: Language Universal Space Integration for Enhanced Multilingual Embeddings with Large Language Models (Read more on arXiv or HuggingFace) | anoperson, Franck-Dernoncourt, ryanrossi, ntnghia1811, Hieuman | LUSIFER is a zero-shot approach that enhances multilingual embeddings of English-centric large language models (LLMs) without requiring multilingual training data. The main research objective is to adapt LLM-based embedding models for multilingual tasks without requiring explicit multilingual supervision. The key methodology involves integrating a multilingual encoder (XLM-R) with an English-centric LLM (Mistral-7B) using a connector with minimal trainable parameters, trained in two stages: alignment and representation finetuning. The primary result is that LUSIFER achieved a state-of-the-art average score of 62.63 across 14 languages on five embedding tasks, outperforming the previous best baseline by 3.19 points. For AI practitioners, LUSIFER offers an effective method to enhance multilingual performance of English-centric LLM embedding models without the need for multilingual training data or architectural modifications, significantly improving performance in medium and low-resource languages. |
| BoxingGym: Benchmarking Progress in Automated Experimental Design and Model Discovery (Read more on arXiv or HuggingFace) | Louise Li, Lyle Goodyear, ngoodman, michaelyli, obiwan96 | BoxingGym is a benchmark for evaluating AI agents on scientific reasoning tasks. Main research question or objective: How well can current language models perform automated experimental design and model discovery in a variety of scientific domains? Key methodology used: The authors introduce BoxingGym, a benchmark with 10 environments based on real-world scientific models, where agents interact by proposing experiments, observing outcomes, and refining models, evaluated using expected information gain (EIG) and a communication-based model discovery metric. Primary results: GPT-4o struggles with both experimental design and model discovery, with an average standardized prediction error of 0.74 on the hyperbolic discounting choice task after 10 experiments. Augmenting the agent with an explicit statistical model does not reliably improve these results. Principal implication for AI practitioners: The benchmark highlights significant limitations of current large language models (LLMs) in performing scientific reasoning, suggesting a need for developing new methods for automated experimental design and model discovery. |
Papers for 2025-01-03
| Title | Authors | Summary |
|---|---|---|
| 2.5 Years in Class: A Multimodal Textbook for Vision-Language Pretraining (Read more on arXiv or HuggingFace) | Yongliang Shen, Jiashuo Sun, Xin Li, Hang Zhang, Wenqi Zhang | A high-quality multimodal textbook corpus, constructed from 2.5 years of instructional videos, is introduced for vision-language model (VLM) pretraining. The research aimed to create a more coherent, knowledge-rich interleaved corpus than existing web-crawled datasets. The methodology involved LLM-based video collection and filtering, followed by progressive extraction and refinement of visual (keyframes), audio (ASR), and textual knowledge (OCR) from the videos. Experiments demonstrated significantly improved pretraining performance, with VLMs achieving an average gain of +4.6% across seven benchmarks in 0-4 shot settings (e.g., +20% improvement on ScienceQA). The resulting textbook dataset offers superior interleaved context awareness, beneficial for improving VLM knowledge and reasoning capabilities. |
| VideoAnydoor: High-fidelity Video Object Insertion with Precise Motion Control (Read more on arXiv or HuggingFace) | Xiang Bai, Sihui Ji, Xi Chen, Hao Luo, Yuanpeng Tu | VideoAnydoor is a zero-shot video object insertion framework achieving high-fidelity detail preservation and precise motion control. The research objective was to develop a method for accurately preserving object identity and precisely controlling object motion during video insertion. The methodology involved an end-to-end framework utilizing an ID extractor, a pixel warper for fine-grained motion control, and a reweighted reconstruction loss. Quantitative results showed VideoAnydoor outperforming existing methods, achieving a 37.7 PSNR score, exceeding previous state-of-the-art techniques. This work provides AI practitioners with a robust, end-to-end framework for high-fidelity video object insertion and precise motion control, applicable to various downstream tasks without task-specific fine-tuning. |
| CodeElo: Benchmarking Competition-level Code Generation of LLMs with Human-comparable Elo Ratings (Read more on arXiv or HuggingFace) | Dayiheng Liu, Bo Zheng, Bowen Yu, Jiaxi Yang, Shanghaoran Quan | CODEELO is a benchmark for evaluating large language models (LLMs) on competition-level code generation using human-comparable Elo ratings. The main research objective is to develop a standardized benchmark that addresses limitations of existing benchmarks, such as the unavailability of private test cases and misaligned execution environments, to effectively assess LLMs’ coding abilities at a competitive level. The key methodology involves submitting LLM-generated code to the CodeForces platform for judging and calculating Elo ratings based on the performance, aligned with the platform’s system but with lower variance. The primary results show that the 01-mini model achieved the highest Elo rating of 1578, surpassing nearly 90% of human participants, while most other models struggled, with many falling in the lowest 20th percentile of human competitors. The principal implication for AI practitioners is that enhancing the length of the chain-of-thought (CoT) presents a promising avenue for improving LLMs’ reasoning abilities in code generation, as evidenced by the significant performance of 01-mini and QwQ-32B-Preview. |
| VideoRefer Suite: Advancing Spatial-Temporal Object Understanding with Video LLM (Read more on arXiv or HuggingFace) | Boqiang Zhang, Zesen Cheng, Wentong Li, Hang Zhang, Yuqian Yuan | VideoRefer Suite introduces a benchmark and model for fine-grained spatial-temporal video understanding. The research objective was to improve Video LLMs’ ability to understand fine-grained spatial and temporal details in videos. A multi-agent data engine created a large-scale object-level video instruction dataset (VideoRefer-700K), and a VideoRefer model with a versatile spatial-temporal object encoder was developed. VideoRefer achieved a 3.46 average score on the VideoRefer-BenchD benchmark (a multi-dimensional evaluation of description generation), exceeding existing methods. This work provides a valuable resource (dataset, model, benchmark) for advancing Video LLM capabilities, particularly in applications requiring fine-grained object-level understanding. |
| Reconstruction vs. Generation: Taming Optimization Dilemma in Latent Diffusion Models (Read more on arXiv or HuggingFace) | Xinggang Wang, Jingfeng Yao | Latent diffusion models with high-dimensional visual tokenizers exhibit an optimization dilemma: improved reconstruction quality comes at the cost of degraded generation performance. The research objective is to address the optimization dilemma in latent diffusion models by improving the training efficiency and generative performance of high-dimensional visual tokenizers. The key methodology is to align the latent space of the visual tokenizer with pre-trained vision foundation models during training, using a novel vision foundation model alignment loss (VF Loss). The primary result shows a significant improvement in training speed; achieving an FID score of 2.11 in just 64 epochs—a 21x speedup compared to the original DiT. Additionally, the integrated system achieved state-of-the-art performance on ImageNet 256x256 generation with an FID score of 1.35. The principal implication for AI practitioners is that the proposed VA-VAE and LightningDiT framework offers a practical solution to a common problem in latent diffusion models, enabling faster convergence and improved generation performance with higher-dimensional tokenizers. |
| ProgCo: Program Helps Self-Correction of Large Language Models (Read more on arXiv or HuggingFace) | Wenbo Su, Jiaheng Liu, Weixun Wang, Yanan Wu, Xiaoshuai Song | ProgCo improves large language model (LLM) self-correction by integrating program-driven verification and refinement. The research aimed to enhance LLM self-correction, particularly for complex reasoning tasks, where existing methods often fail. ProgCo uses self-generated and self-executed verification pseudo-programs to achieve more robust verification, followed by dual refinement of both responses and programs. Experiments showed ProgCo achieved significant improvements, for example, a 5.8% accuracy increase on the MATH dataset with one round of self-correction. This work suggests that incorporating program-driven techniques can significantly improve LLM self-correction capabilities, impacting development of more reliable and robust AI systems. |
| A3: Android Agent Arena for Mobile GUI Agents (Read more on arXiv or HuggingFace) | Guozhi Wang, Liang Liu, Jiayu Zhang, Hanhao Li, Yuxiang Chai | Android Agent Arena (A3) introduces a novel evaluation platform for mobile GUI agents. The research aims to address limitations of existing datasets and benchmarks by providing a comprehensive, interactive evaluation platform for mobile GUI agents operating in real-world scenarios. A3 employs a dynamic evaluation approach incorporating 201 tasks across 21 widely used third-party apps and leverages business-level LLMs for automated task evaluation. Results showed GPT-40 achieved 84% accuracy in LLM-based evaluation of task completion. A3 offers AI practitioners a more realistic and scalable evaluation framework for assessing the performance of mobile GUI agents. |
| MapEval: A Map-Based Evaluation of Geo-Spatial Reasoning in Foundation Models (Read more on arXiv or HuggingFace) | Md Hasebul Hasan, Md Tanvir Parvez, Md Tanvir Hassan, Mahir Labib Dihan, eunus | MAPEVAL is a benchmark for evaluating geo-spatial reasoning in foundation models. The main research objective is to assess foundation models’ ability to handle diverse and complex map-based user queries requiring geo-spatial reasoning. The key methodology used is a new benchmark called MAPEVAL, comprising 700 unique multiple-choice questions across three task types (textual, API-based, and visual) that test spatial relationships, map infographics, travel planning, and navigation. The primary result is that Claude-3.5-Sonnet, GPT-4o, and Gemini-1.5-Pro performed competitively, but Claude-3.5-Sonnet agents outperformed GPT-4o and Gemini-1.5-Pro by 16% and 21% respectively in the MAPEVAL-API task. The principal implication for AI practitioners is that MAPEVAL provides a critical tool for advancing general-purpose foundation models with stronger geo-spatial understanding, as evidenced by the significant performance gaps observed even among the most advanced models. |
| Dynamic Scaling of Unit Tests for Code Reward Modeling (Read more on arXiv or HuggingFace) | Sijia Luo, Jifan Yu, Jing Zhang, Xiaokang Zhang, KAKA22 | This paper investigates improving code generation accuracy by scaling the number of unit tests used for reward modeling. The research objective was to determine if increasing unit test quantity enhances reward signal quality, leading to better code selection. A unit test-based majority voting framework was employed, coupled with a novel unit test generator (CodeRM-8B) and dynamic scaling based on problem difficulty. Results show a positive correlation between unit test quantity and reward signal quality, with a specific finding of an 18.43% performance gain for Llama3-8B on HumanEval Plus. This research indicates that scaling unit tests, particularly using CodeRM-8B and dynamic scaling, can significantly enhance code generation performance in LLMs, providing a practical method for improving model accuracy. |
| MLLM-as-a-Judge for Image Safety without Human Labeling (Read more on arXiv or HuggingFace) | Felix Juefei-Xu, Xiaowen Lin, Shiyu Zhao, Shuming Hu, Zhenting Wang | This paper investigates zero-shot image safety judgment using pre-trained Multimodal Large Language Models (MLLMs). The main objective is to determine if unsafe images can be detected without human labeling, solely by querying MLLMs using a predefined safety constitution. The proposed method, CLUE, involves objectifying safety rules, assessing rule-image relevance, using debiased token probabilities for judgment, and employing cascaded chain-of-thought reasoning. Experiments demonstrate high effectiveness, achieving 95.9% recall and 94.8% accuracy with InternVL2-76B on a complex safety constitution. This work suggests a scalable, human-labeling-free approach for image safety assessment, potentially significantly reducing costs associated with existing methods. |
| MapQaTor: A System for Efficient Annotation of Map Query Datasets (Read more on arXiv or HuggingFace) | Md Rizwan Parvez, Mohammed Eunus Ali, mahirlabibdihan | MapQATOR is a web application designed to efficiently create reproducible map-based question-answering datasets for evaluating large language models’ geospatial reasoning capabilities. The research objective was to develop a system for streamlined annotation of map-based QA datasets, overcoming challenges in creating reliable geospatial QA data. The methodology involved building a plug-and-play web application integrating with multiple map APIs, incorporating data visualization tools, and utilizing a caching mechanism to ensure data consistency. Results demonstrated a 30x speedup in annotation compared to manual methods. The principal implication for AI practitioners is that MapQATOR significantly accelerates the creation of high-quality, reproducible geospatial datasets crucial for training and benchmarking LLMs on complex reasoning tasks. |
| Understanding and Mitigating Bottlenecks of State Space Models through the Lens of Recency and Over-smoothing (Read more on arXiv or HuggingFace) | Jiajun Zhu, Yuehao Wang, Ruisi Cai, Peihao Wang, pragsri8 | Structured State Space Models (SSMs) are investigated for their limitations in capturing long-range dependencies. The research aims to understand and mitigate bottlenecks in SSMs, focusing on recency bias and over-smoothing. A novel polarization technique, modifying state transition matrices, is proposed and empirically evaluated. Results show that polarization consistently improves associative recall accuracy of long-range tokens (e.g., a 93.43% average accuracy in one experiment), unlocking the benefits of deeper architectures in SSMs. This work highlights the inherent limitations of SSMs regarding recency and over-smoothing, directly impacting their scalability and robustness for long sequence processing and suggesting design modifications for improved performance. |
| SeedVR: Seeding Infinity in Diffusion Transformer Towards Generic Video Restoration (Read more on arXiv or HuggingFace) | Ceyuan Yang, Yang Zhao, Meng Wei, Zhijie Lin, Jianyi Wang | SeedVR: a novel diffusion transformer for generic video restoration. The research objective was to develop a diffusion transformer model capable of handling real-world video restoration with arbitrary length and resolution. The key methodology involved a shifted window attention mechanism within a diffusion transformer, a causal video variational autoencoder (CVVAE) for efficient compression, and a multi-stage progressive training strategy. SeedVR demonstrated impressive restoration capabilities; for example, it outperformed existing methods on several benchmark datasets, achieving a 10.508 DOVER score on the SPMCS dataset. The most impactful finding, relevant for AI practitioners, is SeedVR’s superior efficiency compared to existing diffusion-based video restoration approaches, achieving over 2x faster inference speed despite having a larger parameter count. The details regarding the comparison of training time are unclear. |
| SeFAR: Semi-supervised Fine-grained Action Recognition with Temporal Perturbation and Learning Stabilization (Read more on arXiv or HuggingFace) | Haozhou Sun, Zihan Jia, Zhenbang Xu, Haodong Chen, Yongle Huang | SeFAR: Semi-supervised Fine-grained Action Recognition with Temporal Perturbation and Learning Stabilization proposes a novel semi-supervised learning framework for fine-grained action recognition. The research objective is to develop a robust method for fine-grained action recognition using limited labeled data, addressing challenges inherent in existing large language models. The methodology incorporates dual-level temporal element modeling, moderate temporal perturbation as a strong augmentation strategy, and adaptive regulation to stabilize the learning process. SeFAR achieves state-of-the-art performance on fine-grained datasets, outperforming other methods by margins such as 7.8% to 8.4% increase in accuracy on FineDiving depending on the labeling rate. This research demonstrates a significant improvement in semi-supervised fine-grained action recognition and provides AI practitioners with a novel framework applicable to vision-based tasks involving nuanced temporal dynamics and limited data. |
Papers for 2025-01-02
| Title | Authors | Summary |
|---|---|---|
| OS-Genesis: Automating GUI Agent Trajectory Construction via Reverse Task Synthesis (Read more on arXiv or HuggingFace) | Yian Wang, Chuanyang Jin, Kanzhi Cheng, heroding77, QiushiSun | OS-Genesis is a novel pipeline that automates the generation of high-quality trajectory data for training GUI agents without human supervision or predefined tasks. The main research question is how to automatically construct diverse and high-quality GUI agent trajectories to improve their performance on complex computer tasks. The key methodology is a reverse task synthesis process involving interaction-driven exploration of GUI environments to collect state-action triplets, followed by the generation of low-level and high-level instructions using an annotation model and a trajectory reward model to ensure data quality. The primary result is that agents trained with OS-Genesis showed significant performance improvements on online benchmarks, such as achieving a 17.41% success rate on AndroidWorld compared to 9.82% for the self-instruction baseline. The principal implication for AI practitioners is that OS-Genesis provides an effective method for generating high-quality training data for GUI agents, which can significantly improve their ability to automate complex real-world computer tasks, particularly in dynamic environments. |
| Xmodel-2 Technical Report (Read more on arXiv or HuggingFace) | Jiang Ling, Qu Zhijiu, Lin Qingquan, Liu Yang, valeriaWong | Xmodel-2 is a 1.2 billion-parameter language model designed for reasoning tasks, emphasizing efficiency and performance. The main research question is how to optimize a language model for complex reasoning while maintaining low training costs and efficiency. The key methodology involves using the Warmup-Stable-Decay (WSD) learning rate scheduler, optimizing data ratios during the decay phase of training, and employing an architecture that allows different model scales to share a unified set of hyperparameters. The primary results show that Xmodel-2 achieves state-of-the-art performance among 1B-parameter models in complex reasoning tasks, with an average score of 39.62 on complex reasoning benchmarks (GSM8K, MATH, BBH, MMLU, HumanEval, and MBPP). The principal implication for AI practitioners is that Xmodel-2 provides a strong, efficient model for reasoning tasks, demonstrating the effectiveness of the WSD learning rate scheduler and data ratio optimization in enhancing model performance. |
Papers for 2025-01-01
| Title | Authors | Summary |
|---|---|---|
| Explanatory Instructions: Towards Unified Vision Tasks Understanding and Zero-shot Generalization (Read more on arXiv or HuggingFace) | Tao Yuan, Yuxin Song, Yifan Sun, Xiu-Shen Wei, axxkaya | The paper introduces Explanatory Instructions, a method for defining computer vision (CV) tasks through natural language descriptions of transformations between input and output images, to improve zero-shot generalization. The main research question is whether Explanatory Instructions can enable vision-language models (VLMs) to genuinely understand and generalize to unseen CV tasks. The key methodology involves constructing a dataset (DECVT) with 12 million triplets of “image input → explanatory instruction → output” and training an auto-regressive-based VLM on these instructions. The primary results show that the trained model achieved instruction-level zero-shot capabilities and promising task-level zero-shot capabilities on certain tasks; for instance, it achieved a F1 score of 20.69 on the zero-shot Canny-to-Image task using the MultiGen-20M dataset. The principal implication for AI practitioners is that Explanatory Instructions can enhance VLMs’ ability to perform novel vision tasks without explicit training, although the model’s task-level zero-shot generalization ability remains unstable and requires further development. |
| On the Compositional Generalization of Multimodal LLMs for Medical Imaging (Read more on arXiv or HuggingFace) | Yonglin Deng, Weihong Wang, Rongsheng Wang, Junying Chen, Zhenyang Cai | This paper investigates the compositional generalization (CG) capabilities of Multimodal Large Language Models (MLLMs) for medical imaging. The main research question is whether MLLMs can leverage CG to understand unseen medical images by recombining learned elements (Modality, Anatomical area, and Task). The key methodology involved constructing a dataset called Med-MAT from 106 medical datasets, defining the MAT-Triplet, and evaluating MLLMs’ ability to generalize to unseen combinations of these elements through multi-task training and controlled variable experiments. A primary result is that MLLMs trained on multiple tasks achieved 96% accuracy on subset 02 in the in-distribution dataset, significantly outperforming single-task training and demonstrating the effectiveness of CG. The principal implication for AI practitioners is that leveraging CG in MLLMs by training with diverse datasets sharing MAT-Triplets can significantly enhance the models’ ability to understand and generalize to unseen medical images, which has a direct impact on the development of robust medical imaging applications. |
| Bringing Objects to Life: 4D generation from 3D objects (Read more on arXiv or HuggingFace) | Gal Chechik, Dvir Samuel, Ori Malca, Ohad Rahamim | This paper introduces 3to4D, a novel method for generating 4D content from static 3D objects and text prompts. The main research question is how to animate user-provided 3D objects while maintaining their identity and adhering to textual prompts that describe the desired motion. The key methodology involves first converting a 3D mesh into a static 4D Neural Radiance Field (NeRF), then animating it using an Image-to-Video diffusion model conditioned on the initial object and text prompt, with an incremental viewpoint selection protocol and masked Score Distillation Sampling (SDS) loss for improved motion realism. The primary results show that 3to4D outperforms baseline methods, achieving a threefold improvement in identity preservation measured using LPIPS scores (15.0 ±0.1 for 3to4D vs. 44.3 ± 0.2 for the best-performing baseline). The principal implication for AI practitioners is that 3to4D provides a method for creating custom 4D animations from existing 3D assets, leveraging text prompts to guide the desired motion while preserving the original object’s visual characteristics. |
| Efficiently Serving LLM Reasoning Programs with Certaindex (Read more on arXiv or HuggingFace) | Zhongdongming Dai, Zheyu Fu, Siqi Zhu, Junda Chen, Yichao Fu | Dynasor is a system designed to optimize inference-time compute for Large Language Model (LLM) reasoning queries by dynamically allocating resources based on model certainty. The main research question is how to efficiently serve LLM reasoning programs that refine outputs by exploring multiple solution paths. The key methodology involves tracking and scheduling requests within reasoning queries using certaindex, a proxy that measures statistical reasoning progress based on model certainty, to guide compute allocation dynamically. Dynasor reduces compute by up to 50% in batch processing and sustains 3.3x higher query rates or 4.7x tighter latency SLOs in online serving compared to prior state-of-the-art systems. The principal implication for AI practitioners is that Dynasor enables more efficient deployment of LLM reasoning algorithms in real-world applications by optimizing resource use and improving response times. |
| TangoFlux: Super Fast and Faithful Text to Audio Generation with Flow Matching and Clap-Ranked Preference Optimization (Read more on arXiv or HuggingFace) | Rafael Valle, Ambuj Mehrish, Zhifeng Kong, Navonil Majumder, Chia-Yu Hung | TangoFlux is a text-to-audio model that uses flow matching and CLAP-ranked preference optimization for fast and high-quality audio generation. The main research objective is to develop an efficient text-to-audio (TTA) generative model that addresses the challenges of aligning TTA models due to the difficulty of creating preference pairs. The key methodology used is CLAP-Ranked Preference Optimization (CRPO), which iteratively generates and optimizes preference data using a CLAP model as a proxy reward model. The primary results show that TangoFlux achieves state-of-the-art performance with a CLAP score of 0.480 and an FD score of 75.1 in just 3.7 seconds using 515M parameters. The principal implication for AI practitioners is that TangoFlux provides a fast and efficient method for generating high-quality audio with fewer trainable parameters, which can be particularly useful in scenarios where inference time and computational resources are constrained. |
| Edicho: Consistent Image Editing in the Wild (Read more on arXiv or HuggingFace) | Ceyuan Yang, Qiuyu Wang, Yinghao Xu, Hao Ouyang, Qingyan Bai | The paper introduces Edicho, a training-free method for consistent image editing across multiple images using diffusion models. The main research question is how to achieve consistent image editing across diverse in-the-wild images without requiring training. The key methodology involves leveraging pre-estimated explicit image correspondence to guide a modified attention mechanism and classifier-free guidance during the denoising process of diffusion models. The primary results show that Edicho achieves a text alignment score of 0.3228 and an editing consistency score of 0.9355 in global image editing tasks, outperforming existing methods. For AI practitioners, Edicho offers a plug-and-play solution for consistent image editing that can be integrated with existing diffusion-based editing models, enabling applications like generating consistent image sets and 3D reconstruction of edits. |
| Do NOT Think That Much for 2+3=? On the Overthinking of o1-Like LLMs (Read more on arXiv or HuggingFace) | Jianhui Pang, Zhiwei He, Tian Liang, Jiahao Xu, Xingyu Chen | This paper investigates the phenomenon of “overthinking” in o1-like large language models (LLMs), where these models expend excessive computational resources on simple tasks. The main research question is how to quantify and mitigate overthinking in o1-like LLMs during inference. The key methodology involves analyzing solution distributions and proposing outcome and process efficiency metrics, alongside self-training strategies to optimize response generation. A primary result is that the o1-like model QwQ-32B-Preview used 1,953% more tokens than conventional models for the simple query “what is the answer of 2 plus 3?”. The principal implication for AI practitioners is the need to optimize inference efficiency in o1-like LLMs by addressing overthinking, potentially reducing computational overhead without compromising accuracy using methods like self-training with response simplification. |
| Facilitating large language model Russian adaptation with Learned Embedding Propagation (Read more on arXiv or HuggingFace) | Daniil Chernyshev, RefalMachine | This paper introduces Learned Embedding Propagation (LEP) as a cost-effective method for adapting large language models (LLMs) to new languages, specifically Russian, without full retraining. The main research objective is to address the limitations of language adaptation posed by restricted access to high-quality instruction-tuning data and the computational expense of full LLM retraining. The key methodology involves training a new tokenization vocabulary, initializing new embeddings by averaging existing ones, and then propagating these embeddings to an instruction-tuned model using linear transformations derived from fine-tuned variants. The primary results show that LEP applied to LLaMa-3-8B and Mistral-7B achieves competitive performance levels, with the LEP-Extended variant of OpenChat 3.5 achieving a Micro-Avg score of 0.632 on the Darumeru benchmark after calibration. For AI practitioners, the principal implication is that LEP offers a viable and efficient alternative to traditional language-specific instruction-tuning, significantly reducing the costs associated with language adaptation while maintaining or surpassing existing performance benchmarks. |
| OneKE: A Dockerized Schema-Guided LLM Agent-based Knowledge Extraction System (Read more on arXiv or HuggingFace) | Mengshu Sun, Lin Yuan, Kangwei Liu, Xiangyuan Ru, Yujie Luo | OneKE is a dockerized, schema-guided, large language model (LLM) agent-based knowledge extraction system designed for diverse data types and domains. The main research objective is to develop a comprehensive system that can extract knowledge from various data sources following complex schemas and handle debugging/error correction effectively. The key methodology involves a multi-agent design with a configurable knowledge base, utilizing Schema, Extraction, and Reflection Agents to process data, extract information, and refine results, respectively. The primary results show that using the Case Retrieval method, the Extraction Agent achieved significant performance improvements on both CrossNER and NYT-11-HRL datasets, with F1 scores increasing substantially compared to the vanilla method. The principal implication for AI practitioners is that OneKE provides a flexible and adaptable framework for knowledge extraction tasks, supporting various LLMs and data formats without requiring fine-tuning, while the Case Repository enables continuous improvement through error correction. |
| Slow Perception: Let’s Perceive Geometric Figures Step-by-step (Read more on arXiv or HuggingFace) | Liang Zhao, Jia Wang, Yumeng Li, Youyang Yin, Haoran Wei | The paper introduces “Slow Perception,” a novel approach for parsing geometric figures in images by mimicking human-like gradual perception. Main research question or objective: How to improve the accuracy of geometric figure parsing in images by Large Vision Language Models (LVLMs)? Key methodology used: The authors propose a two-stage “Slow Perception” (SP) framework: a) perception decomposition, breaking down complex figures into basic units (points and lines); and b) perception flow, using a “perceptual ruler” to trace lines stroke-by-stroke, avoiding “long visual jumps.” Primary results: SP improves the F1-score of geometric parsing by 6.1% over the baseline when using a perceptual ruler length of 4 in the test set. Slow perception also exhibits an inference time scaling law, where shorter perceptual ruler lengths lead to longer inference times but improved performance. Principal implication for AI practitioners: AI practitioners can leverage the slow perception framework to enhance the accuracy of geometric figure parsing, particularly in applications requiring precise spatial reasoning, and this framework may offer a new pathway to better performance in other visual tasks. |
| PERSE: Personalized 3D Generative Avatars from A Single Portrait (Read more on arXiv or HuggingFace) | Hanbyul Joo, Inhee Lee, Hyunsoo Cha | PERSE is a method for creating animatable 3D avatars from a single portrait image with controllable facial attributes. The main research question is how to build a 3D personalized generative avatar from a single reference portrait image that allows for continuous and disentangled control over various facial attributes while preserving the individual’s identity. The key methodology involves synthesizing large-scale 2D video datasets with facial attribute editing, and training a 3D Gaussian Splatting-based avatar model with a novel latent space regularization technique using interpolated 2D faces as supervision. The primary result is that PERSE generates high-quality avatars with an FID score of 214.46 on interpolated renderings. The principal implication for AI practitioners is that PERSE provides a novel approach for creating personalized 3D avatars with controllable attributes from a single image, offering a valuable tool for applications in VR/AR environments. |
| Training Software Engineering Agents and Verifiers with SWE-Gym (Read more on arXiv or HuggingFace) | Navdeep Jaitly, Graham Neubig, Xingyao Wang, alsuhr, Jiayi-Pan | SWE-Gym is a new benchmark for evaluating software engineering agents on real-world coding tasks. The main research objective is to develop and assess a training environment, SWE-Gym, for improving the performance of language model-based software engineering agents. The key methodology involves fine-tuning language models on agent trajectories sampled from SWE-Gym and employing verifiers trained on these trajectories for inference-time scaling. Primary results show that fine-tuning on SWE-Gym improves agents’ performance, achieving a 32.0% resolve rate on the SWE-Bench Verified test set. The principal implication for AI practitioners is that SWE-Gym can be used to train and improve software engineering agents through scalable learning methods. |
| HumanEval Pro and MBPP Pro: Evaluating Large Language Models on Self-invoking Code Generation (Read more on arXiv or HuggingFace) | Xiao-Ping Zhang, Arman Cohan, Yilun Zhao, Zhaojian Yu | The paper introduces HumanEval Pro and MBPP Pro, benchmarks for evaluating large language models (LLMs) on self-invoking code generation tasks. The main research question is how well LLMs can generate code that solves a complex problem by invoking their own solution to a related, simpler base problem. The key methodology involves generating new, more complex versions of existing benchmarks (HumanEval and MBPP) by creating self-invoking problems that require using the solution of a base problem and evaluating over twenty LLMs using metrics like pass@1. The primary result is that most LLMs experience a significant performance drop on self-invoking tasks compared to traditional code generation; for example, o1-mini achieves 96.2% pass@1 on HumanEval but only 76.2% on HumanEval Pro. The principal implication for AI practitioners is that current LLMs, while proficient in generating code for isolated tasks, still struggle with more complex, multi-step reasoning required for self-invoking code generation, highlighting a crucial area for further development in code-generating models. |
Papers for 2024-12-31
| Title | Authors | Summary |
|---|---|---|
| Explanatory Instructions: Towards Unified Vision Tasks Understanding and Zero-shot Generalization (Read more on arXiv or HuggingFace) | Tao Yuan, Yuxin Song, Yifan Sun, Xiu-Shen Wei, axxkaya | The research introduces Explanatory Instructions, a novel approach for defining computer vision tasks through linguistic descriptions, to improve zero-shot generalization in vision-language models. The main research objective is to enable vision-language models to genuinely understand and generalize to unseen vision tasks by using detailed linguistic transformations from input to output images. The key methodology involves creating a dataset (DECVT) with 12 million “image input → explanatory instruction → output” triplets and training an auto-regressive-based vision-language model (AR-based VLM) on this dataset. The primary results show that the trained model achieved instruction-level zero-shot capabilities and demonstrated promising vision task-level zero-shot generalization, with the model achieving a 20.69 F1 score on the Canny-to-Image task using unseen instructions. The principal implication for AI practitioners is that Explanatory Instructions can enhance the adaptability of vision-language models, allowing them to perform unseen tasks without task-specific fine-tuning, although the paper notes that the model’s task-level zero-shot ability is still limited and unstable. |
| On the Compositional Generalization of Multimodal LLMs for Medical Imaging (Read more on arXiv or HuggingFace) | Yonglin Deng, Weihong Wang, Rongsheng Wang, Junying Chen, Zhenyang Cai | This paper investigates compositional generalization (CG) in multimodal large language models (MLLMs) for medical imaging analysis. The main research question is whether MLLMs can leverage CG to understand unseen medical images by recombining learned elements (Modality, Anatomical area, and Task). The key methodology involved constructing a dataset called Med-MAT from 106 medical datasets, defining image elements by MAT-Triplet, and conducting experiments to assess model performance on unseen combinations. A primary result is that MLLMs trained on combinations sharing the same MAT-Triplet demonstrated successful generalization, with the model achieving 91% accuracy on the X-ray, Brain dataset when trained on combinations like CT, Brain(State) and X-ray, Bones. The principal implication for AI practitioners is that CG can be used by MLLMs for medical imaging analysis, which is a way to understand unseen medical images and improve generalization in multi-task training scenarios involving medical image data. |
| Efficiently Serving LLM Reasoning Programs with Certaindex (Read more on arXiv or HuggingFace) | Zhongdongming Dai, Zheyu Fu, Siqi Zhu, Junda Chen, Yichao Fu | Dynasor is a system designed to optimize inference-time compute for large language model (LLM) reasoning queries. The main research question is how to effectively schedule and allocate inference compute for LLM reasoning programs that generate multiple outputs for a single query. The key methodology is using “certaindex,” a proxy for statistical reasoning progress based on model certainty, to dynamically guide compute allocation and co-adapt scheduling with reasoning progress. Dynasor reduces compute by up to 50% in batch processing and sustains 3.3 times higher query rates or 4.7 times tighter latency SLOs in online serving compared to existing systems. The principal implication for AI practitioners is that using certaindex to dynamically allocate resources for LLM reasoning tasks can significantly improve efficiency and meet latency targets without sacrificing accuracy. |
| TangoFlux: Super Fast and Faithful Text to Audio Generation with Flow Matching and Clap-Ranked Preference Optimization (Read more on arXiv or HuggingFace) | Rafael Valle, Ambuj Mehrish, Zhifeng Kong, Navonil Majumder, Chia-Yu Hung | TangoFlux is a text-to-audio model that uses flow matching and CLAP-Ranked Preference Optimization for fast and high-quality audio generation. The main research objective is to develop an efficient text-to-audio (TTA) model that addresses the challenges of controllability and preference alignment in audio generation. The key methodology involves a rectified flow-based model trained with CLAP-Ranked Preference Optimization (CRPO), a novel framework that iteratively generates and optimizes preference pairs using a CLAP model as a proxy reward model. Primary results show that TangoFlux achieves a CLAP score of 0.480 and an FD score of 75.1 in 3.7 seconds using 50 steps, outperforming other models in objective evaluations and aligning well with human preferences. The principal implication for AI practitioners is that TangoFlux provides a highly efficient and effective solution for generating high-quality, text-aligned audio, making it a valuable tool for practical applications where inference speed and audio quality are critical. |
| Edicho: Consistent Image Editing in the Wild (Read more on arXiv or HuggingFace) | Ceyuan Yang, Qiuyu Wang, Yinghao Xu, Hao Ouyang, Qingyan Bai | Edicho is a training-free method for consistent image editing across multiple in-the-wild images. The main research objective is to achieve consistent edits across diverse images without requiring paired training data or optimization. The key methodology involves using explicit image correspondence to guide the self-attention mechanism and classifier-free guidance during the denoising process of diffusion models. Primary results demonstrate that Edicho achieves a text alignment score of 0.3228 and an editing consistency score of 0.9355 in global editing tasks, outperforming other methods. For AI practitioners, Edicho offers a plug-and-play solution for consistent image editing that can be integrated with existing diffusion-based editing models, enabling applications like generating coherent visual narratives and maintaining characteristics in marketing materials. |
| Bringing Objects to Life: 4D generation from 3D objects (Read more on arXiv or HuggingFace) | Gal Chechik, Dvir Samuel, Ori Malca, Ohad Rahamim | 3to4D generates 4D content from static 3D objects and text prompts. The main research question is how to generate 4D content (dynamic 3D objects) from user-provided 3D assets and text prompts while maintaining the object’s identity. The key methodology involves first converting a 3D mesh into a static 4D Neural Radiance Field (NeRF), then animating it using an Image-to-Video diffusion model guided by text, employing incremental viewpoint selection and masked Score Distillation Sampling (SDS) loss for improved motion realism. The primary results show that 3to4D outperforms baseline methods, achieving a threefold improvement in identity preservation measured using LPIPS scores (15.0 ±0.1 for 3to4D vs 44.3 ± 0.2 for the next best method). The principal implication for AI practitioners is that 3to4D provides a more effective method for generating customized 4D content from existing 3D models compared to adapting existing text-to-4D or image-to-4D methods. |
| HumanEval Pro and MBPP Pro: Evaluating Large Language Models on Self-invoking Code Generation (Read more on arXiv or HuggingFace) | Xiao-Ping Zhang, Arman Cohan, Yilun Zhao, Zhaojian Yu | The paper introduces HumanEval Pro and MBPP Pro, benchmarks for evaluating large language models (LLMs) on self-invoking code generation tasks. The main research objective is to assess LLMs’ ability to solve a base problem and then utilize that solution to address a more complex, related problem. The key methodology involves generating new, more challenging versions of existing benchmarks (HumanEval and MBPP) using Deepseek-V2.5, then manually reviewing and refining them. The primary result is that most LLMs experience a significant performance drop on self-invoking tasks compared to traditional code generation; for instance, the o1-mini model achieves 96.2% pass@1 on HumanEval but only 76.2% on HumanEval Pro. The principal implication for AI practitioners is that current LLMs, while proficient in isolated code generation, struggle with tasks requiring progressive reasoning and self-invoking code, highlighting a need for further research in this area. |
| Facilitating large language model Russian adaptation with Learned Embedding Propagation (Read more on arXiv or HuggingFace) | Daniil Chernyshev, RefalMachine | This paper introduces Learned Embedding Propagation (LEP) as a cost-effective method for adapting large language models (LLMs) to new languages, specifically Russian, while preserving original model knowledge. The main research objective is to address the limitations of language adaptation posed by restricted access to high-quality instruction-tuning data. The key methodology involves training new token embeddings and propagating them to an instruction-tuned LLM using linear transformations derived from parameter decomposition, bypassing the need for full instruction-tuning. The primary results show that LEP applied to LLaMa-3-8B and Mistral-7B achieves competitive performance with OpenChat 3.5, with the LEP-Extended model achieving a Micro-Avg score of 0.632 after calibration. The principal implication for AI practitioners is that LEP offers a viable alternative to traditional language-specific instruction-tuning, reducing costs associated with language adaptation while maintaining or surpassing performance benchmarks. |
| Training Software Engineering Agents and Verifiers with SWE-Gym (Read more on arXiv or HuggingFace) | Navdeep Jaitly, Graham Neubig, Xingyao Wang, alsuhr, Jiayi-Pan | SWE-Gym is a new benchmark for training software engineering agents that can solve real-world GitHub issues. The main research objective is to create an environment for training and evaluating language-model-based software engineering agents using real-world Python tasks. The key methodology involves constructing SWE-Gym, containing 2,438 Python tasks with executable runtime environments, unit tests, and natural language task specifications, and using it to train agents via policy improvement algorithms like rejection sampling, fine-tuning and inference-time scaling through verifiers. The primary result is that fine-tuned models achieved up to 19% absolute gains in resolve rate on SWE-Bench Verified and Lite test sets. The principal implication for AI practitioners is that SWE-Gym enables the development of more capable software engineering agents by providing a realistic and scalable training environment with executable feedback. |
| OneKE: A Dockerized Schema-Guided LLM Agent-based Knowledge Extraction System (Read more on arXiv or HuggingFace) | Mengshu Sun, Lin Yuan, Kangwei Liu, Xiangyuan Ru, Yujie Luo | OneKE is a dockerized system for knowledge extraction that uses LLM-based agents and a configurable knowledge base. The main research objective is to develop a comprehensive system for knowledge extraction that can handle diverse data types, complex schemas, and improve through error debugging. The key methodology involves using three agents (Schema Agent, Extraction Agent, and Reflection Agent) with a configurable knowledge base consisting of a Schema Repository and Case Repository to support schema analysis, knowledge extraction, and error handling. The primary results show that the Case Retrieval method improved performance on both CrossNER and NYT-11-HRL datasets, with F1 scores increasing from approximately 40 to over 60 on CrossNER when using the LLaMA-3-8B-Instruct model. The principal implication for AI practitioners is that OneKE provides a flexible framework for knowledge extraction tasks without requiring model fine-tuning, allowing for easier adaptation to various domains and data formats, although it’s unclear how performance compares to other fine-tuned methods. |
Papers for 2024-12-30
| Title | Authors | Summary |
|---|---|---|
| HuatuoGPT-o1, Towards Medical Complex Reasoning with LLMs (Read more on arXiv or HuggingFace) | Wanlong Liu, Xidong Wang, Ke Ji, Zhenyang Cai, Junying Chen | Here is a concise summary of the research paper “HuatuoGPT-o1, Towards Medical Complex Reasoning with LLMs”: i) The paper introduces HuatuoGPT-o1, a medical large language model (LLM) designed to enhance complex reasoning in the medical domain using verifiable medical problems and a two-stage training approach. ii) The main research objective is to develop an LLM capable of performing complex medical reasoning verifiable through objective ground-truth answers. iii) The key methodology involves a two-stage approach: (1) using a verifier to guide the search for a complex reasoning trajectory for fine-tuning LLMs, and (2) applying reinforcement learning (RL) with verifier-based rewards to enhance reasoning. iv) The primary result is that the 70B parameter version of HuatuoGPT-o1 outperformed other open-source general and medical-specific LLMs across multiple medical benchmarks, achieving an average score of 73.4. v) The principal implication for AI practitioners is that using verifiable problems and a two-stage training process (fine-tuning with complex reasoning trajectories followed by RL with verifier feedback) can significantly enhance the complex reasoning abilities of LLMs in specialized domains like medicine. |
| Orient Anything: Learning Robust Object Orientation Estimation from Rendering 3D Models (Read more on arXiv or HuggingFace) | Hengshuang Zhao, Chao Du, Tianyu Pang, Ziang Zhang, Zehan Wang | Here is a concise summary of the research paper “Orient Anything: Learning Robust Object Orientation Estimation from Rendering 3D Models”: i) Summary: This paper introduces Orient Anything, a novel model for estimating the 3D orientation of objects in single- and free-view images by learning from a dataset of rendered 3D models. ii) Main research question or objective: How can a robust and generalizable model be developed to accurately estimate object orientation in images, overcoming the scarcity of labeled training data? iii) Key methodology: A pipeline was developed to annotate the front face of 3D objects and render 2 million images from random views; the model is trained to predict 3D orientation by fitting probability distributions of three angles, incorporating strategies for synthetic-to-real transfer. iv) Primary results: Orient Anything achieves state-of-the-art accuracy in orientation estimation on both rendered and real images; specifically, it achieved 73.94% accuracy in predicting the azimuth of objects in rendered images. v) Principal implication for AI practitioners: AI practitioners can leverage Orient Anything as a foundational tool for tasks requiring accurate object orientation estimation, such as enhancing spatial reasoning in vision-language models and improving the generation of images with specific object poses. |
| Task Preference Optimization: Improving Multimodal Large Language Models with Vision Task Alignment (Read more on arXiv or HuggingFace) | Kunchang Li, Chenting Wang, Yinan He, Zhilin Li, Ziang Yan | Here is a concise summary of the research paper “Task Preference Optimization: Improving Multimodal Large Language Models with Vision Task Alignment”: i) This paper introduces Task Preference Optimization (TPO), a novel method to enhance multimodal large language models (MLLMs) by aligning them with fine-grained visual tasks. ii) The main research objective is to improve MLLMs’ fine-grained visual understanding and performance on specific visual tasks without compromising their general multimodal capabilities. iii) The key methodology is the use of differentiable task preferences derived from visual tasks, learnable task tokens, and multi-task co-training of task-specific heads with the MLLM. iv) The primary result is that TPO improves the performance of VideoChat and LLaVA on multimodal benchmarks, achieving an overall 14.6% improvement in multimodal performance compared to baseline models. v) For AI practitioners, TPO provides a scalable method to enhance MLLMs with specialized visual perception skills, enabling the development of more robust and versatile multimodal AI systems. |
| The Superposition of Diffusion Models Using the Itô Density Estimator (Read more on arXiv or HuggingFace) | Kirill Neklyudov, Alexander Tong, Avishek Joey Bose, Lazar Atanackovic, Marta Skreta | Here is a concise summary of the AI research paper: i) Summary: The paper introduces SUPERDIFF, a novel framework for combining pre-trained diffusion models during inference using a scalable Itô density estimator. ii) Main research question/objective: Can multiple pre-trained diffusion models be combined solely at inference in a theoretically sound and efficient manner? iii) Key methodology: SUPERDIFF leverages a new Itô density estimator for the log-likelihood of the diffusion SDE to enable superposition, combining models through an automated re-weighting scheme during inference. iv) Primary results: SUPERDIFF outperforms individual models on CIFAR-10, with a Feature Likelihood Divergence (FLD) of 5.33 ± 0.05 compared to 7.51 ± 0.11 for the best single model, and enables effective prompt-based image editing and de novo protein structure design. v) Principal implication for AI practitioners: AI practitioners can use SUPERDIFF to combine multiple pre-trained diffusion models without retraining, enabling efficient generation, improved performance, and novel applications like concept interpolation and protein design. |
| From Elements to Design: A Layered Approach for Automatic Graphic Design Composition (Read more on arXiv or HuggingFace) | Ji Li, Ting Liu, Danqing Huang, Shizhao Sun, Jiawei Lin | Here’s a concise summary of the research paper: i) Summary: This paper introduces LaDeCo, a novel framework for automatic graphic design composition from multimodal elements using a layered approach. ii) Main research question/objective: How to automatically compose multimodal graphic elements into a cohesive and aesthetically pleasing design. iii) Key methodology: LaDeCo employs a layer planning module using GPT-4o to categorize elements and a layered design composition process that uses fine-tuned Large Multimodal Models (LMMs) to predict element attributes layer-by-layer, incorporating rendered images of previous layers as context. iv) Primary results: LaDeCo significantly outperforms baseline models in design composition, achieving an overall LLaVA-OV score of 8.08 compared to 5.34 for FlexDM and 6.53 for GPT-4o on the design composition task. v) Principal implication for AI practitioners: AI practitioners can leverage LaDeCo’s layered approach and LMMs to build more effective and efficient automatic graphic design systems, enabling applications such as resolution adjustment, element filling, and design variation. |
| Safeguard Fine-Tuned LLMs Through Pre- and Post-Tuning Model Merging (Read more on arXiv or HuggingFace) | Shang-Tse Chen, Saurav Sahay, Shachi H Kumar, Hsuan Su, farnhua | Here is a concise summary of the research paper, strictly following your guidelines: i) This paper proposes a method to mitigate safety degradation in fine-tuned large language models (LLMs) by merging the weights of pre- and post-fine-tuned models. ii) The main research question is how to improve downstream task performance while preserving safety in LLMs without relying on additional safety data. iii) The key methodology used is a two-step approach: fine-tuning the base model on a downstream task, then merging the base model with the fine-tuned model via weight interpolation. iv) The primary result shows that merging the models significantly reduces the Attack Success Rate (ASR) across various downstream tasks; for instance, on the medical assistance task, the ASR is reduced by over 30%. v) For AI practitioners, this method offers a practical solution for adapting safety-aligned LLMs to downstream tasks while preserving their inherent safety features without requiring additional safety data. |
| SBS Figures: Pre-training Figure QA from Stage-by-Stage Synthesized Images (Read more on arXiv or HuggingFace) | Yoshitaka Ushiku, Tosho Hirasawa, Shohei Tanaka, Kuniaki Saito, Risa Shinoda | Here’s a concise summary of the research paper “SBS Figures: Pre-training Figure QA from Stage-by-Stage Synthesized Images,” strictly adhering to your guidelines: i) Summary: The paper introduces SBS Figures, a synthetic dataset for pre-training figure-based question-answering models, generated through a novel stage-by-stage pipeline. ii) Main research question/objective: The main objective is to develop a method for creating a large-scale, diverse, synthetic figure QA dataset to improve the performance of figure QA models. iii) Key methodology: A three-stage pipeline was used: (1) generate visualization target data, (2) render figures via Python code, and (3) generate QA pairs using LLMs, all progressively transforming seed data. iv) Primary results: Pre-training with SBS Figures improved the average accuracy on the ChartQA dataset by 6.42 points for the Pix2Struct model. v) Principal implication for AI practitioners: AI practitioners can use the SBS Figures dataset and pipeline to pre-train and fine-tune their models, enhancing performance on figure QA tasks without the need for manual annotation. |
| VideoMaker: Zero-shot Customized Video Generation with the Inherent Force of Video Diffusion Models (Read more on arXiv or HuggingFace) | Junfu Pu, Zhongang Qi, Xiaodong Cun, Yong Zhang, Tao Wu | Here is a concise summary of the research paper “VideoMaker: Zero-shot Customized Video Generation with the Inherent Force of Video Diffusion Models”: i) Summary: VideoMaker is a framework for zero-shot customized video generation that leverages the inherent capabilities of video diffusion models (VDMs) for subject feature extraction and injection without requiring additional modules. ii) Main research question/objective: Can VDMs be utilized to extract and inject subject features for customized video generation without the need for external modules or extensive retraining? iii) Key methodology: The method uses the VDM itself to extract fine-grained subject features from a reference image and injects these features using a modified spatial self-attention mechanism within the VDM, along with a Guidance Information Recognition Loss. iv) Primary results: VideoMaker outperformed existing methods in customized human video generation, achieving a Face Similarity score of 0.8047 compared to the next best result of 0.7323 from ID-Animator. v) Principal implication for AI practitioners: AI practitioners can achieve high-quality, zero-shot customized video generation by fine-tuning the pre-trained VDM to activate the inherent force of video diffusion model, offering a more efficient alternative to existing methods that rely on external modules. |
Papers for 2024-12-27
| Title | Authors | Summary |
|---|---|---|
| YuLan-Mini: An Open Data-efficient Language Model (Read more on arXiv or HuggingFace) | Jie Chen, Jiapeng Wang, Jia Deng, Huatong Song, Yiwen Hu | Here is a concise summary of the AI research paper “YuLan-Mini: An Open Data-efficient Language Model”: i) YuLan-Mini is a 2.42B parameter language model designed for efficient pre-training, achieving high performance with limited data. ii) The main research objective was to develop a high-performing, small-scale language model using only publicly available data with a restricted compute budget, focusing on data efficiency and training stability. iii) Key methodologies used include an elaborate data pipeline with cleaning and scheduling, a robust optimization method to mitigate training instability using scaled initialization, and an annealing approach with targeted data selection and long-context training. iv) The primary result is that YuLan-Mini, trained on 1.08T tokens, achieved a score of 64.00 on the HumanEval (zero-shot) benchmark, comparable to industry-leading models. v) For AI practitioners, YuLan-Mini demonstrates that competitive language models can be developed with limited data and computational resources by focusing on data quality, optimization methods, and efficient training strategies. |
| A Silver Bullet or a Compromise for Full Attention? A Comprehensive Study of Gist Token-based Context Compression (Read more on arXiv or HuggingFace) | Xinting Huang, Shuaiyi Li, Kelong Mao, Zhisong Zhang, ChenlongDeng | Here is a concise summary of the research paper: i) Summary: This paper investigates gist token-based context compression methods for improving long-context processing in large language models (LLMs). ii) Main research question/objective: To what extent can gist-based architectures replace full attention models, and what failure patterns arise from compression? iii) Key methodology: The authors propose a unified framework to categorize gist-based models and conduct experiments on language modeling, weak context-dependent, and long-context tasks using Llama3-8B and Qwen2-7B models. iv) Primary results: Fine-grained KV cache architecture achieves near-lossless performance on many tasks, but struggles with tasks like synthetic recall; at a compression ratio of 4, Fine-KV achieves 40.6% accuracy on synthetic recall compared to full attention’s 93.9%. v) Principal implication for AI practitioners: While gist token-based compression can effectively reduce computational costs for many tasks, practitioners should be aware of its limitations in tasks requiring precise token-level recall and explore the proposed mitigation strategies (fine-grained autoencoding and segment-wise token importance estimation) to enhance performance. |
Papers for 2024-12-26
| Title | Authors | Summary |
|---|---|---|
| Token-Budget-Aware LLM Reasoning (Read more on arXiv or HuggingFace) | Zhenyu Chen, Shiqing Ma, Shiyu Zhao, Chunrong Fang, Tingxu Han | Here is a concise summary of the paper “Token-Budget-Aware LLM Reasoning”: i) Summary: This paper introduces TALE, a framework to reduce token redundancy in large language model (LLM) reasoning by dynamically estimating and incorporating token budgets into prompts. ii) Main research question or objective: How to effectively reduce token costs in Chain-of-Thought (CoT) reasoning while preserving LLM performance. iii) Key methodology: TALE estimates a token budget based on reasoning complexity and uses it to guide the LLM’s reasoning process via a token-budget-aware prompt. iv) Primary results: TALE reduces token usage by 68.64% on average compared to vanilla CoT, with less than a 5% decrease in accuracy. v) Principal implication for AI practitioners: AI practitioners can use TALE to optimize token efficiency in LLM reasoning tasks, significantly reducing computational costs and resource usage while maintaining performance. |
Papers for 2024-12-25
| Title | Authors | Summary |
|---|---|---|
| DepthLab: From Partial to Complete (Read more on arXiv or HuggingFace) | Hao Ouyang, Shuzhe Wang, Qiuyu Wang, Ka Leong Cheng, Zhiheng Liu | Here’s a summary of the research paper “DepthLab: From Partial to Complete” following your guidelines: i) Summary: DepthLab is a foundation model for RGB image-conditioned depth inpainting that leverages image diffusion priors to complete missing or occluded depth information. ii) Main research question or objective: To develop a robust and generalizable model for depth inpainting that preserves scale consistency and demonstrates resilience to depth-deficient regions. iii) Key methodology: A dual-branch depth inpainting diffusion framework is used, processing a reference image through a Reference U-Net for RGB feature extraction and integrating these features into an Estimation U-Net that handles depth and mask inputs. iv) Primary results: DepthLab achieved an AbsRel of 2.3 on the ScanNet dataset, outperforming other methods in numerical performance and visual quality across various downstream tasks. v) Principal implication for AI practitioners: AI practitioners can leverage DepthLab as a foundation model for various depth-related tasks, including 3D scene inpainting, text-to-3D scene generation, sparse-view reconstruction, and LiDAR depth completion, without the need for extensive task-specific training. |
| 3DGraphLLM: Combining Semantic Graphs and Large Language Models for 3D Scene Understanding (Read more on arXiv or HuggingFace) | Dmitry Yudin, wingrune | Here’s a summary of the AI research paper following your strict guidelines: i) 3DGraphLLM combines semantic graphs and large language models for improved 3D scene understanding in vision-language tasks. ii) The research objective was to develop a method for constructing a learnable representation of a 3D scene graph to improve the accuracy of LLMs in performing 3D vision-language tasks. The paper specifically focuses on solving 3D referred object grounding, 3D dense scene captioning, and 3D visual question answering. iii) The key methodology involved creating a learnable representation of a 3D scene graph using object embeddings and their semantic relationships, encoded as triplets, which were fed as input to a pre-trained LLM. The model uses VL-SAT for semantic relationship extraction and k-nearest neighbor selection to create the flat sequence of graph tokens. iv) 3DGraphLLM achieved a 5.8% improvement in F1@0.5 on the Multi3DRefer benchmark for 3D referred object grounding compared to a baseline. (Other quantitative results are presented, but this is one specific example) v) The significant finding, a substantial performance improvement on visual grounding with the integration of semantic relationships, directly implies that incorporating semantic graph structures into LLM inputs can substantially enhance 3D vision-language task performance. This suggests a valuable approach for AI practitioners developing embodied AI agents or systems requiring robust 3D scene understanding. |
| Fourier Position Embedding: Enhancing Attention’s Periodic Extension for Length Generalization (Read more on arXiv or HuggingFace) | Ning Ding, Kaiyan Zhang, Xingtai Lv, Che Jiang, Ermo Hua | Here is a concise summary of the research paper “Fourier Position Embedding: Enhancing Attention’s Periodic Extension for Length Generalization”: i) Summary: This paper introduces Fourier Position Embedding (FoPE) to improve the length generalization of language models (LMs) by enhancing the frequency-domain properties of attention in Rotary Position Embedding (RoPE). ii) Main research question/objective: How to address the limitations of RoPE that hinder length generalization in language models. iii) Key methodology used: The authors use Discrete Signal Processing theory to analyze RoPE, identifying spectral damage as a key issue, and propose FoPE, which constructs Fourier Series and zero-outs destructive frequency components. iv) Primary results: FoPE maintains a more stable perplexity and achieves better accuracy in a needle-in-haystack task compared to RoPE and ALiBi; for example, FoPE achieved an accuracy of 100% on the Passkey Retrieval task with a sequence length of 512, while RoPE’s accuracy dropped to nearly 0% at sequence length of 2048. v) Principal implication for AI practitioners: FoPE offers a method to enhance the length generalization of LMs without significant computational overhead, making it a valuable technique for AI/ML engineers and data scientists working with transformer-based models. |
| DiTCtrl: Exploring Attention Control in Multi-Modal Diffusion Transformer for Tuning-Free Multi-Prompt Longer Video Generation (Read more on arXiv or HuggingFace) | Zhaoyang Zhang, Wenze Liu, Xiaoyu Li, Xiaodong Cun, Minghong Cai | Here’s a summary of the AI research paper following your strict guidelines: i) DiTCtrl is a tuning-free method for generating coherent multi-prompt longer videos using a pre-trained Multi-Modal Diffusion Transformer (MM-DiT). ii) The research objective was to develop a training-free method for multi-prompt video generation capable of producing long videos with smooth transitions and accurate prompt following, overcoming limitations of existing single-prompt methods. iii) The key methodology involved analyzing the MM-DiT’s attention mechanism, designing a KV-sharing mechanism and a latent blending strategy to achieve smooth transitions between video segments generated from sequential prompts. iv) DiTCtrl achieved state-of-the-art performance on the MPVBench benchmark, a new benchmark specifically designed for multi-prompt video generation. A specific quantitative result was not clearly presented, though the paper mentions state-of-the-art performance on CSCV metric. v) The most impactful finding is the development of a training-free method for multi-prompt video generation; this is highly relevant to AI practitioners as it allows leveraging existing pre-trained MM-DiT models for complex video generation tasks without requiring extensive retraining, reducing computational costs and data requirements. |
| In Case You Missed It: ARC ‘Challenge’ Is Not That Challenging (Read more on arXiv or HuggingFace) | Borchmann | Here’s a summary of the AI research paper following the provided guidelines: i) 1-line summary: The paper challenges the established evaluation methodology for several multiple-choice question benchmarks, demonstrating that a seemingly simple change in setup dramatically impacts model performance and potentially misrepresents model capabilities. ii) Main research question or objective: To investigate the impact of different evaluation setups (separate vs. simultaneous presentation of answer choices) on the performance of large language models (LLMs) across multiple-choice question benchmarks. iii) Key methodology used: The authors compared LLM performance on established benchmarks (ARC, OpenBookQA, SIQA) using two evaluation setups: one presenting answer choices separately, and another presenting them simultaneously. They then compared the reported accuracy scores from the literature to their own replications under each setup. The paper does not explicitly detail all aspects of the model training or testing procedures used in its replications. iv) Primary results (include one specific quantitative finding): Switching from presenting ARC Challenge answer choices separately to presenting them all at once increased Llama 3.1 70B accuracy from 64% to 93%. v) Principal implication for AI practitioners: The evaluation setup significantly influences performance metrics and model rankings on multiple-choice question benchmarks. AI practitioners should carefully consider and evaluate the impact of evaluation setup, potentially reconsidering the established methods for existing benchmarks and future design. |
| PartGen: Part-level 3D Generation and Reconstruction with Multi-View Diffusion Models (Read more on arXiv or HuggingFace) | Jianyuan Wang, Tom Monnier, Iro Laina, Roman Shapovalov, Minghao Chen | Here is a concise summary of the research paper “PartGen: Part-level 3D Generation and Reconstruction with Multi-View Diffusion Models”: i) Summary: PartGen is a novel method that generates or reconstructs 3D objects as compositions of meaningful parts, starting from text, images, or unstructured 3D objects. ii) Main research question/objective: How can we automatically segment a 3D object into its meaningful parts and reconstruct these parts in high quality, even when they are partially or fully occluded? iii) Key methodology: PartGen uses a two-stage approach employing multi-view diffusion models, first segmenting objects into parts by generating consistent 2D segmentation maps across multiple views, and then completing and reconstructing each part in 3D while considering the context of the entire object. iv) Primary results: PartGen outperforms segmentation baselines on a dataset of artist-created 3D assets, achieving a 59.3% mAP50 score for automatic segmentation with 10 samples, compared to 37.4% for a fine-tuned SAM2 model. v) Principal implication for AI practitioners: PartGen provides a method for generating structured 3D assets composed of complete, semantically meaningful parts, which is crucial for downstream applications like 3D editing, animation, and robotic manipulation that currently requires significant manual effort. |
| ReMoE: Fully Differentiable Mixture-of-Experts with ReLU Routing (Read more on arXiv or HuggingFace) | Jun Zhu, Jianfei Chen, Ziteng Wang | Here is a summary of the AI research paper “ReMoE: Fully Differentiable Mixture-of-Experts with ReLU Routing” following your strict guidelines: i) One-line summary: This paper introduces ReMoE, a fully differentiable Mixture-of-Experts (MoE) model using ReLU routing to improve performance and scalability compared to traditional TopK routing. ii) Main research question/objective: How can the non-differentiable nature of TopK routing in MoE models be addressed to improve performance and scalability? iii) Key methodology: The authors propose ReMoE, replacing the TopK+Softmax routing mechanism with a ReLU-based router and introduce an adaptive L1 regularization for controlling sparsity and load balancing. iv) Primary results: ReMoE consistently outperforms TopK-routed MoE across various model sizes, expert counts, and levels of granularity; for example, on downstream tasks, ReMoE achieved a 40.03% average zero-shot accuracy compared to MoE’s 38.20% on a specific configuration. v) Principal implication for AI practitioners: ReMoE offers a drop-in replacement for TopK routing in MoE models, enabling fully differentiable training and improved scalability, leading to potentially more efficient and performant large language models. The paper lacks clear details on the computational cost differences between ReMoE and standard MoE during training. |
| SKETCH: Structured Knowledge Enhanced Text Comprehension for Holistic Retrieval (Read more on arXiv or HuggingFace) | Divya Chaudhary, Vinija Jain, Aman Chadha, Vinesh Kumar Gande, Aakash Mahalingam | Here’s a summary of the AI research paper following your strict guidelines: i) SKETCH enhances Retrieval-Augmented Generation (RAG) systems by integrating semantic text retrieval with knowledge graphs for improved text comprehension. ii) The research objective was to improve the efficiency and accuracy of RAG systems in processing large datasets while maintaining a comprehensive understanding of the context. iii) The key methodology involved a novel approach called SKETCH, which integrates semantic text chunking with knowledge graphs to merge structured and unstructured data for holistic comprehension. iv) SKETCH consistently outperformed baseline approaches on multiple datasets; notably, on the Italian Cuisine dataset, it achieved an answer relevancy of 0.94 and a context precision of 0.99. v) The significantly high answer relevancy and context precision (0.94 and 0.99 respectively) on the Italian Cuisine dataset demonstrates SKETCH’s potential to improve the accuracy and contextual relevance of RAG systems, particularly beneficial for applications requiring precise and contextually rich information retrieval. The paper does not explicitly detail the implications for specific engineering or application tasks beyond this general finding. |
Papers for 2024-12-24
| Title | Authors | Summary |
|---|---|---|
| B-STaR: Monitoring and Balancing Exploration and Exploitation in Self-Taught Reasoners (Read more on arXiv or HuggingFace) | Zifei Shan, Yijun Wang, Lulu Zhao, Yuzhen Huang, Weihao Zeng | Here is a concise summary of the research paper “B-STAR: MONITORING AND BALANCING EXPLORATION AND EXPLOITATION IN SELF-TAUGHT REASONERS” based on your guidelines: i) This paper introduces B-STAR, a self-improvement framework for enhancing AI reasoning by dynamically balancing exploration and exploitation during iterative training. ii) The main research question is how to monitor and balance the model’s ability to generate diverse, high-quality responses (exploration) and the effectiveness of external rewards in selecting the best responses (exploitation) during self-improvement. iii) The key methodology involves tracking exploration and exploitation metrics (e.g., Pass@K, Reward@K-S) and automatically adjusting configurations like sampling temperature and reward threshold to maximize a “balance score” that quantifies the interplay between these factors. iv) B-STAR achieved a Pass@1 score of 27.8 on the MATH dataset, outperforming the online RFT baseline, which achieved 23.2 in the same setting. v) For AI practitioners, B-STAR demonstrates that dynamically balancing exploration and exploitation during self-improvement is crucial for maximizing performance gains, particularly in complex reasoning tasks. |
| RobustFT: Robust Supervised Fine-tuning for Large Language Models under Noisy Response (Read more on arXiv or HuggingFace) | Zhiping Xiao, Jingyang Yuan, Xiao Luo, Junyu Luo, kaize0409 | Here’s a concise summary of the research paper “ROBUSTFT: Robust Supervised Fine-tuning for Large Language Models under Noisy Response” following the specified guidelines: i) ROBUSTFT is a framework designed to improve the robustness of supervised fine-tuning for large language models (LLMs) when training data contains noisy responses. ii) Can LLMs detect inevitable noise and enhance data quality to improve their performance on target tasks? iii) The methodology involves a multi-expert collaborative system for noise detection, context-enhanced reasoning for data relabeling, and response entropy-based data selection. iv) ROBUSTFT demonstrated that with 30% noise in the training data, model performance deteriorates by 8.9% compared to the vanilla LLM baseline on the MMLU dataset. v) For AI practitioners, ROBUSTFT provides a method to enhance the performance of fine-tuned LLMs in practical applications where noisy data is unavoidable, emphasizing the need for noise detection and denoising mechanisms. |
| Diving into Self-Evolving Training for Multimodal Reasoning (Read more on arXiv or HuggingFace) | Yu Cheng, Fan Zhou, Xiwen Zhang, Junlong Li, Wei Liu | Here is a concise summary of the research paper “Diving into Self-Evolving Training for Multimodal Reasoning”: i) Summary: This paper investigates self-evolving training methods to enhance the multimodal reasoning capabilities of Large Multimodal Models (LMMs) without relying on human-annotated data. ii) Main Research Question/Objective: How can different factors in self-evolving training, such as training method, reward model, and prompt variation, be optimized to improve multimodal reasoning in LMMs? iii) Key Methodology: The authors conduct controlled experiments, varying factors like training method (iterative, continuous), reward model (binary, process-based), and prompt variation (labeled, unlabeled), while monitoring the dynamics of the self-evolution process. iv) Primary Results: Continuous self-evolving training with a process-based reward model (PRM) and a moderate number of selected responses (Top-2) achieves the best performance; specifically, on the MathVista benchmark, the M-STAR model achieved a 59.5% accuracy. v) Principal Implication for AI Practitioners: AI practitioners can leverage the proposed M-STAR framework, which incorporates optimized design choices and dynamic temperature adjustments, to enhance the multimodal reasoning capabilities of LMMs without additional human annotations. The paper does not clearly indicate how the framework can be integrated into existing LLM development or training pipelines. |
| Distilled Decoding 1: One-step Sampling of Image Auto-regressive Models with Flow Matching (Read more on arXiv or HuggingFace) | Yu Wang, Xuefei Ning, Enshu Liu, fjxmlzn | Here is a concise summary of the research paper “Distilled Decoding 1: One-Step Sampling of Image Auto-regressive Models with Flow Matching”: i) The paper introduces Distilled Decoding (DD), a novel method to accelerate image generation from pre-trained autoregressive (AR) models by enabling one- or few-step sampling. ii) The main research question is whether a pre-trained AR model can be adapted to generate outputs in just one or two steps. iii) The key methodology is leveraging flow matching to create a deterministic mapping from a Gaussian distribution to the output distribution of a pre-trained AR model, then training a network to distill this mapping for few-step generation. iv) Primary results show that for the LlamaGen model, DD reduces generation from 256 steps to 1, achieving a 217.8x speed-up with a comparable FID increase from 4.11 to 11.35 on ImageNet-256. v) The principal implication for AI practitioners is that DD offers a way to significantly speed up inference for image AR models, challenging the notion that they are inherently slow. |
| Large Motion Video Autoencoding with Cross-modal Video VAE (Read more on arXiv or HuggingFace) | Jiaxin Xie, Jingye Chen, Yingqing He, Yang Fei, Yazhou Xing | Here is a concise summary of the research paper “Large Motion Video Autoencoding with Cross-modal Video VAE”: i) This paper introduces a novel cross-modal Video Variational Autoencoder (VAE) designed for high-fidelity video encoding and reconstruction, particularly for videos with large motions. ii) The main research objective is to develop a robust Video VAE that effectively compresses both spatial and temporal dimensions of videos while preserving detail and motion information, and explore the benefits of integrating text guidance. iii) The key methodology involves a two-stage spatiotemporal modeling approach combining temporal-aware spatial compression with a lightweight motion compression model, enhanced by cross-modal learning using text descriptions and joint image-video training. iv) The proposed Video VAE achieves a PSNR of 34.5022 on the WebVid test set, outperforming existing state-of-the-art methods. v) For AI practitioners, this Video VAE offers an effective solution for video compression and reconstruction, directly applicable to improving the performance of Latent Video Diffusion Models by providing a more robust and high-quality latent space representation. |
| Deliberation in Latent Space via Differentiable Cache Augmentation (Read more on arXiv or HuggingFace) | Arthur Szlam, Jun Xie, Jiaxing Wu, Jonas Pfeiffer, Luyang Liu | Here’s a summary of the paper “Deliberation in Latent Space via Differentiable Cache Augmentation” following your guidelines: i) Summary: This paper introduces a method to augment frozen language models with a trainable “coprocessor” that enhances the model’s key-value cache with learned latent embeddings, improving reasoning and prediction capabilities. ii) Main research question or objective: How can a frozen language model be augmented to improve its ability to generate text and perform reasoning tasks without modifying its parameters? iii) Key methodology: A coprocessor is trained to augment the key-value cache of a frozen language model with latent embeddings. This is achieved by predicting future tokens based on the augmented cache, using a modified training framework that allows for multi-position augmentation and ahead-token prediction in a single forward pass. iv) Primary results: Cache augmentation consistently reduces perplexity and improves performance on reasoning tasks. For example, the augmented Gemma-2 2B model with 64 latent embeddings achieved a 10.05% improvement on the GSM8K benchmark compared to the baseline. v) Principal implication for AI practitioners: AI practitioners can enhance the performance of frozen language models on downstream tasks by training a coprocessor to augment the model’s cache, offering a computationally efficient alternative to full model fine-tuning or retraining. |
| Revisiting In-Context Learning with Long Context Language Models (Read more on arXiv or HuggingFace) | Oh, Geunseob, Prakhar Gupta, Sun Jae Lee, Jinheon Baek | Here is a concise summary of the research paper, following the specified guidelines: i) This paper investigates the effectiveness of various sample selection strategies for in-context learning (ICL) with long context language models (LCLMs). ii) The main research question is whether previous sample selection strategies for ICL generalize to the many-shot ICL regime enabled by LCLMs. iii) The key methodology involves extensive experiments on 18 datasets across four tasks (classification, translation, summarization, and reasoning) using three types of sample selection methods (relevance, diversity, and difficulty-based). iv) The primary result is that sophisticated example selection techniques do not yield significant improvements over random sample selection in many-shot ICL with LCLMs, with statistical significance in fewer than 15% of instances. v) For AI practitioners, the principal implication is that random sampling is similarly effective compared to complex sample selection strategies in many-shot ICL scenarios with LCLMs, offering computational efficiency through key-value caching. |
| Outcome-Refining Process Supervision for Code Generation (Read more on arXiv or HuggingFace) | Jindong Wang, Zhengran Zeng, Yidong Wang, Weizheng Gu, Zhuohao Yu | Here’s a concise summary of the research paper “Outcome-Refining Process Supervision for Code Generation”: i) Summary: The paper introduces Outcome-Refining Process Supervision (ORPS), a new method for code generation that treats the refinement of outcomes as the process to be supervised, using a tree-structured search and execution feedback. ii) Main research question/objective: How to improve the performance of large language models (LLMs) in complex code generation tasks that require deep algorithmic reasoning. iii) Key methodology: ORPS leverages a tree-structured exploration space with beam search to maintain multiple solution trajectories, grounding supervision in concrete execution signals rather than solely relying on human-annotated data or reward model judgments. iv) Primary results: ORPS achieves an average Pass@1 improvement of 26.9% across three datasets and five models, demonstrating significant gains in code generation accuracy and performance. v) Principal implication for AI practitioners: AI practitioners can use ORPS to enhance LLMs’ code generation capabilities, particularly for complex tasks, by providing a more structured and verifiable approach to guide the models’ reasoning and solution refinement process without the need for extensive training data. |
| DRT-o1: Optimized Deep Reasoning Translation via Long Chain-of-Thought (Read more on arXiv or HuggingFace) | Jie Zhou, Yunlong Liang, Fandong Meng, Jiaan Wang | Here is a concise summary of the AI research paper “DRT-01: Optimized Deep Reasoning Translation via Long Chain-of-Thought” based on your specifications: i) Summary: This paper introduces DRT-01, a novel system designed to enhance neural machine translation (MT) by incorporating a long chain-of-thought (CoT) approach, specifically for translating literature containing similes and metaphors. ii) Main Research Question/Objective: How to improve the performance of neural machine translation for literary text involving similes and metaphors by simulating the long chain-of-thought process used by human translators. iii) Key Methodology: A multi-agent framework was developed, involving a translator, an advisor, and an evaluator, to iteratively translate sentences via long thought. This framework synthesizes MT data with long thought processes, which is then refined using GPT-40 and used to train the DRT-01 models. iv) Primary Results: DRT-01-7B outperformed Qwen2.5-7B-Instruct by 8.26 BLEU points on literature translation tasks. v) Principal Implication for AI Practitioners: AI practitioners can leverage the multi-agent framework and long-thought training data developed in this study to enhance the ability of large language models to perform nuanced machine translation, especially for complex literary texts. |
| Agent-SafetyBench: Evaluating the Safety of LLM Agents (Read more on arXiv or HuggingFace) | Junxiao Yang, Jingzhuo Zhou, Yida Lu, Shiyao Cui, Zhexin Zhang | Here is a concise summary of the research paper “AGENT-SAFETYBENCH: Evaluating the Safety of LLM Agents”: i) Summary: This paper introduces AGENT-SAFETYBENCH, a new benchmark for evaluating the safety of large language model (LLM) agents in interactive environments. ii) Main research question or objective: The main objective is to develop a comprehensive benchmark to evaluate the safety of LLM agents across various risk categories and failure modes. iii) Key methodology used: The methodology involves constructing 349 interaction environments and 2,000 test cases, and evaluating 16 LLM agents using a fine-tuned scoring model. iv) Primary results: None of the 16 tested LLM agents achieved a safety score above 60% on the AGENT-SAFETYBENCH benchmark. v) Principal implication for AI practitioners: AI practitioners should focus on improving the robustness and risk awareness of LLM agents, as current defense prompts alone are insufficient to address safety issues. |
| NILE: Internal Consistency Alignment in Large Language Models (Read more on arXiv or HuggingFace) | Hongru Wang, Bowei He, Yufei Wang, Qiyuan Zhang, Minda Hu | Here’s a summary of the paper “NILE: Internal Consistency Alignment in Large Language Models” following your guidelines: i) The paper introduces NILE, a framework designed to improve the alignment of Instruction Fine-Tuning (IFT) datasets with Large Language Models’ (LLMs) internal knowledge to enhance performance. ii) Main research question/objective: How can IFT datasets be optimized to enhance consistency with an LLM’s internal knowledge, thereby improving its performance? iii) Key methodology used: NILE uses a three-step process: Internal Knowledge Extraction (IKE), Knowledge-Aware Sample Revision (KSR), and Internal Consistency Filtering (ICF). iv) Primary results: NILE-aligned IFT datasets significantly boost LLM performance across various benchmarks, achieving up to a 66.6% gain on the Arena-Hard dataset. v) Principal implication for AI practitioners: AI practitioners should consider the internal consistency between IFT datasets and LLMs’ pre-trained knowledge to maximize model performance, suggesting a need for methods like NILE in dataset optimization. |
| LearnLM: Improving Gemini for Learning (Read more on arXiv or HuggingFace) | Andrea Huber, Aliya Rysbek, Aditya Srikanth Veerubhotla, Abhinit Modi, LearnLM Team | Here is a concise summary of the research paper “LearnLM: Improving Gemini for Learning” based on your specified format: i) Summary: This paper details the development of LearnLM, a model based on Gemini 1.5 Pro, optimized for educational applications via pedagogical instruction following. ii) Main research question or objective: How can large language models be trained to follow pedagogical system instructions to improve their performance in learning scenarios? iii) Key methodology used: The researchers used supervised fine-tuning (SFT) and Reinforcement Learning from Human Feedback (RLHF) to train LearnLM, with a novel scenario-based human evaluation pipeline to assess pedagogical capabilities. iv) Primary results: Expert raters preferred LearnLM over other models, with an average preference strength of 31% over GPT-4o. v) Principal implication for AI practitioners: AI practitioners can leverage pedagogical instruction following and scenario-based evaluations to develop more effective AI systems for educational use cases, enabling personalized learning at scale. |
| OpenAI o1 System Card (Read more on arXiv or HuggingFace) | Adam Richardson, Adam Lerer, Adam Kalai, Aaron Jaech, OpenAI | Here’s a concise summary of the OpenAI o1 System Card, strictly following your guidelines: i) Summary: OpenAI introduces the o1 model series, trained with large-scale reinforcement learning to reason using the chain of thought, enhancing safety and robustness through deliberate alignment. ii) Main research question or objective: The main objective was to evaluate the safety and robustness of the o1 model series, focusing on its advanced reasoning capabilities and performance on safety benchmarks. iii) Key methodology used: The methodology involved large-scale reinforcement learning with chain-of-thought reasoning, safety evaluations, external red teaming, and Preparedness Framework evaluations, utilizing diverse datasets including publicly available data, proprietary data, and custom datasets. iv) Primary results: The o1 model demonstrated state-of-the-art performance on safety benchmarks, such as achieving 92% accuracy on the challenging refusal evaluation compared to 71.3% for GPT-4o. v) Principal implication for AI practitioners: AI practitioners should prioritize building robust alignment methods and conducting extensive stress-testing, as o1’s enhanced reasoning capabilities improve safety but also highlight the need for meticulous risk management protocols. |
| OpenRFT: Adapting Reasoning Foundation Model for Domain-specific Tasks with Reinforcement Fine-Tuning (Read more on arXiv or HuggingFace) | Jinlin Xiao, Yuhang Wang, Jiangming Shu, Yuqi Yang, Yuxiang Zhang | Here is a concise summary of the AI research paper “OpenRFT: Adapting Reasoning Foundation Model for Domain-specific Tasks with Reinforcement Fine-Tuning” based on your guidelines: i) OpenRFT is a framework for fine-tuning generalist reasoning models for domain-specific tasks using reinforcement learning. ii) The main research objective is to adapt generalist reasoning foundation models to domain-specific tasks when reasoning step data and sufficient training samples are lacking. iii) The key methodology involves data augmentation, supervised fine-tuning with synthesized reasoning processes, and reinforcement learning with a process reward model and few-shot in-context learning. iv) The primary result is that OpenRFT achieved an average performance increase of 11% on the SciKnowEval benchmark using only 100 domain-specific samples per task. v) The principal implication for AI practitioners is that OpenRFT offers a method to create specialized reasoning models from generalist foundation models efficiently, even with limited domain-specific data, although the paper notes that alignment between the teacher and student policy models is important and the absence of a strong open-source generalist reasoning model limits the full potential of RFT. |
| Friends-MMC: A Dataset for Multi-modal Multi-party Conversation Understanding (Read more on arXiv or HuggingFace) | Qun Liu, Jianxin Liang, Xiaojun Meng, Yueqian Wang, ColorfulAI | Here is a concise summary of the research paper “Friends-MMC: A Dataset for Multi-modal Multi-party Conversation Understanding”: i) This paper introduces Friends-MMC, a new dataset for multi-modal multi-party conversation (MMC) understanding, derived from the TV series “Friends,” and studies conversation speaker identification and response prediction tasks. ii) The main research objective is to develop a dataset and baseline methods for understanding multi-modal multi-party conversations, focusing on speaker identification and response prediction in a more complex and realistic setting than existing datasets. iii) The key methodology involves collecting and annotating video clips, utterances, speaker identities, and facial bounding boxes from the TV show “Friends,” and developing a baseline model that combines visual and textual information using an optimization solver. iv) The primary results show that the proposed baseline method for conversation speaker identification achieves 83.21% accuracy on the test set when using both video and text modalities. v) For AI practitioners, the principal implication is that modeling speaker information is crucial for multi-modal multi-party conversation understanding, and the Friends-MMC dataset provides a valuable resource for developing and evaluating models in this domain. |
| PC Agent: While You Sleep, AI Works – A Cognitive Journey into Digital World (Read more on arXiv or HuggingFace) | Runze Fan, Jiadi Su, Shijie Xia, Jiahe Jin, Yanheng He | Here is a concise summary of the AI research paper “PC Agent: While You Sleep, AI Works – A Cognitive Journey into Digital World”: i) Summary: This paper introduces PC Agent, a novel AI system designed to autonomously perform complex computer work by learning from human cognitive processes. ii) Main research question/objective: The main objective is to develop an AI agent capable of efficiently handling complex digital work by transferring human cognitive processes during computer use. iii) Key methodology: The authors introduce a three-part framework: PC Tracker for collecting human-computer interaction data, a cognition completion pipeline to transform raw data into cognitive trajectories, and a multi-agent system for action planning and visual grounding. iv) Primary results: PC Agent, trained on 133 cognitive trajectories, can execute complex tasks with up to 50 steps in PowerPoint presentation creation. v) Principal implication for AI practitioners: AI practitioners can leverage the open-sourced PC Agent framework to develop digital agents that learn from human cognitive data, potentially automating a wide range of complex computer-based tasks. |
Papers for 2024-12-23
| Title | Authors | Summary |
|---|---|---|
| Parallelized Autoregressive Visual Generation (Read more on arXiv or HuggingFace) | jshfeng, zhenheny, Ikuinen, ShuhuaiRen, Epiphqny | Here is a concise summary of the research paper “Parallelized Autoregressive Visual Generation”: i) Summary: This paper introduces a novel approach for parallelized autoregressive visual generation that improves efficiency while maintaining the quality of generated images and videos. ii) Main research question or objective: Can parallel visual generation be achieved while preserving the simplicity and flexibility of standard autoregressive models? iii) Key methodology: The authors propose a parallel generation strategy that generates weakly dependent tokens in parallel across non-local regions while maintaining sequential generation for strongly dependent local tokens, implemented by dividing the image into regions and using a token re-ordering mechanism. iv) Primary results: The proposed method achieves a 3.6x speedup with comparable image quality and up to a 9.5x speedup with minimal quality degradation on image and video generation tasks. Specifically, the method reduces generation time from 12.41s to 3.46s (PAR-4x) on the ImageNet dataset. v) Principal implication for AI practitioners: AI practitioners can integrate this approach into existing autoregressive models to significantly accelerate the visual generation process with minimal impact on quality, enabling more efficient deployment in real-world applications. |
| SCOPE: Optimizing Key-Value Cache Compression in Long-context Generation (Read more on arXiv or HuggingFace) | Yilong Lai, Zhenglin Wang, zhoudeyu, lzhang472, callanwu | Here is a concise summary of the research paper “SCOPE: Optimizing Key-Value Cache Compression in Long-context Generation”: i) Summary: This paper introduces SCOPE, a framework for optimizing Key-Value (KV) cache compression in large language models (LLMs) during long-context generation by separately compressing the prefill and decoding phases. ii) Main research question or objective: How to effectively compress the KV cache in LLMs for long-context generation tasks without significantly degrading performance. iii) Key methodology: SCOPE preserves the KV cache during the prefill phase and uses a sliding strategy with adaptive and discontinuous optimizations to select and manage heavy hitters during the decoding phase. iv) Primary results: SCOPE achieved comparable performance to the full KV cache when the overall compression rate was 35% on the LONGGENBENCH benchmark. v) Principal implication for AI practitioners: AI practitioners can use SCOPE to optimize memory usage and transfer during long-context generation without losing the performance, particularly for reasoning tasks, making it easier to deploy LLMs in resource-constrained environments. |
| Offline Reinforcement Learning for LLM Multi-Step Reasoning (Read more on arXiv or HuggingFace) | yiwu, ZhangShenao, hendrydong, Shibo-UCSD, jwhj | Here is a concise summary of the research paper “Offline Reinforcement Learning for LLM Multi-Step Reasoning”: i) Summary: This paper introduces OREO, an offline reinforcement learning algorithm designed to improve the multi-step reasoning capabilities of large language models (LLMs). ii) Main research question or objective: The main objective is to develop an offline RL method that enhances LLM multi-step reasoning without requiring paired preference data or treating all tokens uniformly. iii) Key methodology used: OREO jointly learns a policy model and value function by optimizing the soft Bellman Equation, enabling finer-grained credit assignment and leveraging unpaired data with sparse rewards. iv) Primary results: OREO outperforms baseline methods, including rejection sampling, DPO, and KTO, on math reasoning and embodied agent control tasks; a 1.5B model trained with OREO achieves a 52.5% accuracy on the MATH dataset. v) Principal implication for AI practitioners: AI practitioners can use OREO to enhance LLMs’ multi-step reasoning abilities using pre-existing datasets without live interaction, and leverage the learned value function for test-time improvements via beam search. |
| CLEAR: Conv-Like Linearization Revs Pre-Trained Diffusion Transformers Up (Read more on arXiv or HuggingFace) | wxcTest, ZhenxiongTang, flyingman | Here is a concise summary of the paper “CLEAR: Conv-Like Linearization Revs Pre-Trained Diffusion Transformers Up”: i) Summary: This paper introduces CLEAR, a method to linearize the attention mechanism in pre-trained Diffusion Transformers (DiTs) for efficient high-resolution image generation. ii) Main Research Question/Objective: Can a pre-trained DiT be converted to achieve linear computational complexity without significant performance degradation? iii) Key Methodology: CLEAR employs a convolution-like local attention strategy that limits feature interactions to a local window around each query token, ensuring linear complexity. Knowledge distillation is used during fine-tuning. iv) Primary Results: CLEAR reduces attention computations by 99.5% and accelerates generation by 6.3 times for 8K-resolution images, achieving comparable results to the teacher model after fine-tuning on 10K self-generated samples. v) Principal Implication for AI Practitioners: AI practitioners can leverage CLEAR to significantly improve the efficiency of high-resolution image generation using DiTs, enabling faster inference and reduced computational costs, particularly for ultra-high-resolution outputs. |
| Taming Multimodal Joint Training for High-Quality Video-to-Audio Synthesis (Read more on arXiv or HuggingFace) | Akio Hayakawa, mittu1204, TakashiShibuyaSony, mi141, hkchengrex | Here’s a concise summary of the paper, following your guidelines: i) Summary: This paper introduces MMAudio, a multimodal framework for generating high-quality and temporally aligned audio for video and text inputs, using joint training on audio-visual and audio-text datasets. ii) Main research question or objective: How to synthesize high-quality audio that is semantically and temporally aligned to video inputs, with optional text conditioning. iii) Key methodology: MMAudio utilizes a multimodal transformer network trained with a flow-matching objective and incorporates a conditional synchronization module for frame-level audio-visual alignment. Additionally, it leverages joint training on large-scale audio-visual and audio-text datasets. iv) Primary results: MMAudio achieves state-of-the-art performance in video-to-audio synthesis among public models, demonstrating improved audio quality, semantic alignment, and temporal alignment; the smallest model (157M parameters) achieves a 10% lower Fréchet Distance compared to previous methods. v) Principal implication for AI practitioners: AI practitioners can leverage MMAudio’s multimodal joint training paradigm and conditional synchronization module to develop more effective video-to-audio synthesis models, enabling the creation of higher-quality, more realistic audio for video content. |
| MixLLM: LLM Quantization with Global Mixed-precision between Output-features and Highly-efficient System Design (Read more on arXiv or HuggingFace) | chuanjieliu, xiaonans, JamesTheZ | Here is a concise summary of the paper “MixLLM: LLM Quantization with Global Mixed-precision between Output-features and Highly-efficient System Design”: i) MixLLM is a quantization method that applies mixed-precision to different output features based on their globally assessed impact on model loss, achieving high accuracy and system efficiency. ii) The main research objective is to develop a quantization solution for Large Language Models (LLMs) that simultaneously optimizes accuracy, memory consumption, and system efficiency. iii) Key methodology involves identifying high-salience output features globally, applying mixed-precision (4-bit and 8-bit) quantization to weights, using 8-bit symmetric quantization for activations, and designing a two-step dequantization process with optimized GPU kernel execution. iv) Primary results show that MixLLM with only 10% more bits (W4.4A8) reduces perplexity (PPL) increasement from about 0.5 in state-of-the-art methods to within 0.2 for Llama 3.1 70B. v) The principal implication for AI practitioners is that MixLLM provides a method for deploying LLMs with significantly reduced memory footprint and improved inference speed without substantial accuracy loss, facilitating more efficient use of computational resources. |
| LLMs Lost in Translation: M-ALERT uncovers Cross-Linguistic Safety Gaps (Read more on arXiv or HuggingFace) | navigli, mbrack, PSaiml, sted97, felfri | Here is a concise summary of the AI research paper “LLMs Lost in Translation: M-ALERT uncovers Cross-Linguistic Safety Gaps”: i) Summary: This paper introduces M-ALERT, a multilingual benchmark for evaluating the safety of Large Language Models (LLMs) across five languages, revealing significant safety inconsistencies. ii) Main research question or objective: The main objective is to evaluate the safety performance of LLMs across multiple languages (English, French, German, Italian, and Spanish) and identify potential safety gaps. iii) Key methodology: The authors developed a translation pipeline using advanced machine translation models to create M-ALERT, a benchmark with 75k safety prompts (15k per language), and evaluated 10 state-of-the-art LLMs using an automated evaluation framework involving a multilingual judge model (LlamaGuard-3). iv) Primary results: The study found that no model achieved the safe threshold (99%) across all languages, and the c4ai-command model exhibited the lowest safety performance, with scores predominantly below 90%. v) Principal implication for AI practitioners: AI practitioners must prioritize language-specific safety analysis and implement robust multilingual safety measures to ensure responsible LLM deployment globally, as current models exhibit significant safety inconsistencies across different languages. |
| Sequence Matters: Harnessing Video Models in 3D Super-Resolution (Read more on arXiv or HuggingFace) | juxhee, blee, yi0109-park, HEOK, lanikoisgod | Here is a concise summary of the AI research paper “Sequence Matters: Harnessing Video Models in 3D Super-Resolution”: i) This paper introduces a novel approach for 3D super-resolution by leveraging video super-resolution (VSR) models to enhance the quality of 3D models reconstructed from low-resolution multi-view images. ii) The main research objective is to improve the consistency and detail of high-fidelity 3D models generated from low-resolution inputs by utilizing VSR models. iii) The key methodology involves ordering unordered low-resolution multi-view images into a sequence using a simple greedy algorithm based on either camera poses or visual features, and applying adaptive-length subsequencing and multiple thresholds to refine the input for VSR models. iv) The proposed method achieved a PSNR of 31.41 on the NeRF-synthetic dataset, outperforming other baseline models. v) The principal implication for AI practitioners is that they can generate more accurate and detailed 3D models from low-resolution images by effectively ordering input images, without requiring additional fine-tuning or training of 3D Gaussian Splatting (3DGS) on low-resolution images to render ‘smooth’ video. |
| Fietje: An open, efficient LLM for Dutch (Read more on arXiv or HuggingFace) | BramVanroy | Here’s a concise summary of the research paper “Fietje: An open, efficient LLM for Dutch” by Bram Vanroy, following your guidelines: i) Summary: This paper introduces Fietje, a 2.7 billion parameter language model specifically adapted for Dutch, alongside instruction-tuned and chat-optimized variants, with a focus on transparency and reproducibility. ii) Main research question/objective: To develop and evaluate an efficient, open-source language model specifically for the Dutch language that demonstrates competitive performance. iii) Key methodology: Continued pretraining of the English-centric Phi-2 model on 28 billion Dutch tokens sourced from filtered web data (CulturaX) and Wikipedia, followed by supervised fine-tuning and preference alignment using synthetic Dutch datasets. iv) Primary results: Fietje Chat outperformed larger models like GEITje 7B Ultra in two out of five tasks, and on the DBRD benchmark, Boreas Chat achieved a 94.38% F1 score. v) Principal implication for AI practitioners: AI practitioners can leverage Fietje’s open-source nature (model weights, datasets, training, and evaluation code) to advance the development and assessment of efficient, high-performing LLMs and SLMs for underrepresented languages like Dutch, but should be aware of rapid changes in state-of-the-art models and the limitations of current evaluation methodologies. |
Papers for 2024-12-20
| Title | Authors | Summary |
|---|---|---|
| Qwen2.5 Technical Report (Read more on arXiv or HuggingFace) | Losin94, bowenYu, bzheng, huybery, Baosong | Here’s a concise summary of the Qwen2.5 Technical Report, strictly following the specified guidelines: i) A 1-line summary Qwen2.5 is a series of large language models designed with enhanced pre-training and post-training techniques to improve performance across various tasks. ii) Main research question or objective The main objective was to develop Qwen2.5, an improved iteration of large language models (LLMs) with enhanced capabilities in language understanding, reasoning, mathematics, coding, and human preference alignment. iii) Key methodology used The key methodology involved scaling pre-training data to 18 trillion tokens, implementing supervised finetuning with over 1 million samples, and using multistage reinforcement learning including offline learning DPO and online learning GRPO. iv) Primary results (include one specific quantitative finding) The Qwen2.5-72B-Instruct model outperformed numerous open and proprietary models, achieving a score of 83.1 on the MATH benchmark. v) Principal implication for AI practitioners (e.g., AI/ML/Software Engineers, Data Scientist) AI practitioners can leverage Qwen2.5’s architecture and training techniques as a foundation for developing specialized models or applications requiring advanced language understanding and generation capabilities, particularly in domains requiring strong mathematical reasoning. |
| MegaPairs: Massive Data Synthesis For Universal Multimodal Retrieval (Read more on arXiv or HuggingFace) | BoZhaoHuggingFace, yzwang, Shitao, zl101, JUNJIE99 | Here is a concise summary of the AI research paper “MegaPairs: Massive Data Synthesis For Universal Multimodal Retrieval”: i) Summary: The paper introduces MegaPairs, a new method for synthesizing large-scale multimodal datasets for training universal multimodal retrieval models. ii) Main Research Question/Objective: To develop a method for creating high-quality, large-scale instruction-tuning datasets to improve multimodal retrieval performance. iii) Key Methodology: MegaPairs constructs heterogeneous KNN triplets from open-domain images using multiple similarity models and utilizes open-source VLM and LLM annotators to generate instructions for sampled image pairs. iv) Primary Results: Models trained on MegaPairs achieved state-of-the-art zero-shot performance on composed image retrieval benchmarks; notably, the MMRet-MLLM model achieved 42.2% mAP@5 on the CIRCO benchmark. v) Principal Implication for AI Practitioners: AI practitioners can leverage the publicly available MegaPairs dataset, well-trained models, and data synthesis pipeline to develop more powerful and versatile multimodal retrieval systems. |
| Progressive Multimodal Reasoning via Active Retrieval (Read more on arXiv or HuggingFace) | douzc, yutaozhu94, dengmengjie, Snow-Nation, dongguanting | Here’s a concise summary of the research paper “Progressive Multimodal Reasoning via Active Retrieval”: i) This paper introduces AR-MCTS, a framework that enhances multimodal reasoning in large language models (MLLMs) by integrating active retrieval with Monte Carlo Tree Search (MCTS). ii) The main research objective is to improve the performance of MLLMs on complex multi-step multimodal reasoning tasks. iii) The key methodology involves a unified retrieval module for acquiring key insights, an active retrieval strategy during MCTS expansion, and a progressively aligned process reward model (PRM). iv) The primary results show that AR-MCTS significantly improves performance across various MLLMs; for example, Qwen2-VL-7B with AR-MCTS achieved a 5.3% improvement on the MATHVISTA benchmark compared to its zero-shot setting. v) For AI practitioners, AR-MCTS offers a plug-and-play framework to enhance MLLMs’ reasoning capabilities without retraining the foundational models, providing a way to optimize sampling diversity and accuracy in multimodal reasoning tasks. |
| LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitasks (Read more on arXiv or HuggingFace) | wangxz098, haopeng01, NeoZ123, tsq2000, bys0318 | Here is a concise summary of the paper “LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitasks” based on your requirements: i) Summary: LongBench v2 is a benchmark designed to evaluate the deep understanding and reasoning capabilities of large language models (LLMs) on long-context, real-world multitasks. ii) Main research question or objective: The main objective is to create a challenging benchmark to assess whether LLMs can genuinely comprehend, learn from, and reason over long texts, ranging from 8k to 2M words, across diverse real-world scenarios. iii) Key methodology used: The researchers collected 503 multiple-choice questions from nearly 100 human experts, categorized into six task types, and implemented a rigorous annotation and review process involving both automated checks using LLMs and manual verification by human experts to ensure data quality and difficulty. iv) Primary results: The best-performing LLM (01-preview model) achieved 57.7% accuracy when incorporating longer reasoning, whereas human experts achieved only 53.7% accuracy under a 15-minute time constraint. v) Principal implication for AI practitioners: AI practitioners should focus on enhancing the reasoning capabilities and scaling inference-time compute of LLMs to address the challenges posed by long-context tasks that require deep understanding, as opposed to mere retrieval or shallow processing of information. |
| How to Synthesize Text Data without Model Collapse? (Read more on arXiv or HuggingFace) | XingtaiHF, iseesaw, Hengli, daixuancheng, xuekai | Here is a concise summary of the research paper “How to Synthesize Text Data without Model Collapse?”: i) This paper investigates the impact of synthetic data on language model training and proposes a token-level editing method to mitigate model collapse. ii) The main research questions are: what is the impact of synthetic data on language model training, and how can data be synthesized without causing model collapse? iii) The key methodology used is pre-training language models on varying proportions of synthetic and human-produced data, statistical analysis of synthetic data distributions, and a proposed token-level editing approach with theoretical proof and empirical validation. iv) The primary results show a negative correlation between the proportion of synthetic data and model performance, with the perplexity of models trained on synthetic data reaching 49.30 on average compared to 21.37 for human data. v) The principal implication for AI practitioners is that directly using synthetic data in training can lead to performance degradation (model collapse), and token-level editing can be used to improve data quality and enhance model performance. |
| Flowing from Words to Pixels: A Framework for Cross-Modality Evolution (Read more on arXiv or HuggingFace) | Andrew Brown, Alan Yuille, Xi Yin, mannatsingh, QHL067 | Here is a concise summary of the research paper “Flowing from Words to Pixels: A Framework for Cross-Modality Evolution”: i) The paper introduces CrossFlow, a framework that directly evolves one modality into another using flow matching without additional conditioning. ii) The main research question is whether flow matching models can learn a direct mapping between the distributions of different modalities, obviating noise and conditioning mechanisms. iii) The key methodology involves using Variational Encoders to encode source modality data to the same shape as the target modality and a novel method to enable Classifier-free guidance in a cross-modal flow matching setting. iv) CrossFlow achieved a zero-shot FID-30K score of 9.63 on COCO for text-to-image generation, outperforming standard flow matching baselines. v) For AI practitioners, CrossFlow offers a simpler and more scalable framework for cross-modal generation tasks, demonstrating that direct evolution between modalities is achievable and efficient. |
| LeviTor: 3D Trajectory Oriented Image-to-Video Synthesis (Read more on arXiv or HuggingFace) | lmwang, cqf, felixcheng97, qiuyuu, hlwang06 | Here is a concise summary of the research paper “LeviTor: 3D Trajectory Oriented Image-to-Video Synthesis”: i) Summary: LeviTor is a novel image-to-video synthesis method that enables precise 3D trajectory control of objects by combining depth information with K-means clustered points. ii) Main research question or objective: The main objective was to develop a method for controlling object trajectories in image-to-video synthesis that can handle out-of-plane movements and occlusions in 3D space, overcoming the limitations of existing 2D trajectory-based methods. iii) Key methodology: The authors propose representing control signals by combining depth information with K-means clustered points derived from object masks and using this representation to guide a fine-tuned video diffusion model (Stable Video Diffusion). iv) Primary results: LeviTor achieves accurate 3D trajectory control, demonstrated by a Frechet Video Distance (FVD) of 190.44 on the DAVIS dataset with the multi-points setting, compared to 330.17 for DragNUWA 1.5 in single point setting. v) Principal implication for AI practitioners: AI practitioners can utilize LeviTor to generate videos with precise control over object movements in 3D space, enabling more realistic and complex video synthesis without requiring explicit 3D trajectory inputs from users. |
| Affordance-Aware Object Insertion via Mask-Aware Dual Diffusion (Read more on arXiv or HuggingFace) | Ye Liu, hpfister, dwei, EthanTaylor, Kakituken | Here is a concise summary of the research paper “Affordance-Aware Object Insertion via Mask-Aware Dual Diffusion”: i) Summary: This paper introduces a new task and method for inserting objects into images realistically, guided by affordance and position prompts, using a novel dataset and a dual-diffusion model. ii) Main research question/objective: How to develop a model for affordance-aware object insertion that can seamlessly integrate any object into any scene with various position prompts. iii) Key methodology: The authors propose a Mask-Aware Dual Diffusion (MADD) model, which uses a dual-stream architecture to denoise the RGB image and the insertion mask simultaneously, trained on a new dataset (SAM-FB) derived from SA-1B. iv) Primary results: MADD outperforms state-of-the-art methods on the affordance-aware object insertion task; for example it achieves an FID score of 13.53 with mask prompts, compared to 15.41 for Stable Diffusion. v) Principal implication for AI practitioners: AI practitioners can utilize the MADD model and the SAM-FB dataset for realistic image composition, with explicit control over object placement and appearance via diverse prompts. |
| DI-PCG: Diffusion-based Efficient Inverse Procedural Content Generation for High-quality 3D Asset Creation (Read more on arXiv or HuggingFace) | Yuejiang Dong, yshan2u, bluestyle97, pookiefoof, thuzhaowang | Here is a concise summary of the research paper “DI-PCG: Diffusion-based Efficient Inverse Procedural Content Generation for High-quality 3D Asset Creation” based on the provided guidelines: i) DI-PCG is a diffusion-based method for efficient inverse procedural content generation (I-PCG) that creates high-quality 3D assets from image conditions. ii) The main research objective is to automatically estimate the best-fit parameters for procedural generators under given image conditions to achieve controllable 3D content generation. iii) The key methodology is a lightweight diffusion transformer model that treats PCG parameters as the denoising target and observed images as conditions to control parameter generation. iv) The primary result is that DI-PCG achieves a Chamfer Distance (CD) of 0.093 on the ShapeNet chair subset, demonstrating accurate parameter recovery. v) The principal implication for AI practitioners is that DI-PCG offers an efficient and effective way to perform inverse procedural content generation, which can be used for high-quality image-to-3D generation. |
| AceMath: Advancing Frontier Math Reasoning with Post-Training and Reward Modeling (Read more on arXiv or HuggingFace) | wping, ctnzr, shoeybi, ychenNLP, zihanliu | Here is a concise summary of the research paper “AceMath: Advancing Frontier Math Reasoning with Post-Training and Reward Modeling”: i) Summary: The paper introduces AceMath, a suite of math-specialized language models and reward models designed to enhance mathematical reasoning capabilities. ii) Main research question or objective: The main objective is to develop advanced supervised fine-tuning (SFT) and reward modeling (RM) techniques to improve the performance of large language models (LLMs) on complex mathematical reasoning tasks. iii) Key methodology used: The methodology involves a two-stage SFT process (general domain followed by math-specific fine-tuning) using curated prompts and synthetically generated responses, and a systematic approach to build math reward models evaluated on a new benchmark called AceMath-RewardBench. iv) Primary results: The resulting AceMath-72B-Instruct model outperforms Qwen2.5-Math-72B-Instruct, GPT-40, and Claude-3.5 Sonnet on math reasoning benchmarks. Specifically, AceMath-72B-Instruct achieves an average score of 71.84 across seven math reasoning benchmarks, compared to 68.16 for Qwen2.5-Math-72B-Instruct. v) Principal implication for AI practitioners: AI practitioners can leverage the proposed SFT and RM techniques, along with the provided open-source models and data, to develop more powerful and accurate math-specialized LLMs, pushing the boundaries of automated mathematical reasoning. |
| UIP2P: Unsupervised Instruction-based Image Editing via Cycle Edit Consistency (Read more on arXiv or HuggingFace) | Federico Tombari, Yongqin Xian, thofmann, Alessiot, enisimsar | Here’s a concise summary of the research paper “UIP2P: Unsupervised Instruction-based Image Editing via Cycle Edit Consistency” based on the provided guidelines: i) Summary: The paper introduces UIP2P, an unsupervised instruction-based image editing model that uses Cycle Edit Consistency (CEC) to enable reversible and coherent edits without requiring ground-truth edited images during training. ii) Main research question or objective: How to develop an instruction-based image editing model that does not rely on supervised datasets containing triplets of input image, edited image, and edit instruction. iii) Key methodology used: Cycle Edit Consistency (CEC) is enforced by applying forward and reverse edits in one training step and ensuring consistency in image, attention, and CLIP embedding spaces, leveraging unified prediction with varying diffusion steps. iv) Primary results: UIP2P outperforms InstructPix2Pix on the IP2P test dataset in both CLIP image similarity and CLIP text-image similarity metrics; for instance, it achieves a 22% preference score in user studies compared to 8% for InstructPix2Pix when evaluating how well the edit matches the instruction and localization. v) Principal implication for AI practitioners: AI practitioners can leverage UIP2P to train image editing models on real-image datasets without the need for ground-truth edited images, enabling the use of large-scale datasets that lack such annotations. |
| Descriptive Caption Enhancement with Visual Specialists for Multimodal Perception (Read more on arXiv or HuggingFace) | Ke Zhu, Jing Hao, FuNz, cloud913, syp115 | Here’s a summary of the paper, following your specified guidelines: i) The paper introduces Descriptive Caption Enhancement (DCE), a method that enhances image captions by integrating outputs from multiple visual specialist models. ii) The main objective is to generate more detailed and accurate image captions than existing methods, which rely on human annotations or large multimodal models (LMMs). iii) DCE leverages various visual specialists (e.g., for object detection, depth estimation, emotion recognition) to extract attributes, then uses a large language model (LLM) to combine these into a coherent caption. iv) When trained with DCE, LLaVA-v1.5 achieved an accuracy of 80.9 on the VQAv2 benchmark. v) AI practitioners can use DCE to improve the performance of LMMs on visual understanding tasks by providing them with more comprehensive and detailed image captions, generated without relying on expensive human annotation. |
| TOMG-Bench: Evaluating LLMs on Text-based Open Molecule Generation (Read more on arXiv or HuggingFace) | Qing Li, Yunqing Liu, Jiatong Li, schrodingers-tiger, Duke-de-Artois | Here is a concise summary of the research paper “TOMG-Bench: Evaluating LLMs on Text-based Open Molecule Generation”: i) Summary: This paper introduces TOMG-Bench, a benchmark for evaluating large language models (LLMs) on text-based open molecule generation, alongside an instruction-tuning dataset, OpenMolIns. ii) Main research question or objective: The main objective was to evaluate the capability of LLMs to generate novel molecules based on open-ended textual instructions, moving beyond targeted molecule generation. iii) Key methodology: The authors developed a benchmark (TOMG-Bench) with three tasks (molecule editing, optimization, and customized generation), each with three subtasks. They also used an automated evaluation system and a new instruction-tuning dataset (OpenMolIns) to assess 25 LLMs. iv) Primary results: The best performing model, Claude-3.5, achieved a weighted average accuracy of 35.92% on TOMG-Bench, while instruction-tuned Llama3.1-8B outperformed all open-source general LLMs. v) Principal implication for AI practitioners: AI practitioners can leverage TOMG-Bench to assess LLMs for open-domain molecule generation tasks and use OpenMolIns to improve model performance in this area, although there is still significant room for improvement in generating molecules from scratch. |
| Move-in-2D: 2D-Conditioned Human Motion Generation (Read more on arXiv or HuggingFace) | Feng Liu, Difan Liu, Jui-Hsien Wang, Yang Zhou, hsinh | Here is a concise summary of the research paper “Move-in-2D: 2D-Conditioned Human Motion Generation”: i) This paper introduces a novel method, Move-in-2D, for generating realistic human motion sequences conditioned on a 2D scene image and a text prompt. ii) The main research objective is to generate diverse human motion sequences that are semantically aligned with a text prompt and spatially compatible with a given 2D background image. iii) The key methodology is a multi-conditional diffusion model that utilizes a transformer architecture with in-context learning to integrate scene image and text prompt conditions. iv) The proposed model achieved an FID score of 44.639, which is better than other compared models. v) For AI practitioners, this method provides a new modality for motion generation by incorporating scene awareness without requiring 3D scene data and improves motion quality in human video generation tasks. |
Papers for 2024-12-19
| Title | Authors | Summary |
|---|---|---|
| TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks (Read more on arXiv or HuggingFace) | Kritanjali Jain, Yuxuan Tang, Boxuan Li, Yufan Song, Frank F. Xu | Here is a concise summary of the paper “TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks” based on your specified guidelines: i) Summary: This paper introduces TheAgentCompany, a benchmark for evaluating large language model (LLM) agents on realistic, consequential tasks within a simulated software company environment. ii) Main research question or objective: To assess the capability of LLM agents to autonomously perform complex, multi-step, work-related tasks in a realistic setting. iii) Key methodology used: A self-contained, simulated software company environment was created using internal websites and data, with tasks requiring agents to browse the web, code, run programs, and communicate with simulated coworkers. iv) Primary results: The best-performing agent, powered by Claude 3.5 Sonnet, achieved a 24.0% task completion rate and a 34.4% partial completion score. v) Principal implication for AI practitioners: The benchmark demonstrates that while current LLM agents can complete some work-related tasks, significant improvements are needed, particularly in handling complex user interfaces, social interactions, and tasks that lack public training data before they can be reliably deployed for a wide range of real-world applications. |
| AniDoc: Animation Creation Made Easier (Read more on arXiv or HuggingFace) | Wen Wang, Qiuyu Wang, Hanlin Wang, Hao Ouyang, Yihao Meng | Here is a concise summary of the research paper “AniDoc: Animation Creation Made Easier”: i) AniDoc is a novel AI model designed to automate 2D animation coloring by converting sketch sequences into colored animations based on a reference character image. ii) Main research question/objective: How to automate the colorization of 2D animation line art while maintaining fidelity to a reference character design and ensuring temporal consistency across frames? iii) Key methodology: A video diffusion model with correspondence-guided colorization, binarization, background augmentation, and a two-stage sparse sketch training strategy. iv) Primary results: AniDoc achieved a PSNR of 19.23, demonstrating superior performance in colorization accuracy compared to existing methods. v) Principal implication for AI practitioners: AI practitioners can utilize AniDoc to significantly reduce the labor costs and time required for 2D animation production by automating the colorization process. |
| FashionComposer: Compositional Fashion Image Generation (Read more on arXiv or HuggingFace) | Hao Luo, Xiaogang Xu, Xi Chen, Yiyang Wang, Sihui Ji | Here is a concise summary of the research paper “FashionComposer: Compositional Fashion Image Generation”: i) FashionComposer is a novel framework for generating fashion images that allows for detailed control over garment styles, human poses, and appearances using multi-modal inputs. ii) The main research objective is to develop a highly flexible system capable of handling diverse input modalities and composing multiple visual assets (garments, faces) in a single fashion image generation process. iii) The key methodology involves a diffusion-based model with a universal framework for multi-modal inputs, a reference UNet for extracting appearance features from an “asset library”, and a subject-binding attention mechanism to bind appearance features to corresponding text features. iv) The primary result is that FashionComposer outperforms existing methods in multi-object reference generation, achieving a CLIP-I score of 77.60 compared to 69.70 for Emu2. v) For AI practitioners, FashionComposer offers a powerful and flexible framework for compositional fashion image generation, which has direct applications in virtual try-on, controllable model image generation, and human album generation. |
| Efficient Diffusion Transformer Policies with Mixture of Expert Denoisers for Multitask Learning (Read more on arXiv or HuggingFace) | Rudolf Lioutikov, Pulkit Agrawal, Jyothish Pari, Moritz Reuss | Here’s a concise summary of the research paper, strictly adhering to the specified guidelines: i) Summary: The paper introduces Mixture-of-Denoising Experts (MoDE), a novel policy for Imitation Learning that uses a Mixture-of-Experts Transformer architecture with noise-conditioned routing and self-attention for efficient multitask learning. ii) Main research question or objective: The main objective is to develop a more computationally efficient Diffusion Policy for Imitation Learning that maintains or surpasses the performance of state-of-the-art Transformer-based Diffusion Policies. iii) Key methodology used: The key methodology is a Mixture-of-Experts (MoE) Transformer architecture with a novel noise-conditioned router that assigns tokens to experts based on noise levels during the denoising process, combined with a noise-conditioned self-attention mechanism. iv) Primary results: MoDE outperforms existing Diffusion Policies on 134 tasks across four benchmarks, achieving 4.01 on the CALVIN ABC benchmark and surpassing baselines by an average of 57% while using 90% fewer FLOPs. v) Principal implication for AI practitioners: AI practitioners can leverage MoDE’s architecture for more efficient and scalable Imitation Learning, reducing computational costs during training and inference of Diffusion Policies without sacrificing performance, particularly in multitask settings. |
| Prompting Depth Anything for 4K Resolution Accurate Metric Depth Estimation (Read more on arXiv or HuggingFace) | Jiaming Sun, Songyou Peng, Jingxiao Chen, Sida Peng, Haotong Lin | Here is a concise summary of the research paper “Prompting Depth Anything for 4K Resolution Accurate Metric Depth Estimation” following the specified guidelines: i) Summary: This paper introduces “Prompt Depth Anything,” a novel paradigm for metric depth estimation that utilizes low-cost LiDAR data as a prompt to guide a depth foundation model, achieving accurate depth output at up to 4K resolution. ii) Main research question or objective: How to effectively prompt depth foundation models to achieve accurate metric depth estimation at high resolution. iii) Key methodology: A concise prompt fusion architecture is used to integrate LiDAR depth at multiple scales within the depth decoder, combined with a scalable data pipeline that includes synthetic LiDAR simulation and real data pseudo-GT depth generation, along with an edge-aware depth loss. iv) Primary results: The method achieves state-of-the-art results on ARKitScenes and ScanNet++ datasets, with a quantitative finding of 0.0132 L1 error on the ARKitScenes dataset at 384 x 512 resolution. v) Principal implication for AI practitioners: AI practitioners can leverage Prompt Depth Anything to enhance the accuracy and resolution of metric depth estimation in applications such as 3D reconstruction and robotic grasping by effectively integrating low-cost LiDAR prompts with depth foundation models. |
| GUI Agents: A Survey (Read more on arXiv or HuggingFace) | Namyong Park, Gang Wu, Yu Wang, Jian Chen, dangmn | Here is a concise summary of the research paper “GUI Agents: A Survey”: i) This survey provides a comprehensive overview of GUI agents powered by Large Foundation Models (LFMs) that automate human-computer interactions. ii) The main objective is to categorize and analyze existing GUI agent benchmarks, evaluation metrics, architectures, and training methods. iii) The key methodology used is a literature review, synthesizing various types of contributions within the field and proposing a unified framework based on GUI agents’ perception, reasoning, planning, and acting capabilities. iv) The primary results include a structured analysis of datasets (e.g., Mind2Web contains 2000 diverse tasks) and environments for evaluating GUI agents across various platforms, along with architectural designs and training strategies. v) The principal implication for AI practitioners is the need for standardized benchmarks and evaluation metrics to systematically assess and advance the development of GUI agents. |
| AnySat: An Earth Observation Model for Any Resolutions, Scales, and Modalities (Read more on arXiv or HuggingFace) | Loic Landrieu, Clement Mallet, Nicolas Gonthier, Guillaume Astruc | Here is a concise summary of the research paper “AnySat: An Earth Observation Model for Any Resolutions, Scales, and Modalities”: i) AnySat is a novel self-supervised multimodal Earth observation (EO) model designed to handle heterogeneous data with varying resolutions, scales, and modalities. ii) The main research objective is to develop a single EO model capable of integrating diverse datasets for training and prediction without modality-specific adaptations. iii) The key methodology is a joint embedding predictive architecture (JEPA) with scale-adaptive spatial encoders, trained on a new multimodal dataset collection called GeoPlex. iv) The primary results show that AnySat achieves state-of-the-art or near state-of-the-art performance on multiple EO tasks; for instance, it achieved a 72.8 weighted F1 score on the TreeSatAI-TS classification task. v) For AI practitioners, AnySat offers a versatile pretrained model that can be fine-tuned or linearly probed for various downstream EO tasks, even with new combinations of modalities not seen during pretraining, simplifying the development of applications with diverse EO data. |
| RAG-RewardBench: Benchmarking Reward Models in Retrieval Augmented Generation for Preference Alignment (Read more on arXiv or HuggingFace) | Yubo Chen, Pengfei Cao, Tianyi Men, Hongbang Yuan, Zhuoran Jin | Here is a concise 4-5 sentence summary of the paper: i) Summary: The paper introduces RAG-RewardBench, a benchmark for evaluating reward models (RMs) in retrieval-augmented generation (RAG) systems tailored to align with human preferences. ii) Research Question/Objective: How to evaluate and select a reliable reward model for preference alignment in RAG language models. iii) Methodology: The authors designed four RAG-specific scenarios (multi-hop reasoning, fine-grained citation, appropriate abstain, conflict robustness), incorporated 18 RAG subsets, six retrievers, and 24 RAG language models, and used an LLM-as-a-judge approach for preference annotation. iv) Results: Existing RMs are challenged by RAG-RewardBench, with the top-ranked RM, Skywork-Critic-Llama-3.1-70B, achieving only 78.3% accuracy. v) Implication: AI practitioners should prioritize developing specialized reward models tailored for RAG systems to improve the alignment of these models with human preferences, as existing reward models show limitations in RAG-specific scenarios. |
| Mix-LN: Unleashing the Power of Deeper Layers by Combining Pre-LN and Post-LN (Read more on arXiv or HuggingFace) | Shiwei Liu, Lu Yin, Pengxiang Li | Here’s a concise summary of the research paper “Mix-LN: Unleashing the Power of Deep Layers by Combining Pre-LN and Post-LN”: i) Summary: This paper introduces Mix-LN, a novel normalization technique that combines Pre-Layer Normalization (Pre-LN) and Post-Layer Normalization (Post-LN) to improve the training and performance of deep layers in Large Language Models (LLMs). ii) Main research question/objective: The main research objective is to investigate whether the choice of layer normalization (Pre-LN vs. Post-LN) impacts the effectiveness of deeper layers in LLMs and to develop a method that addresses the limitations of both approaches. iii) Key methodology: The authors empirically evaluated layer effectiveness using angular distance and performance drop metrics across various model sizes (70M to 7B parameters) and compared Pre-LN, Post-LN, and the proposed Mix-LN, which applies Post-LN to earlier layers and Pre-LN to deeper layers. iv) Primary results: Mix-LN consistently outperformed both Pre-LN and Post-LN in pre-training; specifically, Mix-LN achieved a perplexity of 18.18 on the LLaMA-1B model, compared to 18.65 for Pre-LN. v) Principal implication for AI practitioners: AI practitioners can leverage Mix-LN to enhance the training of LLMs by ensuring more uniform gradient norms across all layers, leading to improved model capacity without increasing model size. |
| Learning from Massive Human Videos for Universal Humanoid Pose Control (Read more on arXiv or HuggingFace) | Junjie Ye, Tianheng Shi, Siqi Song, Siheng Zhao, Jiageng Mao | Here’s a concise summary of the AI research paper “Learning from Massive Human Videos for Universal Humanoid Pose Control”: Summary: i) This paper introduces Humanoid-X, a large-scale dataset of over 20 million humanoid robot poses with corresponding text-based motion descriptions, and UH-1, a Transformer-based model for universal language-conditioned pose control of humanoid robots. ii) The main research objective is to investigate whether a universal humanoid pose control model can be trained using large-scale text-action pairs derived from massive human videos. iii) The key methodology involves curating Humanoid-X through data mining, video captioning, motion retargeting from humans to humanoids, and reinforcement learning, followed by training UH-1 to map text instructions to humanoid actions using a Transformer architecture. iv) The primary results show that UH-1 achieves state-of-the-art performance on the HumanoidML3D benchmark, with a Frechet Inception Distance (FID) score of 0.379. v) The principal implication for AI practitioners is that leveraging massive human video data and the proposed training pipeline can enable the development of highly generalizable and scalable humanoid control models, significantly advancing the deployment of adaptable humanoid robots in real-world applications. |
| ChatDiT: A Training-Free Baseline for Task-Agnostic Free-Form Chatting with Diffusion Transformers (Read more on arXiv or HuggingFace) | Yupeng Shi, Zhi-Fan Wu, Wei Wang, Lianghua Huang, bibona | Here is a concise summary of the research paper “ChatDiT: A Training-Free Baseline for Task-Agnostic Free-Form Chatting with Diffusion Transformers”: i) Summary: ChatDiT is a zero-shot, general-purpose, interactive visual generation framework that uses pretrained diffusion transformers to perform various visual tasks based on free-form natural language instructions, without any additional training. ii) Main research question or objective: The main objective was to develop a training-free framework leveraging the inherent in-context generation capabilities of pretrained diffusion transformers for interactive and general-purpose image generation. iii) Key methodology used: The methodology involved a multi-agent system with Instruction-Parsing, Strategy-Planning, and Execution Agents, using an in-context toolkit to perform actions with diffusion transformers. iv) Primary results: ChatDiT achieved a Top-1 performance score of 23.19 out of 100 on the IDEA-Bench, outperforming other models. v) Principal implication for AI practitioners: AI practitioners can leverage ChatDiT as a baseline for zero-shot task generalization in image generation, but should be aware of its limitations in handling long contexts and preserving fine-grained details, and work towards addressing these. |
| VidTok: A Versatile and Open-Source Video Tokenizer (Read more on arXiv or HuggingFace) | Li Song, Xinle Cheng, Junliang Guo, Tianyu He, Anni Tang | Here is a concise summary of the paper “VidTok: A Versatile and Open-Source Video Tokenizer” adhering to the specified guidelines: Summary: i) The paper introduces VidTok, an open-source video tokenizer that achieves state-of-the-art performance in both continuous and discrete video tokenization. ii) The main research objective is to develop a versatile video tokenizer that outperforms existing methods in video reconstruction quality across various metrics. iii) The key methodology includes a novel model architecture with separate spatial and temporal sampling, the integration of Finite Scalar Quantization (FSQ) for discrete tokenization, and a two-stage training strategy. iv) In discrete tokenization, VidTok with FSQ (codebook size 262,144) achieves a PSNR of 29.82 on the MCL-JCV dataset, outperforming previous methods. v) For AI practitioners, VidTok offers an advanced tool for video generation and understanding tasks, providing improved video tokenization performance. |
| CAD-Recode: Reverse Engineering CAD Code from Point Clouds (Read more on arXiv or HuggingFace) | Anis Kacem, Kseniya Cherenkova, Dimitrios Mallis, Elona Dupont, Danila Rukhovich | Here is a concise summary of the research paper “CAD-Recode: Reverse Engineering CAD Code from Point Clouds” based on your specific guidelines: i) CAD-Recode translates 3D point clouds into executable Python code to reconstruct CAD models. ii) The main research objective is to develop a method for reverse engineering CAD models from point clouds by leveraging the code generation capabilities of large language models (LLMs). iii) The key methodology involves fine-tuning a pre-trained LLM (Qwen2-1.5B) augmented with a point cloud projector to map input point clouds into Python code representations of CAD sketch-extrude sequences, utilizing a novel synthetic dataset of one million CAD models. iv) The primary results show that CAD-Recode achieves a 10 times lower mean Chamfer distance compared to state-of-the-art methods on the DeepCAD dataset. v) The principal implication for AI practitioners is that CAD-Recode offers a new approach to CAD model reconstruction, providing an effective way to generate editable and interpretable CAD models directly from point cloud data using LLMs, without the need for large, hand-crafted datasets. |
| AntiLeak-Bench: Preventing Data Contamination by Automatically Constructing Benchmarks with Updated Real-World Knowledge (Read more on arXiv or HuggingFace) | Shuai Zhao, Ruiwen Zhou, Yuxi Xie, Liangming Pan, Xiaobao Wu | Here is a concise summary of the research paper “AntiLeak-Bench: Preventing Data Contamination by Automatically Constructing Benchmarks with Updated Real-World Knowledge”: i) Summary: This paper introduces AntiLeak-Bench, a framework for automatically constructing contamination-free benchmarks for evaluating large language models (LLMs) using updated real-world knowledge. ii) Main research question/objective: To develop a method for creating LLM evaluation benchmarks that are free from data contamination and can be easily updated without human labor. iii) Key methodology: The authors use Wikidata to identify knowledge updated after an LLM’s cutoff time, construct question-answering samples based on this knowledge with supporting documents from Wikipedia, and automate the entire benchmark creation and update process. iv) Primary results: Evaluations on AntiLeak-Bench show most models score below 50 in Exact Match (EM), with only GPT-40-mini and GPT-40 achieving EM scores around 70. v) Principal implication for AI practitioners: AI practitioners should use AntiLeak-Bench to obtain a more reliable assessment of LLMs’ true capabilities, ensuring evaluations are not inflated by data contamination, especially when evaluating on knowledge-dependent tasks. |
| LLaVA-UHD v2: an MLLM Integrating High-Resolution Feature Pyramid via Hierarchical Window Transformer (Read more on arXiv or HuggingFace) | Xuesong Yang, Yidan Zhang, Yifan Liu, Yipeng Zhang, guozonghao96 | Here is a concise summary of the research paper “LLaVA-UHD v2: an MLLM Integrating High-Resolution Feature Pyramid via Hierarchical Window Transformer”: i) Summary: The paper introduces LLaVA-UHD v2, a multimodal large language model (MLLM) that integrates a high-resolution feature pyramid via a hierarchical window transformer to enhance visual understanding. ii) Main research question/objective: The main objective is to address the limitation of vision transformers (ViTs) in capturing diverse visual granularity in MLLMs by constructing and integrating a high-resolution feature pyramid. iii) Key methodology: The key methodology involves a Hiwin transformer comprising an inverse feature pyramid constructed by a ViT-derived feature up-sampling process and a hierarchical window attention mechanism that condenses multi-level feature maps. iv) Primary results: LLaVA-UHD v2 achieved superior performance over existing MLLMs, demonstrating an average boost of 3.7% across 14 benchmarks compared with the baseline method. v) Principal implication for AI practitioners: AI practitioners can leverage the Hiwin transformer to develop MLLMs capable of handling tasks requiring diverse visual granularity, such as high-resolution image perception and visual grounding, with improved accuracy. |
Papers for 2024-12-18
| Title | Authors | Summary |
|---|---|---|
| Are Your LLMs Capable of Stable Reasoning? (Read more on arXiv or HuggingFace) | Linchen Xiao, Hongwei Liu, Junnan Liu, zsytony, Harold-lkk | Here’s a concise summary of the research paper “Are Your LLMs Capable of Stable Reasoning?”: i) Summary: This paper introduces G-Pass@k, a new metric to evaluate both the problem-solving ability and performance consistency of Large Language Models (LLMs), alongside a new benchmark, LiveMathBench, for assessing mathematical reasoning. ii) Main research question or objective: How can we assess both the peak performance and stability of LLMs in complex reasoning tasks, particularly in mathematical problem-solving? iii) Key methodology used: The authors propose G-Pass@k, which measures performance consistency across multiple sampling attempts, and LiveMathBench, a dynamic benchmark with contemporary mathematical problems. They evaluate various LLMs using these tools. iv) Primary results: The study found significant instability in LLM reasoning on challenging tasks, with performance drops exceeding 50% in many cases when evaluated using G-Pass@k. For instance, the Llama-3.1-8B-Instruct model’s accuracy plummeted from 18.1% (Greedy) to 0.8% (G-Pass@161.0) on the LiveMathBench. v) Principal implication for AI practitioners: AI practitioners should use G-Pass@k to gain a more realistic assessment of LLM capabilities in complex reasoning, as it reveals that current evaluation metrics may overestimate actual performance consistency, highlighting the need for more stable models in real-world applications. |
| Multi-Dimensional Insights: Benchmarking Real-World Personalization in Large Multimodal Models (Read more on arXiv or HuggingFace) | Xiaoshuai Song, Zhuoma GongQue, Runqi Qiao, Shanglin Lei, YiFan Zhang | Here is a concise summary of the AI research paper “Multi-Dimensional Insights: Benchmarking Real-World Personalization in Large Multimodal Models” based on your guidelines: i) This paper introduces the Multi-Dimensional Insights (MDI) benchmark to evaluate the performance of large multimodal models (LMMs) on real-world personalization tasks across various scenarios, age groups, and problem complexities. ii) The main research objective is to assess whether LMMs can align with the diverse needs of humans in real-world scenarios and address the specific demands of distinct demographic groups. iii) The key methodology involves constructing a dataset of over 500 images and 1.2k human-posed questions spanning six common scenarios, stratified by three age groups and two levels of complexity, and evaluating several LMMs using this benchmark. iv) The primary result is that the strongest model tested, GPT-4o, achieved 79% accuracy on age-related tasks, but with noticeable gaps across different scenarios and complexities. v) The principal implication for AI practitioners is that current LMMs still have considerable room for improvement in addressing real-world applications, particularly in tailoring responses to diverse user needs, highlighting the need for continued development to enhance personalized AI assistant capabilities. |
| OmniEval: An Omnidirectional and Automatic RAG Evaluation Benchmark in Financial Domain (Read more on arXiv or HuggingFace) | Ji-Rong Wen, Zhicheng Dou, Jiejun Tan, ShootingWong | Here is a concise summary of the research paper “OmniEval: An Omnidirectional and Automatic RAG Evaluation Benchmark in Financial Domain”: i) Summary: This paper introduces OmniEval, an automatic and multidimensional benchmark for evaluating Retrieval-Augmented Generation (RAG) models in the financial domain. ii) Main research question/objective: The main objective is to develop a comprehensive benchmark to evaluate the performance of RAG models on various financial topics and tasks. iii) Key methodology: The methodology involves a matrix-based RAG scenario evaluation system, multi-dimensional evaluation data generation using GPT-4 and human annotation, a multi-stage evaluation of retrieval and generation, and multi-dimensional evaluation metrics including rule-based and Large Language Model (LLM)-based ones. iv) Primary results: The automated data generation approach achieved an 87.47% acceptance ratio in human evaluations. v) Principal implication for AI practitioners: OmniEval provides a standardized framework for evaluating and improving RAG models in specialized domains like finance, using the benchmark’s publicly available code. |
| Emergence of Abstractions: Concept Encoding and Decoding Mechanism for In-Context Learning in Transformers (Read more on arXiv or HuggingFace) | Pulkit Agrawal, Jeff Gore, Jinyeop Song, Seungwook Han | Here is a concise summary of the research paper: i) This paper introduces a concept encoding-decoding mechanism to explain how transformers perform in-context learning (ICL). ii) The main research question is how transformers form and use internal abstractions during ICL. iii) The key methodology involves analyzing the training dynamics of a small transformer on synthetic ICL tasks and evaluating concept encoding-decoding across pretrained models of varying scales using techniques like UMAP visualization, concept decodability, and mechanistic intervention. iv) The primary results are that transformers concurrently learn to map latent concepts into separable representations and develop context-specific decoding algorithms, with a positive correlation (R² = 0.781) between concept decodability and ICL performance observed in the POS tagging task using the Llama-3.1 8B model. v) The principal implication for AI practitioners is that enhancing the quality of concept encoding (e.g., through early layer finetuning) can directly improve the ICL performance of transformers. |
| MIVE: New Design and Benchmark for Multi-Instance Video Editing (Read more on arXiv or HuggingFace) | Munchurl Kim, Jihyong Oh, Soo Ye Kim, Agus Gunawan, Samuel Teodoro | Here is a concise summary of the research paper “MIVE: New Design and Benchmark for Multi-Instance Video Editing” based on the provided guidelines: i) The paper introduces MIVE, a zero-shot mask-based framework for multi-instance video editing that disentangles edits and prevents editing leakage. ii) The main research objective is to develop a method for localized editing of multiple objects in videos without unintended changes to other parts of the video. iii) The key methodology uses Disentangled Multi-instance Sampling (DMS) to prevent editing leakage and Instance-centric Probability Redistribution (IPR) to ensure precise localization. iv) Primary results show that MIVE outperforms state-of-the-art methods in multi-instance video editing, achieving a Cross-Instance Accuracy (CIA) Score of 0.7100 in evaluations. v) For AI practitioners, MIVE provides a framework for performing precise, multi-instance video edits without requiring additional training, enabling more efficient and accurate video editing applications. |
Papers for 2024-12-17
| Title | Authors | Summary |
|---|---|---|
| RetroLLM: Empowering Large Language Models to Retrieve Fine-grained Evidence within Generation (Read more on arXiv or HuggingFace) | douzc, Benen2024, wuyongkang, jinjiajie, lixiaoxi45 | Here is a concise summary of the research paper “RetroLLM: Empowering Large Language Models to Retrieve Fine-grained Evidence within Generation” based on the provided guidelines: i) Summary: RetroLLM is a unified framework that integrates retrieval and generation into a single process, enabling large language models (LLMs) to directly generate fine-grained evidence from a corpus during the generation process using constrained decoding. ii) Main Research Question/Objective: How to address the limitations of existing retrieval-augmented generation (RAG) methods, such as the need for separate retrievers, redundant input tokens, and the lack of joint optimization of retrieval and generation. iii) Key Methodology: The authors propose hierarchical FM-Index constraints and a forward-looking constrained decoding strategy to guide the LLM in generating corpus-constrained clues and relevant evidence. iv) Primary Results: RetroLLM outperforms RAG methods across both in-domain and out-of-domain tasks; for example, RetroLLM achieves an accuracy of 61.6% on the NQ dataset, compared to 52.4% for the Naive RAG method. v) Principal Implication for AI Practitioners: AI practitioners can leverage RetroLLM to develop more efficient and accurate RAG systems by eliminating the need for separate retrievers and enabling joint optimization of retrieval and generation, leading to improved performance in knowledge-intensive tasks. |
| Evaluation Agent: Efficient and Promptable Evaluation Framework for Visual Generative Models (Read more on arXiv or HuggingFace) | Yu Qiao, liuziwei7, Ziqi, shulin16, Fan-s | Here is a concise summary of the research paper “Evaluation Agent: Efficient and Promptable Evaluation Framework for Visual Generative Models”: i) The paper introduces Evaluation Agent, a framework for efficiently evaluating visual generative models using dynamic, multi-round assessments tailored to user-specified criteria. ii) The main research objective is to develop an evaluation framework that overcomes the limitations of existing methods by efficiently assessing visual generative models’ capabilities based on user needs and providing detailed, interpretable results. iii) The key methodology employs Large Language Model (LLM)-based agents in a two-stage process: a proposal stage for planning and prompt generation, and an execution stage for sampling and evaluating visual content using an extensible toolkit. iv) The primary result is that Evaluation Agent reduces evaluation time to 10% of traditional methods while achieving comparable accuracy to standard benchmarks like VBench and T2I-CompBench. v) The principal implication for AI practitioners is that they can leverage Evaluation Agent to conduct faster, more flexible, and user-specific evaluations of visual generative models, facilitating more targeted development and refinement. |
| BrushEdit: All-In-One Image Inpainting and Editing (Read more on arXiv or HuggingFace) | yshan2u, ZyZcuhk, juxuan27, BianYx, Yw22 | Here is a concise summary of the BrushEdit research paper, strictly adhering to your guidelines: i) BrushEdit is a novel framework for inpainting-based, instruction-guided image editing that integrates multimodal large language models (MLLMs) and a dual-branch image inpainting model. ii) The main research objective is to develop a new image editing paradigm that overcomes challenges related to inference efficiency, scalable data curation, editability, and controllability in existing methods. iii) The key methodology involves a four-step process: editing category classification, primary editing object identification, acquisition of editing mask and target caption via MLLMs and detection models, and image inpainting using a dual-branch model (BrushNet). iv) Primary results demonstrate that BrushEdit achieves superior performance across seven metrics, including a PSNR score of 32.16 for background preservation in edited images, which is the best result compared to other methods. v) The principal implication for AI practitioners is that BrushEdit provides a user-friendly, free-form, multi-turn interactive framework for instruction-based image editing, enabling more precise control and superior editing quality without the need for extensive training. |
| ColorFlow: Retrieval-Augmented Image Sequence Colorization (Read more on arXiv or HuggingFace) | Yong Liu, yshan2u, ZyZcuhk, juxuan27, JunhaoZhuang | Here is a concise summary of the research paper “ColorFlow: Retrieval-Augmented Image Sequence Colorization”: i) The paper introduces ColorFlow, a novel three-stage diffusion-based framework for reference-based colorization of black-and-white image sequences that preserves object and character identity. ii) The main research objective is to develop a method for automatic image sequence colorization that maintains color consistency and identity preservation across frames, using a pool of color reference images. iii) The key methodology involves a three-stage pipeline: Retrieval-Augmented Pipeline (RAP) for extracting relevant color patches, In-context Colorization Pipeline (ICP) for performing colorization with a two-branch design using a self-attention mechanism, and Guided Super-Resolution Pipeline (GSRP) for upsampling to high-resolution images. iv) ColorFlow outperforms existing models across multiple metrics, achieving over 37% reduction in FID score compared to state-of-the-art colorization models. v) For AI practitioners, ColorFlow offers a robust framework for high-quality, reference-based image sequence colorization, setting a new standard with the potential for direct industrial application in fields such as manga and animation production. |
| Byte Latent Transformer: Patches Scale Better Than Tokens (Read more on arXiv or HuggingFace) | spermwhale, Chunting, marg33, benjamin-mlr, artidoro | Here’s a concise summary of the AI research paper “Byte Latent Transformer: Patches Scale Better Than Tokens”: i) Summary: This paper introduces the Byte Latent Transformer (BLT), a new byte-level language model architecture that dynamically groups bytes into patches to improve efficiency and robustness compared to tokenization-based models. ii) Main research question/objective: How can a byte-level language model be designed to match the performance of tokenization-based models at scale while improving inference efficiency and robustness? iii) Key methodology: BLT uses a dynamic, learnable method for grouping bytes into patches based on next-byte entropy and a new model architecture that mixes byte and patch information processed by local and global transformer blocks. iv) Primary results: BLT models match training FLOP-controlled performance of Llama 3 up to 8B parameters and achieve up to 50% inference FLOP savings; a BLT-Entropy model outperforms the Llama 3 tokenizer-based model on 4 out of 7 tasks while trained on the same amount of data. v) Principal implication for AI practitioners: BLT demonstrates that dynamically allocating compute based on input complexity via patching can lead to more efficient and robust language models, offering a viable alternative to tokenization-based models. |
| Causal Diffusion Transformers for Generative Modeling (Read more on arXiv or HuggingFace) | Haoqi Fan, Shi Guan, Deyao Zh, Chaorui Deng, Andy1621 | Here’s a concise summary of the research paper “Causal Diffusion Transformers for Generative Modeling”: i) Summary: This paper introduces CausalFusion, a decoder-only transformer that unifies autoregressive (AR) and diffusion models for generative modeling by factorizing data across both sequential tokens and diffusion noise levels. ii) Main research question or objective: How can sequential factorization be introduced to a diffusion model to improve its performance and enable a smooth transition between AR and diffusion generation modes? iii) Key methodology: The authors propose a dual-factorization approach in a decoder-only transformer that processes data across sequential tokens and diffusion noise levels, with adjustable AR and diffusion steps, and introduce a generalized causal attention mechanism. iv) Primary results: CausalFusion achieves state-of-the-art results on the ImageNet class-conditional generation benchmark; for instance, CausalFusion-XL achieves a FID-50k score of 1.77 on 256x256 images with classifier-free guidance. v) Principal implication for AI practitioners: AI practitioners can leverage CausalFusion as a powerful and versatile generative modeling framework that combines the strengths of AR and diffusion models, offering improved performance and flexibility for tasks like image generation, multimodal modeling, and zero-shot image manipulation. |
| Smaller Language Models Are Better Instruction Evolvers (Read more on arXiv or HuggingFace) | Hua Zhou, Yaqi Zhang, Lulu Zhao, dongguanting, Chaox72 | Here is a concise summary of the research paper “Smaller Language Models Are Better Instruction Evolvers”: i) Summary: This study investigates the efficacy of smaller language models (SLMs) in evolving instructions for large language models (LLMs) compared to larger models, challenging the notion that larger models inherently possess superior instruction evolution capabilities. ii) Main research question/objective: Do SLMs outperform LLMs in evolving instructions, and if so, why? iii) Key methodology: The authors conducted experiments across three instruction evolution scenarios (Evol-Instruct, AutoIF, and Auto Evol-Instruct) using SLMs and LLMs from the Llama-3 and Qwen-2 families and evaluated performance on various benchmarks, including IFEval and FollowBench. iv) Primary results: SLMs can synthesize more effective and diverse instructions than LLMs; specifically, on the FollowBench benchmark, SLM-evolved instructions (SLM-INST) achieved nearly a 10% improvement over Llama-3-8B and Llama-3.1-8B when supervised by Llama-3.1-70B-Instruct. v) Principal implication for AI practitioners: AI practitioners can leverage SLMs to generate more complex and diverse instructions for instruction tuning, potentially leading to more capable LLMs while using fewer computational resources. |
| IDArb: Intrinsic Decomposition for Arbitrary Number of Input Views and Illuminations (Read more on arXiv or HuggingFace) | Jiaqiwang, Dubhe-zmc, jingtan, tongwu2020, lizb6626 | Here is a concise summary of the research paper “IDArb: Intrinsic Decomposition for Arbitrary Number of Input Views and Illuminations”: i) Summary IDArb is a diffusion-based model for intrinsic decomposition of an arbitrary number of images under varying illuminations, achieving multi-view consistency and disentangling intrinsic components from lighting effects. ii) Main research question or objective The main objective is to develop a model that can perform accurate and multi-view consistent intrinsic decomposition (surface normals, albedo, roughness, metallic) on an arbitrary number of images captured under varying, unconstrained illuminations. iii) Key methodology used The proposed method, IDArb, utilizes a diffusion-based model with a cross-view, cross-component attention module and an illumination-augmented, view-adaptive training strategy, trained on a new dataset (ARB-Objaverse) containing 5.7M multi-view RGB images. iv) Primary results IDArb outperforms state-of-the-art methods in intrinsic decomposition, achieving a PSNR of 33.62 for albedo estimation in multi-view settings. v) Principal implication for AI practitioners IDArb provides a unified solution for inverse rendering across different input regimes, offering AI practitioners a robust method for generating accurate intrinsic components from arbitrary image sets, directly applicable in tasks like relighting, photometric stereo, and 3D reconstruction. |
| SPaR: Self-Play with Tree-Search Refinement to Improve Instruction-Following in Large Language Models (Read more on arXiv or HuggingFace) | howang, yuxiaod, lrxl, wangcunxiang, CCCCCC | Here’s a summary of the paper “SPAR: SELF-PLAY WITH TREE-SEARCH REFINEMENT TO IMPROVE INSTRUCTION-FOLLOWING IN LARGE LANGUAGE MODELS” following your guidelines: i) Summary: This paper introduces SPAR, a self-play framework that uses tree-search refinement to improve instruction-following in large language models (LLMs) by creating better preference pairs. ii) Main research question/objective: How to improve the instruction-following capabilities of LLMs using a self-play framework that addresses limitations of existing preference learning methods. iii) Key methodology: SPAR employs a self-play framework where an LLM acts as both an actor and a refiner, using a tree-search algorithm to refine responses and generate valid preference pairs for training. iv) Primary results: After three iterations, SPAR improved a LLaMA3-8B-Instruct model to surpass GPT-4-Turbo on the IFEval benchmark, achieving an average accuracy of 81.8. v) Principal implication for AI practitioners: AI practitioners can use SPAR to enhance the instruction-following abilities of LLMs without relying on external models, enabling the development of more accurate and reliable AI systems. |
| Wonderland: Navigating 3D Scenes from a Single Image (Read more on arXiv or HuggingFace) | Hanwen Liang, ZanyRumata, guochengqian, vidit98, jlcao2 | Here is a concise summary of the research paper “Wonderland: Navigating 3D Scenes from a Single Image”: i) Wonderland is a novel framework for efficiently generating high-quality, wide-scope 3D scenes from a single image using a feed-forward reconstruction model operating on the latent space of a video diffusion model. ii) Main research question: How can we efficiently create high-quality, wide-scope 3D scenes from a single arbitrary image? iii) Key methodology: A large-scale reconstruction model uses latents from a camera-guided video diffusion model to predict 3D Gaussian Splattings in a feed-forward manner, with a dual-branch camera conditioning module for precise pose control and a progressive training strategy. iv) Primary results: The method significantly outperforms existing methods for single-view 3D scene generation, achieving a FID score of 16.16 on the RealEstate10K dataset, compared to 20.89 for the next best method, ViewCrafter. v) Principal implication for AI practitioners: Wonderland demonstrates that a 3D reconstruction model can be effectively built upon the latent space of a diffusion model to realize efficient 3D scene generation, providing a novel and effective approach to single image 3D scene generation. |
| GaussianProperty: Integrating Physical Properties to 3D Gaussians with LMMs (Read more on arXiv or HuggingFace) | junweiliang, StarYDY, zhifeichen097, spongy, Xxlbigbrother | Here is a concise summary of the research paper “GaussianProperty: Integrating Physical Properties to 3D Gaussians with LMMs”: i) Summary: This paper introduces GaussianProperty, a training-free framework that leverages Large Multimodal Models (LMMs) to assign physical properties to 3D Gaussian representations for applications in physics-based simulation and robotic grasping. ii) Main research question/objective: The main objective is to develop a method for accurately estimating and integrating physical properties of materials into 3D Gaussian representations from multi-view 2D images. iii) Key methodology: The methodology combines global-local physical property reasoning using Segment Anything (SAM) for image segmentation and GPT-4V for property recognition, followed by a multi-view projection and voting strategy to assign properties to 3D Gaussians. iv) Primary results: The proposed method achieved a material segmentation mean Intersection over Union (mIoU) of 55.83% on the ABO dataset, demonstrating the effective integration of physical properties into 3D Gaussian representations. v) Principal implication for AI practitioners: AI practitioners can leverage this method to enhance 3D models with physical properties without the need for manual annotation, enabling more realistic physics-based simulations and improved robotic grasping strategies directly from visual data. |
| SepLLM: Accelerate Large Language Models by Compressing One Segment into One Separator (Read more on arXiv or HuggingFace) | Xiaozhe Ren, Yihang Gao, Jiawei Li, Guoxuan Chen, shihan96 | Here is a concise summary of the research paper “SepLLM: Accelerating Large Language Models by Compressing One Segment into One Separator”: i) Summary: This paper introduces SepLLM, a novel framework that accelerates large language models (LLMs) by compressing segments of text into separator tokens within a sparse attention mechanism. ii) Main research question/objective: The main objective is to accelerate LLM inference and training by addressing the quadratic complexity of self-attention through a data-dependent sparse attention mechanism. iii) Key methodology: The key methodology involves identifying and leveraging the disproportionate attention scores of separator tokens to condense segment information, implementing a sparse attention mechanism that retains only initial, neighboring, and separator tokens, and utilizing efficient kernels for training acceleration. iv) Primary results: SepLLM achieves over 50% reduction in KV cache usage on the GSM8K-CoT benchmark using the Llama-3-8B backbone while maintaining comparable performance to the original model. v) Principal implication for AI practitioners: AI practitioners can leverage SepLLM as a plug-and-play framework to accelerate the inference and training of LLMs, particularly in streaming settings with long sequences, without significant loss of performance, by strategically managing and compressing the KV cache. |
| Wonderful Matrices: Combining for a More Efficient and Effective Foundation Model Architecture (Read more on arXiv or HuggingFace) | wubingheng, JingzeShi | Here is a concise summary of the paper “Wonderful Matrices: Combining for a More Efficient and Effective Foundation Model Architecture”: i) The paper introduces “Wonderful Matrices,” a novel foundation model architecture that integrates sequence and state transformations to enhance efficiency and effectiveness. ii) The main research objective is to develop a foundation model architecture that combines the strengths of State Space Duality and Quadratic Causal Self-Attention algorithms while mitigating their respective limitations. iii) The key methodology involves unifying position encoding with Rotary Position Embedding, introducing Dynamic Mask Attention for selective information filtering, and designing Cross Domain Mixture of Experts for efficient parameter utilization. iv) Primary results show that Dynamic Mask Attention maintains 100% accuracy in the multi-query associative recall task, outperforming Quadratic Causal Self-Attention and State Space Duality. v) The principal implication for AI practitioners is that Wonderful Matrices provides a more efficient and effective architecture for language modeling, as demonstrated by improved performance on benchmark tasks. |
| StrandHead: Text to Strand-Disentangled 3D Head Avatars Using Hair Geometric Priors (Read more on arXiv or HuggingFace) | Jian Yang, Zeyu Cai, yingtai, JesseZhang, XiaokunSun | Here is a concise summary of the research paper “StrandHead: Text to Strand-Disentangled 3D Head Avatars Using Hair Geometric Priors”: i) StrandHead is a novel framework that generates 3D head avatars with strand-disentangled hair from text descriptions without using 3D hair data for supervision. ii) The main research objective is to develop a method for generating realistic 3D head avatars with detailed, strand-based hair directly from text prompts. iii) The key methodology involves distilling 2D generative diffusion models, using a differentiable prismatization algorithm to convert hair strands into meshes, and applying orientation consistency and curvature regularization losses based on hair geometric priors. iv) Primary results show that StrandHead outperforms state-of-the-art methods in head and hair generation; for example, it achieved a 58.00% Text-Image Alignment Preference (TAP) score in head generation tasks. v) The principal implication for AI practitioners is that StrandHead provides a new, effective way to generate high-fidelity 3D head avatars with realistic hair from text descriptions, which can be directly integrated into existing simulation and rendering systems without requiring 3D hair data. |
| MOVIS: Enhancing Multi-Object Novel View Synthesis for Indoor Scenes (Read more on arXiv or HuggingFace) | YuLiu, BuzzBeater, JunfengNi, YixinChen, JasonAplp | Here is a concise summary of the research paper “MOVIS: Enhancing Multi-Object Novel View Synthesis for Indoor Scenes”: i) Summary: This paper introduces MOVIS, a novel method designed to improve the structural awareness and cross-view consistency of diffusion-based novel view synthesis (NVS) models for multi-object indoor scenes. ii) Main research question or objective: How can the structural awareness of current diffusion-based novel view synthesizers be enhanced to improve cross-view consistency in multi-object scenarios? iii) Key methodology: MOVIS incorporates structure-aware features (depth and object mask) as inputs, employs an auxiliary novel view mask prediction task, and utilizes a structure-guided timestep sampling scheduler during training. iv) Primary results: MOVIS outperforms existing methods on multi-object NVS tasks, demonstrating superior object placement, geometry, and appearance recovery; quantitatively, MOVIS achieves a PSNR of 17.432 on the C3DFS test set, compared to 14.811 for the next best method, Zero-1-to-3+. v) Principal implication for AI practitioners: MOVIS provides AI practitioners with a method to generate more consistent and realistic novel views in complex multi-object scenes by enhancing the structural awareness of diffusion models, making them more viable for real-world applications like AR/VR and robotics. |
| Whisper-GPT: A Hybrid Representation Audio Large Language Model (Read more on arXiv or HuggingFace) | prateekv | Here’s a summary of the research paper “WHISPER-GPT: A Hybrid Representation Audio Large Language Model” following the specified guidelines: i) Summary: This paper introduces WHISPER-GPT, a generative large language model (LLM) for speech and music that combines continuous audio representations (mel-spectrogram) with discrete acoustic tokens (ENCODEC) in a hybrid architecture. ii) Main research question or objective: Can an architecture that simultaneously utilizes continuous and discrete representation in the LLM setup improve the next token prediction compared to a token-based LLM for speech and music? iii) Key methodology used: The authors adapted a Whisper-like encoder-decoder architecture to a seq-to-seq model for generative modeling, replacing the Whisper encoder with a decoder and performing early fusion of learned representations with decoder-only architecture on acoustic tokens. They also employed a Transformer decoder-only architecture trained on the LibriSpeech TTS dataset and a dataset of instrumental music to predict the next coarse acoustic token. iv) Primary results: The hybrid model outperformed a purely token-based GPT model in next token prediction. Specifically, for the music dataset, the hybrid model achieved a negative log-likelihood (NLL) of 2.52 compared to 2.78 for the baseline GPT-S model. v) Principal implication for AI practitioners: AI/ML/Software Engineers and Data Scientists can leverage this hybrid input representation approach to achieve better performance in generative audio models, potentially enabling smaller, more efficient models with performance comparable to larger, purely token-based models. |
| TidyBot++: An Open-Source Holonomic Mobile Manipulator for Robot Learning (Read more on arXiv or HuggingFace) | Yihuai Gao, Aaditya Prasad, Robert Holmberg, William Chong, jimmyyhwu | Here is a concise summary of the research paper “TidyBot++: An Open-Source Holonomic Mobile Manipulator for Robot Learning”: i) Summary: This paper introduces TidyBot++, an open-source holonomic mobile manipulator designed for robot learning, featuring a powered-caster mobile base and a mobile phone teleoperation interface. ii) Main research question/objective: The main objective is to develop an inexpensive, robust, and flexible holonomic mobile manipulator to facilitate the collection of large-scale demonstration data for mobile manipulation tasks. iii) Key methodology: The key methodology involves designing a holonomic base using powered casters, developing a mobile phone teleoperation interface using the WebXR API, and training diffusion policies with collected demonstration data. iv) Primary results: The researchers successfully trained policies for six household tasks, with the open fridge task achieving a 10/10 success rate in policy rollouts. v) Principal implication for AI practitioners: This open-source design and teleoperation interface can enable AI practitioners to easily collect mobile manipulation data and develop policies for real-world applications, significantly lowering the barrier to entry for mobile manipulation research. |
| Just a Simple Transformation is Enough for Data Protection in Vertical Federated Learning (Read more on arXiv or HuggingFace) | Aleksandr Beznosikov, Philip Zmushko, pichuginad, Andron00e | Here is a concise summary of the research paper “Just a Simple Transformation is Enough for Data Protection in Vertical Federated Learning”: i) This paper investigates data protection in Vertical Federated Learning (VFL) against feature reconstruction attacks, focusing on the impact of model architecture. ii) The main research objective is to determine whether Multi-Layer Perceptron (MLP)-based models are more resistant to feature reconstruction attacks than Convolutional Neural Network (CNN)-based models in VFL. iii) The key methodology involves theoretical analysis of orthogonal transformations on data and weights in VFL, and empirical evaluation of state-of-the-art Model Inversion and Feature-space Hijacking attacks on various datasets using MLP and CNN architectures. iv) The primary results show that MLP-based models, unlike CNN-based models, are resistant to UnSplit and Feature-space Hijacking attacks; for instance, the Feature-space Hijacking attack on MNIST with a CNN-based model achieved a reconstruction error of 0.25, while on an MLP-based model, the error was 0.8. v) The principal implication for AI practitioners is that using MLP architectures in VFL can enhance data protection against feature reconstruction attacks without requiring additional defense mechanisms, although they might provide less utility compared to CNNs on image datasets. |
Papers for 2024-12-16
| Title | Authors | Summary |
|---|---|---|
| GenEx: Generating an Explorable World (Read more on arXiv or HuggingFace) | danyaljj, jiahaoplus, lambertxiao, tshu, TaiMingLu | Here’s a summary of the research paper “GenEx: Generating an Explorable World” following your guidelines: 1. Summary: GenEx is a system that generates explorable, 3D-consistent virtual worlds from a single RGB image, enabling embodied AI agents to navigate and interact within these generated environments. 2. Main research question/objective: How can an agent make more informed decisions through exploration in a generative 360° world? 3. Key methodology: GenEx employs a physics-based data engine to create panoramic video streams representing 360° environments, uses GPT-assisted agents for exploration, and implements an imagination-augmented policy for decision-making. 4. Primary results: GenEx achieves high-quality world generation, with its earlier version demonstrating a PSNR of 30.2 and SSIM of 0.94 in video quality metrics. 5. Principal implication for AI practitioners: GenEx provides a platform for AI practitioners to develop and evaluate embodied AI agents in realistic, dynamically generated environments, enabling advancements in areas such as navigation, interactive gaming, and VR/AR. |
| Apollo: An Exploration of Video Understanding in Large Multimodal Models (Read more on arXiv or HuggingFace) | minione, lichengyu, YannDubs, nicholswang, orrzohar | This paper explores design choices impacting video understanding in Large Multimodal Models (LMMs). The research investigates how various architectural and training decisions affect video-LMM performance. A combination of controlled experiments on smaller models (demonstrating “Scaling Consistency”) and large-scale training was used, leading to the development of the Apollo family of models. Apollo-3B achieved a score of 68.7 on the MLVU benchmark, outperforming most existing 7B models. This work suggests AI practitioners can leverage Scaling Consistency to perform efficient experimentation on smaller models before scaling up, thereby saving computational resources and accelerating the development of high-performing video-LMMs. |
| BiMediX2: Bio-Medical EXpert LMM for Diverse Medical Modalities (Read more on arXiv or HuggingFace) | Saeed Yahya Alseiari, Mohammed Irfan Kurpath, hishamcholakkal, HuggingSara, sahalshajim | Here is a concise summary of the research paper “BiMediX2: Bio-Medical EXpert LMM for Diverse Medical Modalities” based on your specified format: i) Summary: BiMediX2 is a bilingual Arabic-English Large Multimodal Model (LMM) designed for advanced medical image understanding and text-based interactions, leveraging the Llama3.1 architecture. ii) Main research question or objective: To develop a unified bilingual (Arabic-English) multimodal AI model that excels in both medical image understanding and text-based medical tasks. iii) Key methodology used: The model was trained on a 1.6M sample bilingual healthcare dataset, utilizing a Vision Encoder, a Projector for image-text alignment, and LoRA adapters for fine-tuning the Llama 3.1 language model. iv) Primary results: BiMediX2 achieved state-of-the-art performance on several medical benchmarks, outperforming GPT-4 by over 9% in UPHILL factual accuracy evaluations. v) Principal implication for AI practitioners: AI practitioners can leverage BiMediX2’s unified architecture and training methodology to develop advanced, multilingual medical AI systems capable of handling diverse modalities and achieving high accuracy in both image and text-based tasks without compromising the advanced text based medical understanding of LLMs. |
| InstanceCap: Improving Text-to-Video Generation via Instance-aware Structured Caption (Read more on arXiv or HuggingFace) | BradyFU, zhenheny, SherryX, nankepan, AnonMegumi | Here’s a summary of the paper “InstanceCap: Improving Text-to-Video Generation via Instance-aware Structured Caption” based on your specifications: i) This paper introduces InstanceCap, a novel instance-aware structured captioning framework for text-to-video generation, enhancing video fidelity and consistency. ii) The main research objective is to develop a method for generating detailed, instance-level video captions that improve the accuracy and fidelity of text-to-video generation models. iii) The key methodology involves an Auxiliary Models Cluster (AMC) to isolate video instances and an improved Chain-of-Thought (CoT) process with Multimodal Large Language Models (MLLMs) to refine dense prompts into structured phrases. iv) Primary results show that InstanceCap significantly outperforms previous models, with finetuned models achieving a 37.88% average metric in a specific quantitative evaluation (Table 2). v) For AI practitioners, InstanceCap provides a method to enhance the fidelity of text-to-video models by utilizing detailed, structured captions, enabling the generation of videos with accurate instance details and motion actions. |
| Large Action Models: From Inception to Implementation (Read more on arXiv or HuggingFace) | Eliblo1969, substill, shilhe, Lujunting, vyokky | This paper introduces Large Action Models (LAMs), designed to perform actions in digital and physical environments. The objective is to develop a framework for creating LAMs, transitioning from Large Language Models (LLMs) limited to textual output, focusing on action generation and execution within dynamic environments. A four-phase training approach is employed, encompassing task-plan pretraining, expert imitation, self-boosting exploration, and reward model-based optimization, using a Windows OS-based GUI agent as a case study. The developed LAM achieved a Task Success Rate (TSR) of 81.2% in offline evaluation on Word tasks, surpassing the 67.2% TSR of GPT-40. This demonstrates the effectiveness of specialized training for action-oriented tasks and provides a practical workflow for AI practitioners developing agents capable of interacting with and manipulating real-world environments through actions rather than just text. |
| FreeScale: Unleashing the Resolution of Diffusion Models via Tuning-Free Scale Fusion (Read more on arXiv or HuggingFace) | JacobYuan, Ruihang, weilllllls, StevenZhang, MoonQiu | Here is a concise summary of the research paper “FreeScale: Unleashing the Resolution of Diffusion Models via Tuning-Free Scale Fusion”: i) Summary: This paper introduces FreeScale, a tuning-free inference paradigm that enhances the resolution of pre-trained diffusion models for image and video generation via scale fusion. ii) Main Research Objective: The main research objective is to enable pre-trained diffusion models to generate high-fidelity, high-resolution visual content without requiring additional training or fine-tuning. iii) Key Methodology: FreeScale employs tailored self-cascade upscaling, restrained dilated convolution, and scale fusion, which processes and fuses information from different receptive scales by extracting desired frequency components within the self-attention layers. iv) Primary Results: FreeScale successfully generates 8K-resolution images and outperforms existing methods; for example, when generating 4096x4096 images, it achieves a FID score of 49.796, compared to 72.378 for DemoFusion. v) Principal Implication: AI practitioners can use FreeScale to extend the capabilities of existing diffusion models to generate higher-resolution images and videos without the need for model retraining, offering a practical solution for high-resolution visual content creation. |
| ObjectMate: A Recurrence Prior for Object Insertion and Subject-Driven Generation (Read more on arXiv or HuggingFace) | Dana Berman, Matan Cohen, Asaf Shul, yedid, danielwinter | Here’s a concise summary of the research paper “ObjectMate: A Recurrence Prior for Object Insertion and Subject-Driven Generation” : i) Summary: This paper introduces ObjectMate, a tuning-free method for photorealistic object insertion and subject-driven generation using a recurrence prior over large unlabeled datasets. ii) Main research question/objective: How to achieve photorealistic object composition into a scene while preserving the object’s identity without requiring test-time tuning. iii) Key methodology: ObjectMate leverages a recurrence prior to create a supervised dataset from mass-produced objects across multiple images, then trains a text-to-image diffusion architecture to map object and scene descriptions to a composited image. iv) Primary results: ObjectMate demonstrates superior identity preservation and photorealistic composition compared to state-of-the-art methods in both object insertion and subject-driven generation; users preferred ObjectMate’s composition over ObjectDrop’s 76% of the time. v) Principal implication for AI practitioners: AI practitioners can use the recurrence prior, which exploits the natural repetition of objects in large-scale datasets, to build more powerful and efficient models for object insertion and subject-driven generation, without the need for test-time fine-tuning or manual data collection. |
| FireFlow: Fast Inversion of Rectified Flow for Image Semantic Editing (Read more on arXiv or HuggingFace) | Fan Tang, Changwang Mei, duke1852022, MagicBag, yingying87 | Here is a concise summary of the research paper “FireFlow: Fast Inversion of Rectified Flow for Image Semantic Editing”: i) This paper introduces FireFlow, a novel zero-shot method for fast inversion and semantic editing of images using Rectified Flow (ReFlow) models. ii) Main research question/objective: How to achieve accurate and efficient inversion and editing in ReFlow-based generative models, specifically within 8 steps. iii) Key methodology: A new numerical solver is proposed that achieves second-order precision while maintaining the computational cost of a first-order Euler method by reusing intermediate velocity approximations. iv) Primary results: FireFlow achieves a 3x runtime speedup compared to state-of-the-art ReFlow inversion techniques, with a reconstruction error of 0.1579 in the proposed method compared to 0.2926 for the next best performing method (RF-Solver). v) Principal implication for AI practitioners: AI practitioners can leverage FireFlow for faster and more accurate image inversion and editing using ReFlow models, enabling more efficient development of applications requiring fine-grained control over image generation. |
| Multimodal Music Generation with Explicit Bridges and Retrieval Augmentation (Read more on arXiv or HuggingFace) | morninghaze, baochenxi, wzk1015, JackyZhuo, wbs2788 | Here is a concise summary of the research paper “Multimodal Music Generation with Explicit Bridges and Retrieval Augmentation”: i) Summary: This paper introduces VMB, a novel multimodal music generation framework that utilizes text and music as explicit bridges for aligning and generating music from various input modalities. ii) Main research question/objective: The main objective is to address challenges in multimodal music generation such as data scarcity, weak cross-modal alignment, and limited controllability. iii) Key methodology: The key methodology involves a Multimodal Music Description Model to create text bridges, a Dual-track Music Retrieval module to provide music bridges, and an Explicitly Conditioned Music Generation framework based on a diffusion transformer. iv) Primary results: VMB achieved a KLpasst score of 48.84 on the SymMV dataset for video-to-music generation, outperforming existing methods. v) Principal implication for AI practitioners: AI practitioners can leverage VMB’s explicit text and music bridges to improve the quality, alignment, and controllability of multimodal music generation models, which could be applied in areas like automatic video soundtrack creation. |
| SynerGen-VL: Towards Synergistic Image Understanding and Generation with Vision Experts and Token Folding (Read more on arXiv or HuggingFace) | wzk1015, Einsiedler, hehesang, Changyao, cpsxhao | Here is a concise summary of the research paper “SynerGen-VL: Towards Synergistic Image Understanding and Generation with Vision Experts and Token Folding”: i) SynerGen-VL is an encoder-free Multimodal Large Language Model (MLLM) that integrates image understanding and generation capabilities using vision experts and token folding. ii) The main research objective is to develop a unified MLLM that simplifies the model architecture and training pipeline while effectively supporting high-resolution image understanding and generation. iii) Key methodologies include a token folding mechanism to reduce visual token sequence length, a vision-expert-based progressive alignment pretraining strategy, and a unified next-token prediction objective for both image understanding and generation. iv) Primary results show that SynerGen-VL achieves competitive performance; for instance, with only 2.4B activated parameters, it achieves a Multi-Modal Massive Multitask Understanding (MMMU) score of 34.2, comparable to existing encoder-free unified MLLMs with larger parameter sizes. v) For AI practitioners, SynerGen-VL offers a simplified and scalable approach to building unified MLLMs, potentially streamlining development by eliminating the need for separate encoders or complex training objectives for image understanding and generation tasks. |
| SCBench: A KV Cache-Centric Analysis of Long-Context Methods (Read more on arXiv or HuggingFace) | Chengruidong, luoxufang, qianhuiwu, iofu728, liyucheng | SCBench benchmarks long-context language models (LLMs) focusing on KV cache usage. The research investigates the performance of long-context methods in scenarios involving KV cache reuse, like multi-turn dialogue. A comprehensive benchmark comprising 12 tasks across four long-context abilities (string retrieval, semantic retrieval, global information processing, and multi-tasking) was created. MInference, a dynamic sparse attention method, shows superior performance in shared context and multi-turn scenarios, particularly in retrieval tasks, achieving up to 51.2% accuracy. AI practitioners can leverage these insights to choose efficient long-context methods based on task needs, especially in dynamic conversational applications, focusing on strategies that maintain or dynamically compress KV cache for optimal performance. |
| FluxSpace: Disentangled Semantic Editing in Rectified Flow Transformers (Read more on arXiv or HuggingFace) | Pinar Yanardag, Kavana Venkatesh, ydalva | Here is a concise summary of the research paper “FluxSpace: Disentangled Semantic Editing in Rectified Flow Transformers”: i) Summary: The paper introduces FluxSpace, a novel method for performing disentangled semantic editing on images generated by rectified flow transformers. ii) Main research question/objective: To develop a domain-agnostic image editing method that allows for precise, attribute-specific modifications without affecting unrelated aspects of the image in rectified flow models. iii) Key methodology: FluxSpace leverages the attention layer outputs within the joint transformer blocks of rectified flow models to create a semantically interpretable representation space, enabling linear editing operations for both fine-grained and coarse-level image modifications. iv) Primary results: FluxSpace achieves disentangled image editing, outperforming existing methods in quantitative evaluations; for instance, it achieved a CLIP-I score of 0.9417 for eyeglass editing, indicating high content preservation. v) Principal implication for AI practitioners: AI practitioners can utilize FluxSpace for precise and disentangled semantic editing of images generated by rectified flow transformers without additional training, offering enhanced control and efficiency in image generation and manipulation tasks. |
| SmolTulu: Higher Learning Rate to Batch Size Ratios Can Lead to Better Reasoning in SLMs (Read more on arXiv or HuggingFace) | SultanR | Here’s a summary of the paper “SmolTulu: Higher Learning Rate to Batch Size Ratios Can Lead to Better Reasoning in SLMs” adhering to your guidelines: i) The paper introduces SmolTulu, a 1.7B parameter instruction-tuned language model that achieves state-of-the-art performance among sub-2B parameter models by adapting the Tulu 3 post-training pipeline. ii) The main research question is how the relationship between learning rate and batch size impacts the performance of small language models (SLMs) during supervised finetuning across different types of tasks. iii) The key methodology involved empirical analysis using a 135M parameter model and a 1.7B parameter model, with ablations of learning rate and batch size during supervised finetuning and direct preference optimization. iv) The primary result is that higher learning rate to batch size ratios improved performance on reasoning tasks, with SmolTulu-DPO-1130 achieving 67.7% on IFEval. v) The principal implication for AI practitioners is that optimal learning rate to batch size ratios for SLMs may differ significantly from larger models and are task-dependent, necessitating careful tuning for optimal performance in different applications. |
| Prompt2Perturb (P2P): Text-Guided Diffusion-Based Adversarial Attacks on Breast Ultrasound Images (Read more on arXiv or HuggingFace) | Ilker Hacihaliloglu, Leonid Sigal, Clayton Allard, moein99, yasimed | Here is a summary of the research paper “Prompt2Perturb (P2P): Text-Guided Diffusion-Based Adversarial Attacks on Breast Ultrasound Images”: i) The paper introduces Prompt2Perturb (P2P), a novel method for generating text-guided adversarial attacks on breast ultrasound images using diffusion models without retraining. ii) Main research question/objective: How can adversarial examples be generated for breast ultrasound images using text prompts, bypassing the need for retraining diffusion models and ensuring clinical relevance? iii) Key methodology: P2P leverages learnable prompts within a frozen text encoder to directly update text embeddings, optimizing only the early reverse diffusion steps to create subtle yet impactful perturbations guided by text instructions. iv) Primary results: P2P achieved a 98% attack success rate on the DenseNet121 model using the BUSI dataset, while maintaining low LPIPS (0.13) and FID (45.84) scores, indicating high visual quality and stealthiness. v) Principal implication for AI practitioners: AI practitioners can use P2P to generate effective and stealthy adversarial attacks on medical imaging models using only text prompts, highlighting potential vulnerabilities in these systems without requiring extensive data or model retraining. |
Papers for 2024-12-13
| Title | Authors | Summary |
|---|---|---|
| InternLM-XComposer2.5-OmniLive: A Comprehensive Multimodal System for Long-term Streaming Video and Audio Interactions (Read more on arXiv or HuggingFace) | Rui Qian, Yuhang Zang, Yuhang Cao, Xiaoyi Dong, Pan Zhang | Here is a concise summary of the research paper “InternLM-XComposer2.5-OmniLive: A Comprehensive Multimodal System for Long-term Streaming Video and Audio Interactions”: i) Summary: The paper introduces InternLM-XComposer2.5-OmniLive (IXC2.5-OL), a multimodal system designed for real-time interaction with streaming video and audio, featuring disentangled perception, memory, and reasoning modules. ii) Main research question/objective: The main objective is to develop an AI system that can continuously process and interact with long-term streaming multimodal (video and audio) inputs and outputs, similar to human cognition. iii) Key methodology: The methodology involves a modular framework with a Streaming Perception Module for real-time multimodal input processing, a Multi-modal Long Memory Module that integrates and compresses short-term and long-term memories, and a Reasoning Module that interacts with the other modules to respond to queries. iv) Primary results: IXC2.5-OL achieves state-of-the-art results among models with less than 10B parameters on the MLVU benchmark, obtaining an M-Avg of 66.2%. v) Principal implication for AI practitioners: AI practitioners can utilize the publicly available IXC2.5-OL framework and models to develop and deploy multimodal AI systems capable of continuous, adaptive interaction with long-term streaming video and audio data, potentially enhancing AI assistants and other real-time applications. |
| Phi-4 Technical Report (Read more on arXiv or HuggingFace) | Ronen Eldan, Sébastien Bubeck, Harkirat Behl, Jyoti Aneja, Marah Abdin | Here is a concise summary of the Phi-4 technical report, strictly following the specified guidelines: 1. Summary: Phi-4 is a 14-billion parameter language model that focuses on data quality, incorporating synthetic data to improve reasoning and problem-solving capabilities beyond its predecessor, the Phi-3. 2. Main research question or objective: The paper does not explicitly state a main research question. The objective is to develop a language model that achieves strong performance relative to its size, particularly on reasoning-focused benchmarks, by optimizing data quality. 3. Key methodology used: The key methodology involves generating high-quality synthetic data through techniques like multi-agent prompting, self-revision, and instruction reversal, combined with curated organic data and optimized training curriculum, as well as innovations in the post-training scheme such as pivotal token search. 4. Primary results: Phi-4 surpasses its teacher model, GPT-4, on STEM-focused QA capabilities, notably scoring 56.1 on the GPQA benchmark compared to GPT-4’s 50.6. 5. Principal implication for AI practitioners: AI practitioners can leverage synthetic data generation and innovative post-training methods detailed in the paper to enhance the reasoning and problem-solving capabilities of smaller language models, achieving performance comparable to or surpassing much larger models. |
| Euclid: Supercharging Multimodal LLMs with Synthetic High-Fidelity Visual Descriptions (Read more on arXiv or HuggingFace) | Willie Neiswanger, Jinyi Hu, Tianyu Yu, Ollie Liu, jrzhang | Here’s a concise summary of the research paper “Euclid: Supercharging Multimodal LLMs with Synthetic High-Fidelity Visual Descriptions”: i) Summary: The paper introduces “Euclid,” a multimodal large language model (MLLM) specifically designed to improve low-level visual perception (LLVP) in geometric tasks using synthetic data. ii) Main research question or objective: How can MLLMs’ ability to accurately perceive and describe geometric details in images be improved? iii) Key methodology: A new benchmark, “Geoperception,” was developed to evaluate MLLMs on 2D geometric perception, and a synthetic data engine was used to create high-fidelity visual descriptions for training a family of models called “Euclid.” The paper also explored various model architectures, training techniques, and data strategies, including a curriculum-based training approach. iv) Primary results: Euclid outperformed the best closed-source model, Gemini-1.5-Pro, by up to 58.56% on certain Geoperception benchmark tasks, demonstrating the effectiveness of using synthetic data and curriculum learning for enhancing geometric perception. v) Principal implication for AI practitioners: AI practitioners can leverage synthetic high-fidelity data and curriculum-based training to enhance MLLMs’ performance on tasks requiring precise low-level visual perception, particularly in domains like geometric reasoning. This is the most impactful finding and offers a way to improve MLLMs on these tasks. |
| Multimodal Latent Language Modeling with Next-Token Diffusion (Read more on arXiv or HuggingFace) | Li Dong, Zhiliang Peng, Wenhui Wang, Hangbo Bao, Yutao Sun | Here is a concise summary of the research paper: i) Summary: The paper introduces Latent Language Modeling (LatentLM), a method that unifies the handling of discrete and continuous data in multimodal generative models using causal Transformers and next-token diffusion. ii) Main Research Question/Objective: How to seamlessly integrate both discrete (e.g., text, code) and continuous data (e.g., image, audio) within a unified multimodal generative model. iii) Key Methodology: LatentLM employs a variational autoencoder (VAE) with a novel σ-VAE to represent continuous data as latent vectors, uses next-token diffusion for autoregressive generation of these vectors, and utilizes causal Transformers for unified processing. iv) Primary Results: LatentLM surpasses Diffusion Transformers in image generation performance and scalability; in image generation tasks on ImageNet, LatentLM achieved a FID score of 2.24. v) Principal Implication for AI Practitioners: AI practitioners can use LatentLM as an effective and scalable approach to develop large multimodal models that unify multimodal generation and understanding with a general-purpose interface. |
| EasyRef: Omni-Generalized Group Image Reference for Diffusion Models via Multimodal LLM (Read more on arXiv or HuggingFace) | Hao Shao, Guanglu Song, Bingqi Ma, Dongzhi Jiang, Zhuofan Zong | Here is a concise summary of the research paper “EasyRef: Omni-Generalized Group Image Reference for Diffusion Models via Multimodal LLM”: i) Summary: This paper introduces EasyRef, a plug-and-play method for conditioning diffusion models on multiple reference images and text prompts using a multimodal large language model (MLLM). ii) Main research question/objective: How to enable diffusion models to effectively capture and utilize consistent visual elements from multiple reference images for personalized image generation. iii) Key methodology: EasyRef leverages an MLLM to encode consistent visual elements from multiple images and text prompts, using an efficient reference aggregation strategy and a progressive training scheme. iv) Primary results: EasyRef outperforms existing methods in multi-reference image generation, achieving a 0.223 higher DINO-I score than IP-Adapter-SDXL in single-image reference experiments on the COCO dataset. v) Principal implication for AI practitioners: AI practitioners can use EasyRef to generate high-fidelity images based on multiple images and text descriptions without the need for model finetuning, representing a significant advancement in controllable image generation. |
| AgentTrek: Agent Trajectory Synthesis via Guiding Replay with Web Tutorials (Read more on arXiv or HuggingFace) | Zhennan Shen, Dunjie Lu, Yiheng Xu, cxiong, ZeonLap | Here is a concise summary of the AgentTrek research paper, strictly following your guidelines: i) Summary: AgentTrek is a scalable pipeline that synthesizes high-quality web agent trajectories by leveraging web tutorials to guide agent actions in a digital environment. ii) Main research question/objective: How to generate high-quality, multi-step trajectory data for training GUI agents without relying on expensive and labor-intensive human annotation. iii) Key methodology: The authors used web tutorials to guide a visual-language model (VLM) agent’s actions in a real digital environment and employed a VLM-based evaluator to ensure trajectory correctness. iv) Primary results: Training GUI agents with synthesized trajectories improved performance; for instance, fine-tuning with the AgentTrek dataset improved Qwen2-VL’s grounding ability on the ScreenSpot benchmark, achieving a score of 67.4. v) Principal implication for AI practitioners: AI practitioners can use AgentTrek as a cost-effective method to generate training data for GUI agents, improving their grounding and planning capabilities without the need for extensive manual annotation. |
| Neural LightRig: Unlocking Accurate Object Normal and Material Estimation with Multi-Light Diffusion (Read more on arXiv or HuggingFace) | Ziwei Liu, Xingang Pan, Xin Huang, Tengfei Wang, Zexin He | Here is a concise summary of the research paper “Neural LightRig: Unlocking Accurate Object Normal and Material Estimation with Multi-Light Diffusion”: i) Summary: Neural LightRig is a framework that utilizes a multi-light diffusion model to enhance the estimation of object geometry and materials from a single image. ii) Main research question or objective: Can a multi-light diffusion model simulate images illuminated by different directional light sources to improve surface normal and material estimation from a single image? iii) Key methodology: The authors developed a multi-light diffusion model to generate multiple consistent images of an object under various lighting conditions. This was achieved by training on a synthetic relighting dataset, followed by training a large G-buffer model using a U-Net architecture to predict surface normals and materials from these multi-light images. iv) Primary results: The method significantly outperforms state-of-the-art methods in surface normal and PBR material estimation. Specifically, the proposed method achieved a mean angular error of 6.413 in surface normal estimation, compared to 8.034 for the next best method, StableNormal. v) Principal implication for AI practitioners: AI practitioners can leverage Neural LightRig to obtain more accurate surface normal and PBR material estimations from single images, enhancing the fidelity of 3D object reconstruction and rendering in applications like computer vision and graphics. |
| SnapGen: Taming High-Resolution Text-to-Image Models for Mobile Devices with Efficient Architectures and Training (Read more on arXiv or HuggingFace) | Arpit Sahni, Huseyin Coskun, Xijie Huang, Jierun Chen, Dongting Hu | Here is a concise summary of the research paper “SnapGen: Taming High-Resolution Text-to-Image Models for Mobile Devices with Efficient Architectures and Training”: i) Summary: This paper introduces SnapGen, a novel text-to-image (T2I) model designed for efficient, high-resolution image generation on mobile devices. ii) Main research question/objective: How can a T2I model be trained from scratch to generate high-quality, high-resolution images on resource-constrained mobile devices? iii) Key methodology: The authors optimize network architecture (UNet and autoencoder), employ multi-level knowledge distillation with timestep-aware scaling from a larger teacher model (SD3.5-Large), and use adversarial step distillation for few-step generation. iv) Primary results: SnapGen achieves 1024x1024 pixel image generation on mobile devices in approximately 1.4 seconds, and the UNet model with only 379 million parameters achieves a GenEval score of 0.66. v) Principal implication for AI practitioners: AI practitioners can deploy high-resolution T2I models on mobile devices by using the architectural optimizations and training techniques presented, enabling new applications in mobile image generation. |
| PIG: Physics-Informed Gaussians as Adaptive Parametric Mesh Representations (Read more on arXiv or HuggingFace) | Eunbyung Park, Youngjoon Hong, Jaemin Oh, kangnamgyu27 | Here is a concise summary of the research paper “PIG: Physics-Informed Gaussians as Adaptive Parametric Mesh Representations” following your guidelines: i) Summary: This paper introduces Physics-Informed Gaussians (PIGs), a novel method for approximating solutions to partial differential equations (PDEs) using a combination of Gaussian functions and neural networks. ii) Main research question or objective: The main objective is to develop a more efficient and accurate PDE solver that overcomes the limitations of existing Physics-Informed Neural Networks (PINNs) and parametric grid-based methods. iii) Key methodology: PIGs employ a mixture of Gaussian functions with trainable parameters (mean, variance) to create adaptive feature embeddings, which are then processed by a lightweight neural network to approximate PDE solutions. iv) Primary results: PIGs demonstrate competitive accuracy and faster convergence compared to state-of-the-art methods across various PDEs; for example, PIG achieved a best relative L² error of 5.93 x 10^-5 on the Allen-Cahn equation. v) Principal implication for AI practitioners: AI practitioners can leverage PIGs as a robust and efficient tool for solving complex PDEs, offering an alternative to traditional PINNs with improved performance in terms of accuracy and computational cost. |
| Learned Compression for Compressed Learning (Read more on arXiv or HuggingFace) | Neeraja J. Yadwadkar, Dan Jacobellis | Here is a concise summary of the research paper “Learned Compression for Compressed Learning”: i) Summary: This paper introduces WaLLoC, a novel neural codec architecture for lossy compression that combines linear transform coding with nonlinear dimensionality-reducing autoencoders to enable efficient compressed-domain learning. ii) Main research question or objective: The main objective is to develop a compression method that simultaneously achieves computational efficiency, high compression ratios, and uniform dimensionality reduction for accelerating machine learning models. iii) Key methodology used: WaLLoC utilizes a wavelet packet transform followed by a shallow, asymmetric autoencoder and an entropy bottleneck, with a deep, nonlinear synthesis transform in the decoder. iv) Primary results: WaLLoC achieves up to 20x dimensionality reduction and outperforms existing methods in compression ratio, distortion, perceptual quality, and computational efficiency; for image classification, WaLLoC provides a 27.2% accuracy improvement over baseline resolution reduction. v) Principal implication for AI practitioners: WaLLoC enables AI practitioners to train and deploy machine learning models on compressed data with significantly reduced computational cost and latency while maintaining high accuracy, offering a practical solution for resource-constrained environments. |
| Lyra: An Efficient and Speech-Centric Framework for Omni-Cognition (Read more on arXiv or HuggingFace) | Longxiang Tang, Senqiao Yang, Yuqi Liu, Chengyao Wang, Zhisheng Zhong | Here’s a concise summary of the research paper “Lyra: An Efficient and Speech-Centric Framework for Omni-Cognition” following your specified guidelines: i) Summary: Lyra is a new multimodal large language model (MLLM) framework designed for efficient omni-cognition with a focus on enhanced speech processing capabilities. ii) Main research question or objective: How to develop an MLLM that efficiently integrates speech with other modalities (vision, language) to achieve state-of-the-art performance in multi-modal understanding and reasoning while minimizing computational resources and data requirements. iii) Key methodology: Lyra leverages existing open-source LLMs and VLMs, a proposed multi-modality LoRA, a latent multi-modality regularizer and extractor, and a newly constructed dataset including 1.5M multi-modal data samples and 12K long speech samples. iv) Primary results: Lyra outperforms previous models on various vision-language, vision-speech, and speech-language benchmarks, achieving 81.0% accuracy on the image-speech task [TextVQAS, DocVQAS, ChartQAS], and demonstrating significant improvements in processing long speech inputs lasting several hours. v) Principal implication for AI practitioners: AI practitioners can utilize Lyra to develop more efficient and versatile AI assistants capable of advanced speech comprehension, seamless cross-modality interactions, and handling long-context multi-modality applications with reduced computational demands. |
| RuleArena: A Benchmark for Rule-Guided Reasoning with LLMs in Real-World Scenarios (Read more on arXiv or HuggingFace) | Xiaobao Wu, Sitao Cheng, Liangming Pan, Wenyue Hua, Ruiwen Zhou | Here’s a concise summary of the research paper “RULEARENA: A Benchmark for Rule-Guided Reasoning with LLMs in Real-World Scenarios”: i) Summary: This paper introduces RULEARENA, a new benchmark for evaluating large language models (LLMs) on their ability to perform rule-guided reasoning in complex, real-world scenarios across domains like airline baggage fees, NBA transactions, and tax regulations. ii) Main research question or objective: To assess the proficiency of LLMs in understanding and applying complex, real-world rules expressed in natural language to solve practical reasoning problems. iii) Key methodology: The authors created 816 test problems across three domains, providing LLMs with task instructions, reference rules, and user instances, and then evaluated the models’ reasoning and computation based on a set of proposed metrics, including rule-wise and problem-wise recall, precision, and rule application correctness. iv) Primary results: State-of-the-art LLMs, including GPT-4o and Claude-3.5 Sonnet, generally failed on complex rule-guided reasoning tasks in the benchmark; for example, in the airline domain, even the best-performing model (GPT-4o) achieved a problem-wise accuracy of only 5% on the most challenging problems. v) Principal implication for AI practitioners: AI practitioners should be aware that even the most advanced LLMs currently exhibit significant limitations in accurately performing complex rule-guided reasoning in real-world applications. Therefore, relying solely on these models for tasks that require strict adherence to intricate rules may lead to unreliable or erroneous results. Developing specialized techniques to enhance rule grounding and multi-step reasoning in LLMs is crucial. |
| Gaze-LLE: Gaze Target Estimation via Large-Scale Learned Encoders (Read more on arXiv or HuggingFace) | Judy Hoffman, Daniel Bolya, Sangmin Lee, Ajay Bati, Fiona Ryan | Here is a concise summary of the research paper “Gaze-LLE: Gaze Target Estimation via Large-Scale Learned Encoders”: i) Summary: This paper introduces Gaze-LLE, a novel framework for gaze target estimation that leverages features from a frozen, pre-trained DINOv2 encoder. ii) Main research question or objective: Can a streamlined architecture using a frozen, large-scale learned encoder achieve state-of-the-art performance in gaze target estimation? iii) Key methodology: A transformer-based gaze decoder with a person-specific positional prompt is trained on top of a frozen DINOv2 encoder to predict gaze targets from a single scene representation. iv) Primary results: Gaze-LLE achieves state-of-the-art performance across multiple gaze estimation benchmarks, achieving an AUC of 0.956 on the GazeFollow dataset with only 2.8M learnable parameters. v) Principal implication for AI practitioners: AI practitioners can leverage Gaze-LLE’s streamlined architecture and frozen encoder to develop efficient and accurate gaze estimation models, simplifying the process compared to prior multi-branch approaches. |
| JuStRank: Benchmarking LLM Judges for System Ranking (Read more on arXiv or HuggingFace) | Lilach Eden, Roy Bar-Haim, Yotam Perlitz, Odellia Boni, Ariel Gera | Here’s a concise summary of the research paper “JuStRank: Benchmarking LLM Judges for System Ranking” following your guidelines: i) Summary: This paper introduces JuStRank, a benchmark for evaluating the performance of large language models (LLMs) as judges for ranking system outputs, revealing discrepancies between instance-level and system-level judging abilities. ii) Main research question/objective: How effectively can LLMs rank systems based on their outputs, and how does this system-level performance compare to their instance-level judging capabilities? iii) Key methodology: JuStRank evaluates 48 LLM judges by comparing their system rankings, derived from aggregating scores over multiple system outputs, against a human-based ranking using the Arena Hard v0.1 dataset. iv) Primary results: The study found that system-level performance does not directly correlate with instance-level performance; the Qwen2.5-72B-Instruct model achieved the highest agreement with the gold ranking at a Kendall’s Tau of 0.83. v) Principal implication for AI practitioners: AI practitioners should prioritize system-level evaluation when selecting LLM judges for system ranking tasks, as strong instance-level performance does not guarantee accurate system-level ranking. |
| OLA-VLM: Elevating Visual Perception in Multimodal LLMs with Auxiliary Embedding Distillation (Read more on arXiv or HuggingFace) | Jianwei Yang, Jianfeng Gao, Humphrey Shi, Zhengyuan Yang, Jitesh Jain | Here is a concise summary of the research paper “OLA-VLM: Elevating Visual Perception in Multimodal LLMs with Auxiliary Embedding Distillation”: i) Summary: The paper introduces OLA-VLM, a novel approach that enhances visual perception in Multimodal Large Language Models (MLLMs) by distilling knowledge from multiple target visual encoders into the LLM’s intermediate representations during pre-training. ii) Main Research Question/Objective: Can the visual understanding ability of MLLMs be improved by optimizing intermediate LLM representations through a vision-centric objective, specifically by distilling knowledge from a set of target visual encoders? iii) Key Methodology: OLA-VLM employs a predictive visual embedding optimization approach alongside the standard next text-token prediction objective during pre-training, using embedding losses to align LLM representations with features from specialized visual encoders for segmentation, depth estimation, and image generation. iv) Primary Results: OLA-VLM outperforms single and multi-encoder baselines on various benchmarks. Notably, it achieves an 8.7% improvement on the Depth task in CV-Bench compared to the baseline. v) Principal Implication for AI Practitioners: AI practitioners can leverage OLA-VLM’s embedding distillation technique to improve the visual perception of MLLMs, which directly enhances performance on vision-centric tasks without the need for multiple visual encoders during inference. |
| The Impact of Copyrighted Material on Large Language Models: A Norwegian Perspective (Read more on arXiv or HuggingFace) | David Samuel, Freddy Wetjen, Lemei Zhang, Vladislav Mikhailov, Javier de la Rosa | Here is a concise summary of the research paper: i) Summary: This study empirically evaluates the impact of copyrighted materials on the performance of large language models (LLMs) for the Norwegian language. ii) Main research question/objective: To assess how the inclusion of copyrighted Norwegian books and newspapers affects LLM performance on a suite of Norwegian benchmarks. iii) Key methodology: Researchers trained various LLMs on datasets with and without copyrighted materials, and compared their performance using quantitative NLP metrics and linguistic analysis. iv) Primary results: Models trained with copyrighted materials outperformed those without, with the model trained on the extended dataset (which includes copyrighted materials) achieving an average gain of 6.73% over the base model trained without copyrighted materials. v) Principal implication for AI practitioners: The inclusion of high-quality copyrighted material enhances the performance of Norwegian LLMs, suggesting that AI practitioners should carefully consider the legal and ethical implications of using such data in model training. |
| Word Sense Linking: Disambiguating Outside the Sandbox (Read more on arXiv or HuggingFace) | Roberto Navigli, Alberte Fernández-Castro, Luigi Procopio, Edoardo Barba, Andrei Stefan Bejgu | Here is a concise summary of the research paper “Word Sense Linking: Disambiguating Outside the Sandbox”: i) Summary: This paper introduces Word Sense Linking (WSL), a new task that extends Word Sense Disambiguation (WSD) by requiring systems to identify and disambiguate spans in text using a sense inventory, without prior span identification. ii) Main research question/objective: How can WSD be adapted to real-world scenarios where the spans to be disambiguated and their sense candidates are not pre-defined? iii) Key methodology: A retriever-reader architecture is proposed, where the retriever generates sense candidates and the reader identifies spans and assigns the most suitable sense. iv) Primary results: The proposed model achieved an F1-score of 75.9 on the WSL task, outperforming adaptations of state-of-the-art WSD systems. v) Principal implication for AI practitioners: AI practitioners can leverage the proposed WSL framework and architecture for more robust and practical lexical disambiguation in downstream applications, moving beyond the constrained assumptions of traditional WSD. |
| FreeSplatter: Pose-free Gaussian Splatting for Sparse-view 3D Reconstruction (Read more on arXiv or HuggingFace) | Ying Shan, Shenghua Gao, Jiale Xu | Here is a concise summary of the research paper “FreeSplatter: Pose-free Gaussian Splatting for Sparse-view 3D Reconstruction”: i) Summary: FreeSplatter is a feed-forward framework for reconstructing 3D scenes as Gaussians from uncalibrated sparse-view images and estimating their camera parameters in mere seconds. ii) Main research question/objective: Can a model directly predict 3D Gaussian maps from multi-view images to achieve both high-quality 3D modeling and instant camera pose estimation without known camera poses? iii) Key methodology: A transformer-based model predicts per-pixel 3D Gaussians from uncalibrated images, enabling simultaneous 3D reconstruction and camera pose estimation using iterative solvers. iv) Primary results: FreeSplatter-O achieved a PSNR of 31.929 on the OmniObject3D dataset for sparse-view reconstruction, outperforming prior methods. v) Principal implication for AI practitioners: AI practitioners can leverage FreeSplatter for efficient 3D reconstruction from sparse-view images without the need for pre-calibrated camera parameters, simplifying 3D content creation pipelines. |
| DisPose: Disentangling Pose Guidance for Controllable Human Image Animation (Read more on arXiv or HuggingFace) | Zhihong Zhu, Junjie Cao, Yuhang Yang, Yaowei Li, Hongxiang Li | Here’s a summary of the AI research paper following your strict guidelines: i) DisPose improves controllable human image animation by disentangling sparse pose guidance into motion field and keypoint correspondence. ii) The research objective is to improve controllable human image animation by generating more generalizable and effective control signals from sparse skeleton pose without additional dense input. iii) The key methodology involves disentangling sparse skeleton pose into a dense motion field generated from a sparse motion field and reference image, and extracting diffusion features corresponding to pose keypoints from the reference image for transfer to the target pose. A plug-and-play hybrid ControlNet integrates these signals into existing models. iv) Quantitative results show that DisPose outperforms existing methods, achieving a 29.51 score on the dynamic image quality metric in the TikTok dataset VBench, improving on the next best result of 28.42. Other quantitative metrics are reported but their specific values aren’t fully clear from the summary. v) For AI practitioners, DisPose offers a plug-and-play module readily integrable into existing human image animation models. Its enhanced control signals, derived from sparse input only, improve animation quality and consistency without requiring additional computationally expensive dense data. The paper lacks information about the scalability and generalisability across various model architectures and training regimes that would be valuable to developers. |
| LoRACLR: Contrastive Adaptation for Customization of Diffusion Models (Read more on arXiv or HuggingFace) | Pinar Yanardag, Federico Tombari, Thomas Hofmann, enisimsar | Here’s a concise summary of the research paper, strictly following the provided guidelines: i) Summary: The paper introduces LoRACLR, a method for merging multiple Low-Rank Adaptation (LoRA) models to enable multi-concept image generation in diffusion models without additional fine-tuning. ii) Main Research Question/Objective: How to effectively combine multiple pre-trained LoRA models, each customized for a distinct concept, into a single unified model for high-fidelity multi-concept image synthesis. iii) Key Methodology: LoRACLR employs a contrastive learning objective to align the weight spaces of multiple LoRA models, attracting positive pairs (same concept) and repelling negative pairs (different concepts) to ensure compatibility and minimize interference during merging. iv) Primary Results: LoRACLR achieves competitive performance across text, image, and identity alignment metrics, demonstrating superior visual quality and coherence compared to other methods; for instance, LoRACLR achieved an identity alignment score of .828 after merging, compared to .745 for Orthogonal Adaptation. v) Principal Implication for AI Practitioners: AI practitioners can leverage LoRACLR to efficiently merge pre-existing LoRA models, enabling scalable and flexible multi-concept image generation without the need for retraining or accessing original training data, thus advancing the capabilities of personalized image generation. |
| SAME: Learning Generic Language-Guided Visual Navigation with State-Adaptive Mixture of Experts (Read more on arXiv or HuggingFace) | Mohit Bansal, Chongyang Zhao, Zun Wang, Yicong Hong, Gengze Zhou | Here is a concise summary of the research paper “SAME: Learning Generic Language-Guided Visual Navigation with State-Adaptive Mixture of Experts”: i) Summary: This paper introduces SAME, a State-Adaptive Mixture of Experts model designed for versatile language-guided visual navigation across various instruction granularities. ii) Main research question/objective: How to create a unified framework for language-guided visual navigation that can handle diverse navigation tasks with varying levels of instruction granularity. iii) Key methodology: A novel State-Adaptive Mixture of Experts (SAME) model is proposed, enabling the agent to infer decisions based on different-granularity language and dynamic observations using a mixture of experts approach, where experts are selected based on the agent’s state. iv) Primary results: The SAME model achieves state-of-the-art or highly comparable performance across seven navigation tasks, demonstrating an average improvement of 3% in Success Rate (SR) across all tasks compared to the baseline multi-task-tuned model. v) Principal implication for AI practitioners: AI practitioners can utilize the SAME model to develop more generalizable and robust navigation agents capable of interpreting and executing a wide range of language instructions without requiring task-specific model architectures, potentially making the model easier to deploy in varied real-world scenarios. |
| Arbitrary-steps Image Super-resolution via Diffusion Inversion (Read more on arXiv or HuggingFace) | Chen Change Loy, Kang Liao, Zongsheng Yue | Here is a concise summary of the research paper “Arbitrary-steps Image Super-resolution via Diffusion Inversion”: i) The paper introduces InvSR, a diffusion inversion-based image super-resolution (SR) technique that allows for arbitrary-step sampling during inference. ii) The main research objective is to develop an efficient and flexible SR method that harnesses the rich image priors of pre-trained diffusion models while allowing users to freely adjust the number of sampling steps. iii) The key methodology is a Partial noise Prediction (PnP) strategy that constructs an intermediate state using a deep noise predictor to estimate the optimal noise maps for the forward diffusion process. iv) In experiments, InvSR achieved a PSNR of 24.14 and an SSIM of 0.6789 on the ImageNet-Test dataset with a single sampling step. v) For AI practitioners, InvSR offers a flexible and efficient approach to image super-resolution, demonstrating superior or comparable performance to recent state-of-the-art methods even with a single sampling step. |
| Shiksha: A Technical Domain focused Translation Dataset and Model for Indian Languages (Read more on arXiv or HuggingFace) | Srinivasan Umesh, rumourscape | Here is a concise summary of the research paper “Shiksha: A Technical Domain focused Translation Dataset and Model for Indian Languages” based on your specific guidelines: i) The paper introduces “Shiksha,” a novel dataset for machine translation focused on the technical domain, specifically for eight Indian languages. ii) The main research objective was to create a high-quality multilingual parallel corpus for English-to-Indic and Indic-to-Indic translation pairs in the scientific, technical, and educational domains, and to evaluate its impact on NMT model performance. iii) The key methodology involved extracting and cleaning data from NPTEL lecture transcriptions, followed by bitext mining using SentAlign with LABSE embeddings to identify parallel sentences. iv) The primary results showed that fine-tuning the NLLB 3.3B model on the Shiksha dataset achieved an average BLEU score of 48.98 on their in-domain test set. v) The principal implication for AI practitioners is that the Shiksha dataset can be used to significantly improve the performance of NMT models on technical domain translation tasks for Indian languages. |
Papers for 2024-12-12
| Title | Authors | Summary |
|---|---|---|
| SynCamMaster: Synchronizing Multi-Camera Video Generation from Diverse Viewpoints (Read more on arXiv or HuggingFace) | lemonaddie, ziyangy, Xintao, menghanxia, jianhongbai | Here is a concise summary of the AI research paper “SynCamMaster: Synchronizing Multi-Camera Video Generation from Diverse Viewpoints”: i) Summary: SynCamMaster is a novel framework for generating synchronized multi-camera videos from diverse viewpoints using a pre-trained text-to-video model augmented with a plug-and-play module. ii) Main research question or objective: How to achieve dynamic consistency across multiple viewpoints in open-domain multi-camera video generation. iii) Key methodology: A multi-view synchronization module is introduced to maintain appearance and geometry consistency, and a hybrid training scheme leverages multi-camera images, monocular videos, and Unreal Engine-rendered multi-camera videos. iv) Primary results: SynCamMaster outperforms baseline methods in generating view-synchronized videos, achieving a matching pixel count (Mat. Pix) of 527.1K, compared to the next best method’s 116.8K. v) Principal implication for AI practitioners: AI practitioners can utilize SynCamMaster’s multi-view synchronization module to generate consistent multi-camera videos, enhancing applications such as virtual filming. |
| LAION-SG: An Enhanced Large-Scale Dataset for Training Complex Image-Text Models with Structural Annotations (Read more on arXiv or HuggingFace) | MAJIARUI, SYZhang0805, yeezlee, mengcy, hyllbd | Here is a concise summary of the research paper: i) The paper introduces LAION-SG, a large-scale dataset with scene graph annotations for training text-to-image models to generate complex images with multiple objects and intricate relationships. ii) The main research question is how to improve text-to-image models’ performance in generating complex compositional images involving multiple objects and relationships. iii) The key methodology involves automatically generating scene graph annotations using GPT-4 and constructing a new dataset, LAION-SG, based on LAION-Aesthetics V2, along with developing a foundation model, SDXL-SG, that incorporates scene graph information into the Stable Diffusion XL model using graph neural networks. iv) The primary result is that SDXL-SG outperforms existing models on complex scene generation, achieving a 20.1 FID score and 0.558 SG-IoU on LAION-SG, indicating improved image quality and semantic accuracy. v) For AI practitioners, LAION-SG provides a valuable resource for training and evaluating models for complex image generation, and SDXL-SG offers a new approach to incorporating structural information into the generation process, with the potential to enhance the accuracy and controllability of text-to-image models. |
| POINTS1.5: Building a Vision-Language Model towards Real World Applications (Read more on arXiv or HuggingFace) | Xiao Zhou, Le Tian, yangyu1, kavio, YuanLiuuuuuu | Okay, here is a concise summary of the paper “POINTS1.5: Building a Vision-Language Model towards Real World Applications” following your specified guidelines: i) POINTS1.5 is a vision-language model designed for enhanced performance in real-world applications like optical character recognition and diagram analysis. ii) The main research objective is to develop an improved vision-language model, POINTS1.5, that surpasses its predecessor, POINTS1.0, by incorporating native dynamic high-resolution image processing and bilingual support, specifically for English and Chinese. iii) Key methodology involves replacing the CLIP vision encoder with a NaViT-style encoder for dynamic resolution support, creating a large Chinese corpus for pre-training and visual instruction tuning, and implementing rigorous filtering methods for the visual instruction tuning datasets. iv) Primary results show that POINTS1.5-7B outperforms all other models under 10 billion parameters on the OpenCompass leaderboard, achieving a score of 67.4 after model soup. v) Principal implication for AI practitioners is that POINTS1.5 provides a more accurate and efficient framework for real-world vision-language tasks, particularly those requiring high-resolution image understanding and bilingual (Chinese-English) language processing, offering a strong foundation for developing applications that can handle diverse visual and textual data inputs. |
| Learning Flow Fields in Attention for Controllable Person Image Generation (Read more on arXiv or HuggingFace) | AdityaPatel, Wall-dandelion, Yuren, shikunl, franciszzj | Here is a concise summary of the research paper “Learning Flow Fields in Attention for Controllable Person Image Generation”: i) This paper introduces Leffa, a regularization loss that improves controllable person image generation by learning flow fields within attention mechanisms to reduce detail distortion. ii) Main research objective: To alleviate the distortion of fine-grained details in controllable person image generation while maintaining high overall image quality. iii) Key methodology: A regularization loss (Leffa) is proposed that guides target queries to attend to correct reference keys in attention layers by transforming attention maps into flow fields and warping the reference image towards the target image. iv) Primary results: Leffa achieves state-of-the-art performance on virtual try-on and pose transfer, achieving a FID of 4.54 on the VITON-HD dataset (paired setting) for virtual try-on. v) Principal implication for AI practitioners: AI practitioners can use Leffa as a model-agnostic loss function to enhance the performance of existing diffusion models in controllable person image generation tasks by reducing fine-grained detail distortion without additional inference costs or parameters. |
| StyleMaster: Stylize Your Video with Artistic Generation and Translation (Read more on arXiv or HuggingFace) | Huijuan Huang, whluo, qq8933, Xintao, zixuan-ye | Here is a concise summary of the research paper “StyleMaster: Stylize Your Video with Artistic Generation and Translation”: i) StyleMaster is a novel framework for video stylization that achieves high-quality results in both stylized video generation and video-to-video style transfer. ii) Main research question/objective: How to effectively extract and inject style features into video generation models to achieve accurate and consistent stylization while preserving content fidelity? iii) Key methodology: A style extraction module with local patch selection based on prompt-patch similarity and global style projection trained via contrastive learning on a paired style dataset generated through model illusion, coupled with a motion adapter and a gray tile ControlNet. iv) Primary results: StyleMaster outperforms existing methods in style resemblance and temporal coherence, achieving a CLIP-Text similarity score of 0.305 in stylized video generation. v) Principal implication for AI practitioners: AI practitioners can leverage StyleMaster’s style extraction and injection techniques to develop advanced video editing tools and creative applications with enhanced control over stylization. |
| Generative Densification: Learning to Densify Gaussians for High-Fidelity Generalizable 3D Reconstruction (Read more on arXiv or HuggingFace) | JustinOh, LeeYG, lelady, xysun, stnamjef | Here is a concise summary of the research paper “Generative Densification: Learning to Densify Gaussians for High-Fidelity Generalizable 3D Reconstruction”: i) Summary: This paper introduces Generative Densification (GD), a method to improve the detail representation of generalized feed-forward Gaussian models for 3D reconstruction. ii) Main research question/objective: How can the densification strategy used in per-scene 3D Gaussian Splatting be adapted to enhance the representation of high-frequency details in generalized feed-forward Gaussian models? iii) Key methodology: GD selectively densifies the top K Gaussians with large view-space positional gradients based on learned prior knowledge, up-sampling feature representations and generating corresponding fine Gaussians in a single forward pass using a point-level transformer. iv) Primary results: The proposed method outperforms state-of-the-art approaches on object-level and scene-level reconstruction tasks; for instance, it achieved a PSNR of 28.75 on the Gobjaverse dataset, compared to 27.49 for the LaRa baseline. v) Principal implication for AI practitioners: AI practitioners can leverage GD to improve the fidelity of 3D reconstructions from sparse-view inputs by efficiently densifying Gaussians based on learned prior knowledge, enabling more detailed and accurate 3D models. |
| StreamChat: Chatting with Streaming Video (Read more on arXiv or HuggingFace) | Shiyi Lan, hsli-cuhk, LucasFang, Zhiding, jjjjh | Here is a concise summary of the StreamChat paper based on your guidelines: i) Summary: StreamChat is a novel approach that enables large multimodal models (LMMs) to dynamically interact with streaming video by updating the visual context at each decoding step. ii) Main Research Question/Objective: How to enable LMMs to effectively interact with streaming videos and utilize up-to-date video content throughout the decoding process. iii) Key Methodology: Introduction of a cross-attention-based architecture that processes dynamic streaming inputs, a parallel 3D-RoPE mechanism for encoding temporal information, and a new dense instruction dataset for training. iv) Primary Results: StreamChat-7B outperforms the state-of-the-art LLaVA-Video-72B model in streaming interaction scenarios, with the StreamChat-7B model producing equally or more preferable answers in 77% of the evaluation cases compared to VILA-1.5-40B. v) Principal Implication for AI Practitioners: AI practitioners can use StreamChat to develop more interactive and responsive video understanding models that maintain context continuity in streaming scenarios, enhancing user experience in real-time applications. |
| Mogo: RQ Hierarchical Causal Transformer for High-Quality 3D Human Motion Generation (Read more on arXiv or HuggingFace) | Frag1le | Here is a concise summary of the research paper “Mogo: RQ Hierarchical Causal Transformer for High-Quality 3D Human Motion Generation” by Frag1le: i) This paper introduces Mogo, a novel GPT-type model for generating high-quality, long, and open-vocabulary 3D human motion sequences. ii) The main research objective is to develop a model that surpasses the quality of BERT-type models in text-to-motion generation while leveraging the streaming output capability of GPT-type models. iii) The key methodology involves a hierarchical residual vector quantization variational autoencoder (RVQ-VAE) for motion sequence discretization and a Hierarchical Causal Transformer for autoregressive generation and residual inference. iv) On the HumanML3D test set, Mogo achieves a Fréchet Inception Distance (FID) score of 0.079, outperforming the T2M-GPT model. v) For AI practitioners, Mogo offers a new approach that combines the strengths of GPT and BERT-type models in a single transformer model, improving the quality and efficiency of 3D human motion generation without adding extra refinement models. |
| KaSA: Knowledge-Aware Singular-Value Adaptation of Large Language Models (Read more on arXiv or HuggingFace) | Jing Tang, Sunghun Kim, Chansung Park, Juyong Jiang, Fan Wang | Here is a concise summary of the research paper “KaSA: Knowledge-Aware Singular-Value Adaptation of Large Language Models” based on the guidelines provided: 1. Summary: The paper introduces Knowledge-aware Singular-value Adaptation (KaSA), a parameter-efficient fine-tuning (PEFT) method that leverages singular value decomposition (SVD) to dynamically activate relevant knowledge in large language models (LLMs) for specific downstream tasks. 2. Main research question or objective: The main objective is to develop a PEFT method that addresses the limitations of existing methods like LoRA by dynamically activating task-relevant knowledge while minimizing the interference of noisy or irrelevant knowledge during fine-tuning. 3. Key methodology used: KaSA employs SVD with knowledge-aware singular values to adapt LLMs. It performs knowledge-based SVD truncation to remove minor singular components representing noise and reparameterizes task-specific updates in SVD form to maintain a consistent representational space. It introduces knowledge-aware singular values (Δσι, …, Δσr) to activate relevant parametric knowledge based on its relevance to specific downstream tasks and incorporates regularization terms (L2 and L3) to constrain the task-specific updates. 4. Primary results: KaSA consistently outperforms full fine-tuning (FFT) and 14 popular PEFT baselines across 16 benchmarks and 4 synthetic datasets. Specifically, on the GLUE benchmark, KaSA achieved an average performance of 86.3% for RoBERTa-base, surpassing other methods. 5. Principal implication for AI practitioners: AI practitioners can leverage KaSA as a superior PEFT method to efficiently adapt LLMs to various downstream tasks, achieving improved performance with significantly reduced computational and memory costs compared to full fine-tuning and other popular PEFT methods. |
| FlowEdit: Inversion-Free Text-Based Editing Using Pre-Trained Flow Models (Read more on arXiv or HuggingFace) | Tomer Michaeli, Inbar Huberman-Spiegelglas, Matan Kleiner, Vladimir Kulikov | Here is a concise summary of the research paper “FlowEdit: Inversion-Free Text-Based Editing Using Pre-Trained Flow Models”: i) Summary: FlowEdit is a novel, inversion-free, and optimization-free method for text-based image editing using pre-trained flow models. ii) Main research question/objective: The main objective is to develop a text-based image editing method for flow models that directly maps between source and target image distributions without relying on inversion, optimization, or model-specific interventions. iii) Key methodology used: FlowEdit constructs an ordinary differential equation (ODE) that directly maps the source image distribution to the target distribution, corresponding to the source and target text prompts, achieving a lower transport cost than inversion-based methods. iv) Primary results: FlowEdit achieves lower transport cost compared to editing-by-inversion (1376 vs. 2239 for MSE between source-target pairs in a synthetic dataset of model-generated images). v) Principal implication for AI practitioners: AI practitioners can use FlowEdit for efficient and structure-preserving text-based image editing with pre-trained flow models, without the need for computationally intensive inversion or optimization steps. |
| StyleStudio: Text-Driven Style Transfer with Selective Control of Style Elements (Read more on arXiv or HuggingFace) | Chi Zhang, Hao Wang, Beier Zhu, Xue Song, Mingkun Lei | Here is a concise summary of the research paper “StyleStudio: Text-Driven Style Transfer with Selective Control of Style Elements”: i) StyleStudio is a text-driven style transfer model that improves upon existing methods by enhancing the alignment of generated images with text prompts while preserving style fidelity and layout structure. ii) The main objective is to address the challenges of style overfitting, limited stylistic control, and misalignment with textual content in text-driven style transfer. iii) The key methodology includes a cross-modal Adaptive Instance Normalization (AdaIN) for feature integration, a Style-based Classifier-Free Guidance (SCFG) for selective style control, and a teacher model for stabilizing spatial layouts. iv) The proposed method achieves a text alignment score of 0.235, outperforming other methods evaluated. v) For AI practitioners, the principal implication is that StyleStudio can be integrated into existing style transfer frameworks without fine-tuning to improve text-to-image generation alignment and offer finer control over stylistic elements. |
| MIT-10M: A Large Scale Parallel Corpus of Multilingual Image Translation (Read more on arXiv or HuggingFace) | Lijie Wen, Shaolin Zhu, liboaccn | Here is a concise summary of the AI research paper “MIT-10M: A Large Scale Parallel Corpus of Multilingual Image Translation”: i) Summary: This paper introduces MIT-10M, a new dataset for multilingual image translation, addressing limitations in existing datasets regarding scale, diversity, and quality. ii) Main research question or objective: The main objective is to create a large-scale, high-quality parallel corpus for multilingual image translation that reflects real-world data complexities. iii) Key methodology used: The methodology involved web crawling, data cleaning, OCR annotation, and multilingual translation with validation using GPT-4 and Google Translate. iv) Primary results: The MIT-10M dataset contains over 10 million image-text pairs across 14 languages and 840K images; fine-tuning the Qwen2-VL model with MIT-10M improved the BLEU score by 230%. v) Principal implication for AI practitioners: AI practitioners can use MIT-10M to train and evaluate multilingual image translation models, leading to more robust models capable of handling diverse, real-world scenarios. |
Papers for 2024-12-11
| Title | Authors | Summary |
|---|---|---|
| Evaluating and Aligning CodeLLMs on Human Preference (Read more on arXiv or HuggingFace) | JustinLin610, huybery, misakamage, instro, jx-yang | Here is a concise summary of the paper “Evaluating and Aligning CodeLLMs on Human Preference”: i) Summary: This paper introduces CodeArena, a new benchmark for evaluating code language models (codeLLMs) based on human preferences, and SynCode-Instruct, a large-scale synthetic instruction dataset for enhancing codeLLM alignment with human preferences. ii) Main Research Question/Objective: How to evaluate and improve the alignment of codeLLMs with human preferences in realistic code generation scenarios. iii) Key Methodology: Development of CodeArena with 397 human-curated samples across 40 categories and 44 programming languages, and creation of SynCode-Instruct, a 20 billion token synthetic instruction dataset derived from web data. iv) Primary Results: CodeArena reveals a significant performance gap between open-source and proprietary LLMs, with Qwen2.5-SynCoder achieving the best performance among open-source models evaluated (49.2/22.3 win rate/tie rate). v) Principal Implication for AI Practitioners: AI practitioners should consider human preference alignment in codeLLM evaluation and training, utilizing benchmarks like CodeArena and large-scale synthetic instruction datasets for improved performance. |
| DiffSensei: Bridging Multi-Modal LLMs and Diffusion Models for Customized Manga Generation (Read more on arXiv or HuggingFace) | Chao Tang, LXT, zengyh1900, JingboWang, jianzongwu | Here’s a summary of the research paper “DiffSensei: Bridging Multi-Modal LLMs and Diffusion Models for Customized Manga Generation” following your specified guidelines: i) Summary: DiffSensei is a novel framework for customized manga generation that integrates diffusion models with a multimodal large language model (MLLM) for dynamic, multi-character control based on text prompts and user inputs. ii) Main research question/objective: How to generate customized manga panels with multiple characters, precise layout control, and dynamic adaptation to textual prompts. iii) Key methodology: The approach employs an MLLM as a text-compatible identity adapter for diffusion-based image generation, using masked cross-attention to incorporate character features and a dialog embedding technique for precise dialog placement. iv) Primary results: DiffSensei outperforms existing models in experiments, achieving a 0.06 improvement in CLIP metrics compared to the multi-subject customization baseline, MS-Diffusion. v) Principal implication for AI practitioners: AI practitioners can leverage DiffSensei to create manga generation tools with enhanced character customization and layout control, enabling more dynamic and interactive storytelling capabilities. |
| STIV: Scalable Text and Image Conditioned Video Generation (Read more on arXiv or HuggingFace) | jefflai, JesseAllardice, tsujuifu, wenzehu, Jiasenlu | Here is a concise summary of the research paper “STIV: Scalable Text and Image Conditioned Video Generation” following your guidelines: i) Summary: This paper introduces STIV, a scalable text-image-conditioned video generation model based on a Diffusion Transformer (DiT) architecture that can perform both text-to-video (T2V) and text-image-to-video (TI2V) tasks. ii) Main research question/objective: How to develop a robust and scalable video generation model that effectively integrates text and image conditioning within a unified framework. iii) Key methodology: The authors integrated image conditioning into a DiT through frame replacement and text conditioning via joint image-text conditional classifier-free guidance, and conducted a systematic study on model architectures, training recipes, and data curation strategies. iv) Primary results: The 8.7B parameter STIV model achieved a state-of-the-art VBench T2V score of 83.1 and a VBench I2V score of 90.1 at 512x512 resolution, surpassing models like CogVideoX-5B, Pika, Kling, and Gen-3. v) Principal implication for AI practitioners: AI practitioners can leverage the STIV framework and the provided recipes for building and scaling video generation models, enabling the development of more versatile and reliable video generation solutions for various downstream applications. |
| Hidden in the Noise: Two-Stage Robust Watermarking for Images (Read more on arXiv or HuggingFace) | Niv Cohen, chegde, rtealwitter, penfever, kasraarabi | Here’s a concise summary of the research paper “Hidden in the Noise: Two-Stage Robust Watermarking for Images” based on the provided guidelines: i) Summary: The paper introduces WIND, a two-stage watermarking method for images generated by diffusion models, designed to be robust against removal and forgery attacks. ii) Main research question/objective: How to develop a distortion-free watermarking technique for diffusion-generated images that is robust to common attacks while maintaining detection efficiency? iii) Key methodology: WIND employs a two-stage approach, first embedding a group identifier in the Fourier space of the initial noise and then using a secret salt and hash function to generate a unique, reproducible initial noise for watermarking. iv) Primary results: WIND achieved a 94.7% average detection accuracy across various image transformation attacks when using 128 groups of initial noises, and the proposed method demonstrates resilience against a regeneration attack. v) Principal implication for AI practitioners: AI practitioners can utilize WIND to watermark images generated by their models, enabling them to verify image origins and protect against unauthorized use, with a negligible impact on image quality and a demonstrated detection accuracy of 94.7% under various attacks. |
| UniReal: Universal Image Generation and Editing via Learning Real-world Dynamics (Read more on arXiv or HuggingFace) | Yuqian Zhou, He Zhang, Zhifei Zhang, jimmie33, xichenhku | Here is a concise summary of the research paper “UniReal: Universal Image Generation and Editing via Learning Real-world Dynamics”: i) Summary: UniReal is a unified framework for diverse image generation and editing tasks, treating image tasks as discontinuous video generation and learning from large-scale videos. ii) Main research question/objective: To develop a unified framework that can address various image generation and editing tasks within a single model using a scalable training paradigm. iii) Key methodology: The paper proposes leveraging a video generation framework based on a diffusion transformer, treating input/output images as video frames, and employing hierarchical prompts and image index embeddings for task and image coordination. iv) Primary results: UniReal outperforms existing methods in instructive image editing, customized image generation, and object insertion; e.g. UniReal achieves a CLIP score of 0.851 and a DINO score of 0.790 on the EMU Edit test set. v) Principal implication for AI practitioners: AI practitioners can leverage UniReal as a versatile tool for various image generation and editing tasks, simplifying development by using a single model trained on readily available video data instead of task-specific datasets. |
| OmniDocBench: Benchmarking Diverse PDF Document Parsing with Comprehensive Annotations (Read more on arXiv or HuggingFace) | conghui, friskit, Liam-Liu, wanderkid, ouyanglinke | Here’s a concise summary of the research paper “OmniDocBench: Benchmarking Diverse PDF Document Parsing with Comprehensive Annotations” based on your specified guidelines: i) Summary: This paper introduces OmniDocBench, a new benchmark for evaluating PDF document parsing methods, featuring a diverse dataset with comprehensive annotations. ii) Main research question/objective: To develop a robust, diverse, and fair evaluation standard for document content extraction methods. iii) Key methodology: Construction of a high-quality dataset with 981 PDF pages across nine types, with 19 layout category labels and 14 attribute labels for evaluating pipeline and end-to-end document parsing methods. iv) Primary results: Pipeline-based methods like MinerU and Mathpix achieved the best overall parsing performance (e.g., MinerU achieved 0.188 average edit distance across 9 PDF types); however, general VLMs showed stronger generalization on specialized data. v) Principal implication for AI practitioners: OmniDocBench provides a standardized benchmark to systematically evaluate and improve the accuracy, robustness, and generalization capabilities of document parsing models across diverse document types and layouts, which can directly improve the tools that AI practitioners work with. |
| FiVA: Fine-grained Visual Attribute Dataset for Text-to-Image Diffusion Models (Read more on arXiv or HuggingFace) | myownskyW7, guandao, Dubhe-zmc, justimyhxu, tongwu2020 | Here’s a concise summary of the paper: i) Summary: The paper introduces FiVA, a new dataset of 1 million images with fine-grained visual attribute annotations, and FiVA-Adapter, a framework for controlling image generation using these attributes. ii) Main research question or objective: To develop a method for decomposing the aesthetics of an image into specific visual attributes and enable users to control image generation based on these attributes. iii) Key methodology: Construction of a dataset (FiVA) using a pipeline involving attribute definition, prompt creation, LLM-based filtering, and human validation, followed by the development of an adaptation framework (FiVA-Adapter) that integrates a multimodal encoder into an image feature encoder for attribute extraction. iv) Primary results: The FiVA-Adapter achieved a subject accuracy of 0.817 in user studies, outperforming baseline methods. v) Principal implication for AI practitioners: AI practitioners can leverage the FiVA dataset and FiVA-Adapter to enhance the controllability of text-to-image diffusion models, enabling more precise manipulation of fine-grained visual attributes in generated images. |
| Perception Tokens Enhance Visual Reasoning in Multimodal Language Models (Read more on arXiv or HuggingFace) | Dongping Chen, Ethan Shen, Cheng-Yu Hsieh, Zelun Luo, Mahtab Bigverdi | Here is a concise summary of the research paper “Perception Tokens Enhance Visual Reasoning in Multimodal Language Models”: i) Summary: This paper introduces “Perception Tokens,” a novel approach to enhance visual reasoning in multimodal language models (MLMs) by using intermediate image representations as auxiliary reasoning tokens. ii) Main research question or objective: The main objective is to develop a method for augmenting MLMs with the ability to reason over intrinsic image representations, such as depth maps and bounding boxes, to improve performance on visual reasoning tasks. iii) Key methodology: The authors propose AURORA, a multi-task training framework that uses a VQVAE to transform intermediate image representations into tokenized formats and bounding box tokens, which are then used to train MLMs to leverage these “Perception Tokens” as chain-of-thought prompts. iv) Primary results: AURORA significantly improves performance on counting benchmarks, achieving a +10.8% improvement on BLINK. v) Principal implication for AI practitioners: AI practitioners can leverage AURORA to expand the scope of MLMs beyond language-based reasoning, enabling more effective visual reasoning capabilities by incorporating intermediate visual representations directly into the model’s reasoning process. |
| 3DTrajMaster: Mastering 3D Trajectory for Multi-Entity Motion in Video Generation (Read more on arXiv or HuggingFace) | Menghan Xia, Sida Peng, Xintao Wang, Xian Liu, lemonaddie | Here is a summary of the provided AI research paper, strictly adhering to the specified guidelines: i) 3DTrajMaster achieves state-of-the-art accuracy in controlling multi-entity 3D motions in video generation using 6DoF pose sequences as input. ii) The research objective was to manipulate multi-entity 3D motions in video generation, overcoming the limitations of prior methods that primarily used 2D control signals. iii) The core methodology involved a plug-and-play 3D-motion grounded object injector that fused multiple input entities with their 3D trajectories via a gated self-attention mechanism. A 360°-Motion Dataset was created for training, incorporating a domain adaptor and annealed sampling strategy to improve video quality. iv) The primary results showed that 3DTrajMaster achieved a 0.398m translation error and a 0.277-degree rotation error on average in controlling multiple entity motions. v) For AI practitioners, the development of 3DTrajMaster provides a novel approach for controlling multi-entity 3D motions in video generation; the creation of a new dataset with synchronized multi-camera recordings of diverse 3D entities addresses the limited availability of training data for this task. The paper does not explicitly detail the model architecture’s specific components (e.g., layer sizes, activation functions, etc.), limiting direct application without further clarification. |
| Frame Representation Hypothesis: Multi-Token LLM Interpretability and Concept-Guided Text Generation (Read more on arXiv or HuggingFace) | Kazuhiro Fukui, Erica K. Shimomoto, Lincon S. Souza, Pedro H. V. Valois | Here is a concise summary of the research paper “Frame Representation Hypothesis: Multi-Token LLM Interpretability and Concept-Guided Text Generation”: i) Summary: This paper introduces the Frame Representation Hypothesis (FRH) to interpret and control Large Language Models (LLMs) by representing words as frames (ordered sequences of linearly independent token vectors) and concepts as the average of word frames. ii) Main research question/objective: How can multi-token words be effectively modeled to enhance LLM interpretability and control? iii) Key methodology: The authors propose representing words as frames and concepts as the average of word frames within a defined Semantic Frame Space and introduce Top-k Concept-Guided Decoding to steer text generation. iv) Primary results: The FRH is validated by showing that over 99% of words across multiple languages in the Open Multilingual WordNet (OMW) are composed of linearly independent token vectors, and concept-guided generation effectively steers output towards desired concepts. v) Principal implication for AI practitioners: The FRH offers a novel framework for AI researchers and engineers to enhance LLM interpretability and control by leveraging multi-token word representations, enabling more precise manipulation of model outputs. |
| Video Motion Transfer with Diffusion Transformers (Read more on arXiv or HuggingFace) | Sergey Tulyakov, fabvio, philiptorr, aliaksandr-siarohin, alexpondaven | Here is a concise summary of the paper “Video Motion Transfer with Diffusion Transformers”: i) Summary: The paper introduces DiTFlow, a novel method for transferring motion from a reference video to a newly synthesized video using Diffusion Transformers (DiTs). ii) Main research question/objective: How to transfer the motion of a reference video to a newly synthesized one, specifically for Diffusion Transformers (DiT). iii) Key methodology: DiTFlow extracts an Attention Motion Flow (AMF) from a reference video by analyzing cross-frame attention maps in a pre-trained DiT, then uses this AMF to guide the latent denoising process in an optimization-based, training-free manner. iv) Primary results: DiTFlow outperforms all baseline methods in motion transfer on multiple metrics; specifically, it achieves a Motion Fidelity (MF) score of 0.785 on the 5B parameter model, compared to 0.766 for the best-performing baseline. v) Principal implication for AI practitioners: AI practitioners can leverage DiTFlow for improved motion transfer in video synthesis using DiTs, enabling more precise control over the motion of generated video content without the need for model retraining. |
| EMOv2: Pushing 5M Vision Model Frontier (Read more on arXiv or HuggingFace) | Zhucun Xue, Teng Hu, Jiangning Zhang, LXT, hhy724 | Here is a concise summary of the research paper “EMOv2: Pushing 5M Vision Model Frontier” based on the provided guidelines: i) This paper introduces EMOv2, a new family of efficient vision models designed for resource-constrained scenarios, focusing on optimizing the trade-off between parameters, FLOPs, and performance within the 5M parameter magnitude. ii) The main research objective is to establish a new performance frontier for 5M parameter magnitude lightweight models on various downstream visual tasks. iii) The key methodology involves abstracting a Meta Mobile Block (MMBlock) to unify the design of Inverted Residual Block (IRB) and attention-based modules, and deducing an improved Inverted Residual Mobile Block (i2RMB) with a novel spanning attention mechanism. iv) EMOv2-5M achieves 79.4 Top-1 accuracy on ImageNet-1K classification, outperforming prior state-of-the-art models of similar size. v) For AI practitioners, EMOv2 provides a highly efficient and versatile backbone that can be readily adapted to various vision tasks, including classification, detection, segmentation, and generation, offering a strong baseline for mobile and edge device applications with strict parameter constraints. |
| Granite Guardian (Read more on arXiv or HuggingFace) | Tejaswini Pedapati, Subhajit Chaudhury, Manish Nagireddy, Inkit Padhi, Giandomenico | Okay, here is a concise summary of the Granite Guardian AI research paper, following your specified guidelines: 1. Summary: The paper introduces Granite Guardian, a suite of open-source Large Language Model (LLM) safeguards designed for risk detection in prompts and responses across various dimensions, including harmful content and Retrieval-Augmented Generation (RAG) hallucination. 2. Main research question/objective: To develop and evaluate a unified risk detection model family capable of identifying a broad spectrum of risks in LLM inputs and outputs, including those typically overlooked by traditional risk detection models. 3. Key methodology: Supervised fine-tuning of Granite 3.0 language models on a dataset combining human annotations from diverse sources and synthetic data, with a specialized safety instruction template for risk categorization. 4. Primary results: Granite Guardian achieves state-of-the-art risk detection with an AUC score of 0.871 on harmful content benchmarks. 5. Principal implication for AI practitioners: AI practitioners can use Granite Guardian as adaptable, plug-and-play components to enhance the safety and reliability of LLMs in various applications by enabling robust risk detection across multiple risk dimensions. |
| ILLUME: Illuminating Your LLMs to See, Draw, and Self-Enhance (Read more on arXiv or HuggingFace) | Jianhua Han, Runhui Huang, Junwei Yang, Guansong Lu, Chunwei Wang | Here is a concise summary of the research paper “ILLUME: Illuminating Your LLMs to See, Draw, and Self-Enhance”: i) ILLUME is a unified multimodal large language model (MLLM) that integrates visual understanding and generation through a unified next-token prediction formulation. ii) Main research question/objective: Can a unified MLLM be developed more efficiently, and can the discriminative and generative capabilities of an MLLM enhance each other? iii) Key methodology: A semantic vision tokenizer incorporating semantic information and a progressive multi-stage training procedure are used to enhance data efficiency, alongside a novel self-enhancing multimodal alignment scheme. iv) Primary results: ILLUME requires only 15M data for image-text alignment during pretraining and achieves 7.76 FID score on the MJHQ30K benchmark. v) Principal implication for AI practitioners: AI practitioners can leverage ILLUME’s efficient training approach and architecture for developing unified MLLMs with strong visual understanding and generation capabilities, potentially reducing the data and computational resources typically required. |
| ObjCtrl-2.5D: Training-free Object Control with Camera Poses (Read more on arXiv or HuggingFace) | Chen Change Loy, Shangchen Zhou, Yushi Lan, Zhouxia Wang | Here is a concise summary of the research paper “ObjCtrl-2.5D: Training-free Object Control with Camera Poses”: i) Summary: The paper introduces ObjCtrl-2.5D, a training-free method for controlling object motion in image-to-video generation by extending 2D trajectories to 3D and representing them as camera poses. ii) Main research question or objective: The main objective is to achieve more precise and versatile object control in image-to-video (I2V) generation compared to existing methods. iii) Key methodology used: ObjCtrl-2.5D extends 2D trajectories to 3D using depth information, models object movement as camera poses, and utilizes a Layer Control Module and Shared Warping Latent to adapt a camera motion control model for object motion control. iv) Primary results: ObjCtrl-2.5D achieved an Object Motion Control (ObjMC) score of 91.42 on the DAVIS dataset when combining a 2D trajectory with depth from the conditional image. v) Principal implication for AI practitioners: ObjCtrl-2.5D provides a training-free approach for precise object motion control in video generation, offering more diverse control capabilities than existing 2D trajectory-based methods without the need for model training. |
| LoRA.rar: Learning to Merge LoRAs via Hypernetworks for Subject-Style Conditioned Image Generation (Read more on arXiv or HuggingFace) | Umberto Michieli, Pietro Zanuttigh, Mete Ozay, obohdal, donaldssh | Okay, here is a concise summary of the research paper “LoRA.rar: Learning to Merge LoRAs via Hypernetworks for Subject-Style Conditioned Image Generation,” strictly adhering to your guidelines: i) Summary: LoRA.rar is a novel method that efficiently merges subject and style LoRAs using a pre-trained hypernetwork for fast, high-quality, personalized image generation. ii) Main research question or objective: The main objective is to develop a method for merging content and style LoRAs that achieves superior image quality compared to state-of-the-art methods while enabling real-time performance on resource-constrained devices. iii) Key methodology used: The key methodology involves pre-training a hypernetwork on a diverse dataset of content-style LoRA pairs to predict merging coefficients, enabling generalization to unseen pairs during deployment. iv) Primary results: LoRA.rar outperforms existing methods, including ZipLoRA, in both content and style fidelity, achieving a merging speedup of over 4000x and a score of 0.71 in average case using the proposed Multimodal Assistant Rating Subject & Style (MARS2) metric, compared to 0.58 for the next best method. v) Principal implication for AI practitioners: AI practitioners can leverage LoRA.rar for efficient, high-quality, subject-style conditioned image generation, particularly in applications requiring real-time performance on devices with limited computational resources. |
| Fully Open Source Moxin-7B Technical Report (Read more on arXiv or HuggingFace) | Sung-En Chang, Yixin Shen, Zhenglun Kong, Xuan Shen, Pu Zhao | Here is a summary of the research paper “Fully Open Source Moxin-LLM Technical Report” based on your specified format: i) Summary: This paper introduces Moxin-7B, a fully open-source large language model (LLM) developed in accordance with the Model Openness Framework (MOF), emphasizing complete transparency in training, datasets, and implementation. ii) Main research question or objective: The main objective is to develop a high-performing, fully open-source 7B parameter LLM that adheres to the principles of open science, open source, open data, and open access as defined by the MOF. iii) Key methodology used: The model architecture extends the Mistral model, utilizing grouped-query attention and sliding window attention, trained on a mix of SlimPajama and DCLM-BASELINE datasets, with capability enhancement using data from HuggingFace. iv) Primary results: Moxin-7B-finetuned achieves superior performance in zero-shot evaluation compared with popular 7B models, notably scoring 82.24% on the PIQA benchmark. v) Principal implication for AI practitioners: AI practitioners can leverage Moxin-7B’s open-source nature, including its training code, datasets, and checkpoints, to further innovate, customize, and deploy LLMs across diverse applications, fostering a more transparent and collaborative AI ecosystem. |
| Contextualized Counterspeech: Strategies for Adaptation, Personalization, and Evaluation (Read more on arXiv or HuggingFace) | Felice Dell’Orletta, Marco Avvenuti, Amaury Trujillo, Alessio Miaschi, Lorenzo Cima | Here’s a concise summary of the paper based on your guidelines: i) This paper investigates strategies for generating tailored counterspeech using the LLaMA2-13B model, focusing on adaptation to conversation context and personalization to the user. ii) The main research question is whether contextualized counterspeech, adapted to the community and conversation and personalized to the user, is more persuasive than generic counterspeech. iii) The key methodology involved fine-tuning LLaMA2-13B with various configurations of contextual information (community, conversation, user history) and evaluating the generated counterspeech through quantitative indicators and a crowdsourced human evaluation. iv) The primary results show that contextualized counterspeech can outperform generic counterspeech in adequacy and persuasiveness; for instance, the configuration [Ba Pr Hi] outperformed the baseline in user-persuasiveness with a statistically significant difference (p < 0.01). v) The principal implication for AI practitioners is that incorporating contextual information like conversation history can significantly enhance the effectiveness of AI-generated counterspeech, though there exists a discrepancy between algorithmic and human evaluations of the output. |
| Maximizing Alignment with Minimal Feedback: Efficiently Learning Rewards for Visuomotor Robot Policy Alignment (Read more on arXiv or HuggingFace) | Jitendra Malik, Masayoshi Tomizuka, Chenfeng Xu, Yilin Wu, Ran Tian | Here is a concise summary of the research paper: i) Summary: The paper introduces Representation-Aligned Preference-based Learning (RAPL), an observation-only method for learning visual rewards from human preference feedback to align visuomotor robot policies. ii) Main research question or objective: How can visuomotor robot policies be aligned with end-user preferences using minimal human feedback? iii) Key methodology: RAPL focuses human feedback on fine-tuning pre-trained vision encoders to align with the end-user’s visual representation, then constructs a dense visual reward via feature matching using optimal transport in this aligned representation space. iv) Primary results: RAPL can fine-tune visuomotor policies with 5x less real human preference data compared to traditional reinforcement learning from human feedback (RLHF) methods. v) Principal implication for AI practitioners: AI practitioners can leverage RAPL to align pre-trained visuomotor policies with significantly less human feedback, making it more feasible to deploy such policies in real-world scenarios where collecting extensive human feedback is impractical. |
| Chimera: Improving Generalist Model with Domain-Specific Experts (Read more on arXiv or HuggingFace) | Renrui Zhang, Renqiu Xia, Hongbin Zhou, Mingsheng Li, Tianshuo Peng | Here is a concise summary of the research paper “Chimera: Improving Generalist Model with Domain-Specific Experts”: i) Summary: This paper introduces Chimera, a multi-modal pipeline that integrates domain-specific expert models into a generalist large multi-modal model (LMM) to enhance performance on specialized tasks. ii) Main research question or objective: How to effectively improve the performance of generalist LMMs on domain-specific tasks without sacrificing their general capabilities. iii) Key methodology: A progressive training strategy with a Generalist-Specialist Collaboration Masking (GSCM) mechanism was used to merge features from expert models into the input of a generalist LMM, along with a router to determine expert model invocation. iv) Primary results: Chimera achieved state-of-the-art performance on multi-modal reasoning benchmarks, with an overall accuracy of 64.9 on MathVista. v) Principal implication for AI practitioners: AI practitioners can leverage Chimera’s pipeline to scale up existing LMMs with domain-specific experts, significantly enhancing performance on specialized tasks without extensive retraining or compromising generalist capabilities. |
| A New Federated Learning Framework Against Gradient Inversion Attacks (Read more on arXiv or HuggingFace) | Weihong Ren, Xiaodan Zhang, Wenhao Chen, Shuang Zeng, gpx333 | Okay, here is a concise summary of the paper, strictly following your guidelines: i) This paper introduces HyperFL, a new federated learning framework designed to protect against gradient inversion attacks. ii) The main research objective is to develop a federated learning framework that offers a favorable privacy-utility trade-off against gradient inversion attacks without relying on existing defense mechanisms like SMC, HE, and DP. iii) The key methodology involves using hypernetworks to generate the parameters of local models, sharing only hypernetwork parameters for server aggregation, and decomposing local models into shared feature extractors and private classifiers. iv) Primary results show that HyperFL achieves comparable performance to state-of-the-art methods while enhancing privacy; for instance, HyperFL achieved 76.29% accuracy on the EMNIST dataset with 20 clients, surpassing several existing methods. v) The principal implication for AI practitioners is that HyperFL can be used as a more privacy-preserving alternative to traditional federated learning frameworks, particularly in applications where data sensitivity is a critical concern. |
Papers for 2024-12-10
| Title | Authors | Summary |
|---|---|---|
| ProcessBench: Identifying Process Errors in Mathematical Reasoning (Read more on arXiv or HuggingFace) | Keming Lu, Beichen Zhang, Zhenru Zhang, RunjiLin, chujiezheng | Here is a concise summary of the research paper “PROCESSBENCH: Identifying Process Errors in Mathematical Reasoning”: i) PROCESSBENCH is a new benchmark for evaluating the ability of language models to identify erroneous steps in mathematical reasoning. ii) The main research objective is to develop and evaluate a benchmark, PROCESSBENCH, for measuring the capability of models to identify the earliest erroneous step in mathematical reasoning solutions. iii) The key methodology involves curating a dataset of 3,400 mathematical problems with expert-annotated step-by-step solutions, and evaluating various process reward models (PRMs) and critic models (i.e., prompted general language models) on their ability to identify the first incorrect step. iv) The primary result is that the best open-source model, QwQ-32B-Preview, achieved an average F1 score of 71.5 across all subsets, demonstrating competitive performance with the proprietary model GPT-40 (61.9 F1 score) but lagging behind o1-mini (87.9 F1 score). v) The principal implication for AI practitioners is that existing PRMs generally fail to identify process errors in challenging math problems, while prompting large language models as critics shows promise, highlighting the need for better methods for scalable oversight of mathematical reasoning in AI systems. |
| Exploring Multi-Grained Concept Annotations for Multimodal Large Language Models (Read more on arXiv or HuggingFace) | Wanxiang Che, Libo Qin, Yuxi Xie, Tianhao Niu, LooperXX | Here is a concise summary of the AI research paper “Exploring Multi-Grained Concept Annotations for Multimodal Large Language Models” based on your specific guidelines: 1. Summary: This paper introduces MMGIC, a new multimodal dataset featuring multi-grained concept annotations, and demonstrates its effectiveness in improving the performance of Multimodal Large Language Models (MLLMs) on vision-language tasks. 2. Main Research Question/Objective: The main objective was to investigate whether integrating fine-grained concept annotations (e.g., object labels, attributes, and relationships) with coarse-grained annotations (e.g., image captions) can enhance MLLMs’ performance in multimodal comprehension and generation. 3. Key Methodology: The authors constructed the MMGIC dataset by integrating multi-grained concept annotations into image-text interleaved documents using a structured template and trained MLLMs with an autoregressive objective to predict the next visual or textual token in a multimodal sequence. They evaluate different data recipes and compare MMGIC with image-caption data. 4. Primary Results: Experiments showed that multi-grained concept annotations in MMGIC integrate and complement each other, leading to improved performance on 12 multimodal comprehension and generation benchmarks. For instance, the appropriate combination of MMGIC with image-caption data achieved a 3.95% absolute improvement over image-caption data alone on the POPE benchmark. 5. Principal Implication for AI Practitioners: AI practitioners can leverage the MMGIC dataset and the proposed training framework to develop MLLMs with enhanced capabilities in aligning vision and language at multiple granularities, leading to better performance on downstream vision-language tasks. |
| Training Large Language Models to Reason in a Continuous Latent Space (Read more on arXiv or HuggingFace) | Zhiting Hu, Xian Li, DiJia Su, Sainbayar Sukhbaatar, Shibo Hao | Here is a concise summary of the research paper: i) Summary: The paper introduces COCONUT, a novel paradigm that enables large language models (LLMs) to reason in a continuous latent space instead of the discrete language space. ii) Main research question or objective: Can LLMs reason more effectively in an unrestricted continuous latent space compared to the traditional language space? iii) Key methodology: COCONUT utilizes the last hidden state of the LLM as a “continuous thought” and feeds it back as the subsequent input embedding, training with a multi-stage curriculum that replaces language reasoning steps with continuous thoughts. iv) Primary results: COCONUT outperforms the Chain-of-Thought (CoT) method in certain logical reasoning tasks, achieving 97.0% accuracy on the ProsQA dataset compared to 77.5% for CoT. v) Principal implication for AI practitioners: AI practitioners can leverage COCONUT to develop LLMs with enhanced reasoning capabilities, especially for tasks requiring substantial planning and fewer inference tokens. |
| Divot: Diffusion Powers Video Tokenizer for Comprehension and Generation (Read more on arXiv or HuggingFace) | Ying Shan, Yixiao Ge, Yizhuo Li, Yuying Ge | Here is a concise summary of the paper “Divot: Diffusion Powers Video Tokenizer for Comprehension and Generation” based on your specified format: i) Summary: This paper introduces Divot, a diffusion-powered video tokenizer that learns spatiotemporal video representations for unified video comprehension and generation within a large language model (LLM). ii) Main research question/objective: To develop a video tokenizer that captures spatial and temporal video features, enabling LLMs to perform both video comprehension and generation. iii) Key methodology: A diffusion model is trained to de-noise video clips conditioned on the tokenizer’s spatiotemporal representations, thereby optimizing the tokenizer. The tokenizer is then integrated with a pre-trained LLM, Divot-LLM, to predict the parameters of a Gaussian Mixture Model (GMM) for modeling the distribution of continuous video features. iv) Primary results: Divot-LLM achieves competitive performance on video comprehension benchmarks; for example, it obtains a 76.4% accuracy on the MVBench video comprehension benchmark. v) Principal implication for AI practitioners: AI practitioners can leverage the proposed diffusion-based video tokenizer to build unified models for video understanding and generation tasks. |
| You See it, You Got it: Learning 3D Creation on Pose-Free Videos at Scale (Read more on arXiv or HuggingFace) | Tiejun Huang, Zhengxiong Luo, Haoge Deng, Infinite888, bruiiii | Okay, here is a concise summary of the research paper “You See it, You Got it: Learning 3D Creation on Pose-Free Videos at Scale”, strictly adhering to your guidelines: i) Summary: This paper introduces See3D, a visual-conditional multi-view diffusion model for 3D content creation trained on a large-scale dataset of internet videos without pose annotations. ii) Main research question or objective: How can we effectively learn 3D knowledge from large-scale Internet videos without explicit 3D geometry or camera pose annotations? iii) Key methodology: A four-step data curation pipeline was used to create WebVi3D dataset, and a novel visual-conditional multi-view diffusion model, See3D, was trained on this dataset using a time-dependent visual signal generated by adding noise to masked video data, thereby eliminating the need for pose conditions. iv) Primary results: See3D achieved a PSNR of 24.28 on the CO3D dataset for single-view reconstruction, outperforming models trained on constrained 3D datasets. v) Principal implication for AI practitioners: AI practitioners can leverage See3D to develop 3D generation models using large-scale, readily available video data without the need for costly 3D or pose annotations, significantly reducing the barriers to creating scalable 3D content generation systems. |
| Robust Multi-bit Text Watermark with LLM-based Paraphrasers (Read more on arXiv or HuggingFace) | Hang Li, Yang Liu, Yuanshun Yao, Jinghan Jia, xiaojunxu | Here is a concise summary of the research paper: i) Summary: This paper introduces a method for embedding multi-bit watermarks into text using fine-tuned, LLM-based paraphrasers and a trained decoder, achieving high detection accuracy and robustness. ii) Main research question/objective: How can a multi-bit watermark be robustly embedded into text while preserving its semantic meaning and remaining imperceptible? iii) Key methodology: The authors fine-tune a pair of LLM paraphrasers as encoders to inject watermark bits by alternatively paraphrasing text segments, and train an LLM-based text classifier as a decoder to extract the watermark. The encoder-decoder pair is co-trained using PPO-based reinforcement learning techniques. iv) Primary results: The proposed method achieves over 99.99% detection AUC with small (1.1B) text paraphrasers, outperforming existing methods. The watermark is evaluated as robust under word substitution and sentence paraphrasing perturbations. v) Principal implication for AI practitioners: AI practitioners can use this watermarking technique to embed robust and imperceptible multi-bit watermarks in text generated by language models, enabling applications such as copyright protection and tracking of misinformation. |
| CARP: Visuomotor Policy Learning via Coarse-to-Fine Autoregressive Prediction (Read more on arXiv or HuggingFace) | Mingyang Sun, Siteng Huang, Shangke Lyu, Pengxiang Ding, Zhefei Gong | Here is a concise summary of the research paper “CARP: Visuomotor Policy Learning via Coarse-to-Fine Autoregressive Prediction”: i) Summary: The paper introduces Coarse-to-Fine AutoRegressive Policy (CARP), a novel visuomotor policy learning paradigm that redefines the autoregressive action generation process as a coarse-to-fine, next-scale approach for robotic tasks. ii) Main research question/objective: Can a coarse-to-fine autoregressive approach achieve the high performance of diffusion-based models while maintaining the efficiency of traditional autoregressive models in visuomotor policy learning? iii) Key methodology: CARP decouples action generation into two stages: a multi-scale action autoencoder learns representations of the action sequence, and a GPT-style transformer refines the sequence prediction through a coarse-to-fine autoregressive process. iv) Primary results: CARP achieves competitive success rates on state-based and image-based simulation benchmarks and real-world tasks, delivering 10x faster inference compared to state-of-the-art policies. v) Principal implication for AI practitioners: AI practitioners can leverage CARP as a high-performance, efficient, and flexible framework for action generation in robotic tasks, offering a superior balance of performance and efficiency compared to existing methods. |
Papers for 2024-12-09
| Title | Authors | Summary |
|---|---|---|
| Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling (Read more on arXiv or HuggingFace) | Yangzhou Liu, Yue Cao, Zhe Chen, qishisuren, Weiyun1025 | Here’s a summary of the AI research paper following your strict guidelines: i) InternVL 2.5, an advanced multimodal large language model (MLLM), significantly improves open-source multimodal capabilities through model, data, and test-time scaling. ii) To systematically investigate the relationship between model scaling and performance in MLLMs, focusing on how scaling vision encoders, language models, dataset sizes, and inference times impact performance. iii) The study employed a three-stage training pipeline (MLP warmup, optional ViT incremental learning, and full model instruction tuning) combined with dynamic high-resolution training and data filtering techniques. iv) InternVL 2.5 achieved a 3.7-point improvement on the MMMU benchmark (reaching 70.1%) through Chain-of-Thought (CoT) reasoning. The paper also presents many other results across several benchmarks which are not summarized here. v) The significant performance improvement of InternVL 2.5 on MMMU and other benchmarks, especially its surpassing 70% accuracy on MMMU, demonstrates the potential for open-source MLLMs to rival commercial models and provides a strong open-source baseline for future multimodal AI development. Some aspects of the training methodology, such as specifics of the data filtering techniques, are not fully detailed. |
| LiFT: Leveraging Human Feedback for Text-to-Video Model Alignment (Read more on arXiv or HuggingFace) | Cheng Jin, Xiaomeng Yang, Junyan Wang, Zhiyu Tan, Yibin Wang | Here is a concise summary of the research paper “LiFT: Leveraging Human Feedback for Text-to-Video Model Alignment”: i) This paper introduces LiFT, a novel pipeline that utilizes human feedback to improve the alignment of text-to-video (T2V) models with human preferences. ii) Main research question or objective: How can human feedback be effectively leveraged to align T2V models with subjective human expectations regarding video quality and content? iii) Key methodology used: A three-stage pipeline is proposed: human feedback collection to create the LIFT-HRA dataset, training a reward model (LIFT-CRITIC) to predict human feedback scores and reasoning, and fine-tuning the T2V model using reward-weighted likelihood maximization. iv) Primary results: The fine-tuned CogVideoX-2B model using LIFT-CRITIC-40B outperforms the CogVideoX-5B baseline across all 16 metrics of the VBench benchmark. For instance, in the “Object Class” category, CogVideoX-2B-LIFT (40B) achieves a score of 91.77, compared to CogVideoX-5B’s score of 88.99. v) Principal implication for AI practitioners: AI practitioners can use the LiFT pipeline and the LIFT-HRA dataset to improve the alignment of T2V models by incorporating human feedback, but the paper does not specify how generalizable this method is to other T2V models. |
| MAmmoTH-VL: Eliciting Multimodal Reasoning with Instruction Tuning at Scale (Read more on arXiv or HuggingFace) | Yuelin Bai, Tuney Zheng, Jarvis Guo, yuexiang96, luodian | Here’s a summary of the AI research paper following your specified guidelines: i) 1-line summary: MAmmoTH-VL, a novel multimodal instruction-tuning dataset constructed using open-source models, significantly improves multimodal reasoning capabilities in large language models (LLMs). ii) Main research question or objective: How can a scalable and cost-effective method be developed to create a large-scale multimodal instruction-tuning dataset that elicits chain-of-thought (CoT) reasoning, thus improving the reasoning capabilities of open-source MLLMs? iii) Key methodology used: A three-step pipeline: (1) collecting and categorizing open-source multimodal data; (2) augmenting and rewriting tasks using open-source LLMs/MLLMs to elicit CoT reasoning; (3) self-filtering the data using an open-source MLLM to ensure data quality. iv) Primary results: Training an 8B parameter MLLM on the resulting 12M instruction-response pairs yielded an 8.1% improvement on the MathVerse benchmark compared to the previous open-source state-of-the-art. v) Principal implication for AI practitioners: The study provides a cost-effective and scalable methodology for building high-quality, rationale-enriched multimodal datasets using only open-source tools, significantly advancing the development and application of open-source MLLMs. The substantial performance gains demonstrate the importance of high-quality, CoT-style instruction data for enhancing reasoning capabilities in MLLMs. |
| EXAONE 3.5: Series of Large Language Models for Real-world Use Cases (Read more on arXiv or HuggingFace) | Kyunghoon Bae, Soyoung An, LG AI Research, lhg912, Sunkyoung | Here is a summary of the AI research paper following your specified guidelines: i) This technical report introduces EXAONE 3.5, a series of instruction-tuned large language models (LLMs) with varying parameter sizes (2.4B, 7.8B, and 32B) designed for real-world applications. ii) The main objective is to develop and release a series of LLMs addressing user feedback regarding the need for smaller, efficient models deployable on low-resource devices and larger models with enhanced real-world performance capabilities, including superior instruction following and long-context processing. iii) The key methodology involved pre-training on a massive corpus followed by instruction tuning and preference optimization, including decontamination to remove test-set examples from training data. Long-context capability was improved using a long-context fine-tuning method. iv) EXAONE 3.5 models achieved the highest scores across seven benchmarks for real-world instruction following; one specific finding is the 2.4B model outperformed similarly sized baselines across all three evaluation categories. v) The most impactful finding, the superior performance of the smaller 2.4B model, offers implications for AI practitioners by demonstrating cost-effective and high-performing sLLMs, meeting industry demand for models suitable for on-device deployment and resource-constrained environments. The study’s methodology for improving long-context processing also offers insight into improving LLMs. |
| Moto: Latent Motion Token as the Bridging Language for Robot Manipulation (Read more on arXiv or HuggingFace) | Mingyu Ding, Yixiao Ge, Yizhuo Li, Yuying Ge, Yi Chen | Here’s a concise summary of the research paper “Moto: Latent Motion Token as the Bridging Language for Robot Manipulation”: i) Summary: This paper introduces Moto, a novel framework that utilizes latent motion tokens for autoregressive pre-training on videos to enhance robot manipulation learning. ii) Main research question or objective: Can a generative pre-training approach using latent motion tokens, derived from video data, effectively enhance robot learning for manipulation tasks? iii) Key methodology: Moto employs a Latent Motion Tokenizer to convert video content into sequences of latent motion tokens and pre-trains Moto-GPT via next motion token prediction, followed by a co-fine-tuning strategy to bridge motion priors and real robot control. iv) Primary results: Moto outperforms baseline models on the SIMPLER and CALVIN benchmarks; notably, on SIMPLER, Moto achieved an overall success rate of 0.614, surpassing larger models like RT-2-X and OpenVLA. v) Principal implication for AI practitioners: AI practitioners can leverage Moto’s pre-training approach on readily available video datasets to enhance the performance of robot manipulation policies, especially in scenarios with limited action-labeled data. |
| APOLLO: SGD-like Memory, AdamW-level Performance (Read more on arXiv or HuggingFace) | Sem Park, Xi Liu, Wenyan Cong, Hanqing Zhu, Kyriection | Here is a concise summary of the research paper “APOLLO: SGD-like Memory, AdamW-level Performance”: i) Summary: The paper introduces APOLLO, a memory-efficient optimizer for large language model (LLM) training that achieves performance comparable to AdamW while significantly reducing memory usage. ii) Main research question or objective: Can structured learning rate adaptation be converted into a practical, memory-efficient optimization method for LLM training? iii) Key methodology: APOLLO approximates channel-wise or tensor-wise gradient scaling factors using an auxiliary low-rank space based on random projections, eliminating the need for costly SVD operations. iv) Primary results: APOLLO consistently outperforms AdamW in pre-training experiments across various LLaMA model sizes, achieving up to a 2.8 reduction in validation perplexity, and enables 3x throughput on an 8xA100-80GB setup compared to AdamW. v) Principal implication for AI practitioners: APOLLO allows AI practitioners to train LLMs more efficiently by drastically reducing optimizer memory overhead, enabling larger batch sizes, improved model scalability, and training on lower-end GPUs. |
| SwiftEdit: Lightning Fast Text-Guided Image Editing via One-Step Diffusion (Read more on arXiv or HuggingFace) | Cuong Pham, Anh Tran, Khoi Nguyen, Quang Nguyen, Tung11 | Here’s a concise summary of the research paper “SwiftEdit: Lightning Fast Text-Guided Image Editing via One-Step Diffusion,” following your specified guidelines: i) Summary: SwiftEdit is a text-guided image editing tool that achieves editing via a one-step diffusion process. ii) Main research question/objective: Develop an efficient method for instant text-guided image editing that overcomes the speed limitations of existing multi-step diffusion-based methods. iii) Key methodology: A one-step inversion framework for image reconstruction and a mask-guided editing technique with attention rescaling for localized editing are proposed. The inversion framework uses a two-stage training strategy using synthetic and real images. iv) Primary results: SwiftEdit achieves text-guided image editing in 0.23 seconds, which is at least 50 times faster than previous multi-step methods while maintaining competitive editing quality. v) Principal implication for AI practitioners: SwiftEdit offers a highly efficient tool for instant text-guided image editing, enabling faster performance in real-world applications without the need for users to define masks. |
| GenMAC: Compositional Text-to-Video Generation with Multi-Agent Collaboration (Read more on arXiv or HuggingFace) | Yu Wang, Xuefei Ning, Yukun Huang, fjxmlzn, NinaKarine | Here is a concise summary of the research paper “GenMAC: Compositional Text-to-Video Generation with Multi-Agent Collaboration”: i) GENMAC is a multi-agent framework for compositional text-to-video generation that uses an iterative process with DESIGN, GENERATION, and REDESIGN stages. ii) The main research objective is to develop a system that can generate videos adhering to complex compositional text prompts involving multiple objects, attributes, and dynamic actions. iii) The key methodology involves decomposing the REDESIGN stage into sequential tasks (verification, suggestion, correction, and output structuring) handled by specialized MLLM-based agents, and using a self-routing mechanism to select the appropriate correction agent. iv) GENMAC achieved a 0.5166 G-Dino score on the generative numeracy subset of the T2V-CompBench benchmark, outperforming all baselines. v) For AI practitioners, GENMAC offers a framework for enhancing compositional text-to-video generation by leveraging multi-agent collaboration and iterative refinement, demonstrating a method to improve alignment between generated video content and complex textual descriptions. |
| Mind the Time: Temporally-Controlled Multi-Event Video Generation (Read more on arXiv or HuggingFace) | Yuwei Fang, Ivan Skorokhodov, Willi Menapace, Aliaksandr Siarohin, Ziyi Wu | Here is a summary of the paper “Mind the Time: Temporally-Controlled Multi-Event Video Generation” following your guidelines: i) Summary: This paper introduces MinT, a novel video generation model capable of producing multi-event videos with precise temporal control over each event. ii) Main research question/objective: How can AI models generate videos with multiple, temporally distinct events, each with specified start and end times, using individual text prompts? iii) Key methodology: MinT utilizes a temporally-grounded video diffusion transformer with a time-based positional encoding method called ReRoPE to bind each event to its specific time period, enabling time-aware cross-attention between event captions and video tokens. iv) Primary results: MinT outperforms existing open-source video generation models in multi-event video generation, achieving a text-to-video alignment score of 3.00 on the StoryBench dataset, compared to 2.83 for the next best model (MEVG). v) Principal implication for AI practitioners: AI practitioners can leverage MinT to generate videos with multiple events and precise temporal control, enabling more sophisticated and realistic video content creation. |
| 2DGS-Room: Seed-Guided 2D Gaussian Splatting with Geometric Constrains for High-Fidelity Indoor Scene Reconstruction (Read more on arXiv or HuggingFace) | Xiansong Lai, Haodong Xiang, Crayon-Shinchan, ChaosLiao, Valentina-Zhang | Here is a concise summary of the research paper “2DGS-Room: Seed-Guided 2D Gaussian Splatting with Geometric Constraints for High-Fidelity Indoor Scene Reconstruction”: i) Summary: This paper introduces 2DGS-Room, a novel method for high-fidelity indoor scene reconstruction using 2D Gaussian Splatting with a seed-guided mechanism and geometric constraints. ii) Main research question or objective: The main objective is to develop a method for accurate and high-fidelity geometric reconstruction of indoor scenes. iii) Key methodology used: The key methodology involves a seed-guided mechanism to control the distribution of 2D Gaussians, adaptive growth and pruning of seed points, incorporation of monocular depth and normal priors, and multi-view consistency constraints. iv) Primary results: The method achieves state-of-the-art performance in indoor scene reconstruction on the ScanNet and ScanNet++ datasets; quantitatively, the 2DGS-Room achieves an F-score of 0.464 on the ScanNet++ dataset. v) Principal implication for AI practitioners: AI practitioners can utilize 2DGS-Room for improved 3D reconstruction of indoor scenes, leveraging its seed-guided 2D Gaussian Splatting approach for enhanced accuracy in applications like virtual reality and robotics. |
| DEMO: Reframing Dialogue Interaction with Fine-grained Element Modeling (Read more on arXiv or HuggingFace) | Haiyang Yu, Nan Xu, Kun Chen, Xinghua Zhang, iiiiwis | Here is a summary of the AI research paper “DEMO: Reframing Dialogue Interaction with Fine-grained Element Modeling” following your specified guidelines: i) This paper introduces DEMO, a benchmark for Dialogue Element Modeling, encompassing element awareness and dialogue agent interaction, to evaluate large language models’ (LLMs) ability to understand and generate dialogues. ii) The main research objective is to develop a comprehensive framework and benchmark for modeling fine-grained dialogue elements across the entire dialogue lifecycle (prelude, interlocution, and epilogue). iii) The key methodology involves a novel data synthesis framework that distills goals, scenes, and personas, generates dialogues using advanced LLMs, and performs quality control through LLM-based annotation and human verification. They also trained a DEMO agent based on imitation learning. iv) The primary results show that while advanced LLMs like GPT-4o demonstrate strong performance, there is still significant room for improvement in dialogue element modeling, with the DEMO agent built on LLaMA achieving a SOTA element awareness score of 6.008. v) The principal implication for AI practitioners is that the DEMO benchmark and the associated agent provide a valuable tool for developing and evaluating LLMs with enhanced capabilities in understanding and generating nuanced, element-driven dialogue, particularly in social intelligence generalization. |
Papers for 2024-12-06
| Title | Authors | Summary |
|---|---|---|
| Code-as-Monitor: Constraint-aware Visual Programming for Reactive and Proactive Robotic Failure Detection (Read more on arXiv or HuggingFace) | Zhongyuan Wang, Zhizheng Zhang, Qi Su, chengchi, Zhoues | Code-as-Monitor (CaM) uses a vision-language model to generate code that monitors for and prevents robot failures in real time. The research aims to create a unified system for both reactive (detecting failures after they occur) and proactive (preventing foreseeable failures) open-set failure detection in robotic tasks. The key methodology involves formulating robotic failure detection as a constraint satisfaction problem, using visually-prompted code to monitor if these constraints are met during task execution. In simulated “Stack in Order” tasks with severe disturbances, CaM achieved a 17.5% higher success rate than the DoReMi baseline. This allows AI practitioners to build more robust and reliable closed-loop robotic systems capable of handling unexpected events and complex, long-horizon tasks. |
| Aguvis: Unified Pure Vision Agents for Autonomous GUI Interaction (Read more on arXiv or HuggingFace) | tianbaoxiexxx, ludunjie, ZeonLap, kugwzk, ranpox | AGUVIS is a unified, pure vision-based framework for building generalizable GUI agents. The research aimed to develop a cross-platform autonomous GUI agent capable of performing complex tasks independently without relying on external closed-source models. The key methodology involved a two-stage training pipeline using a Vision-Language Model (VLM): first for GUI grounding on a newly created template-augmented dataset, followed by planning and reasoning training on a VLM-augmented trajectory dataset. AGUVIS-72B achieved a task success rate of 89.2% on ScreenSpot, outperforming previous state-of-the-art methods in both offline and real-world online scenarios. This indicates a significant advancement towards creating fully autonomous, vision-based GUI agents, offering AI practitioners a potentially more efficient and adaptable solution for automating interactions with diverse digital environments compared to text-based or LLM-dependent approaches. |
| A Noise is Worth Diffusion Guidance (Read more on arXiv or HuggingFace) | Minjae Kim, Sanghyun Lee, Jiwon Kang, Donghoon Ahn, Min-Jaewon | NoiseRefine improves text-to-image diffusion model quality without guidance methods like classifier-free guidance (CFG). The research explores whether guidance can be replaced by refining initial noise in the diffusion pipeline. The authors train a noise refining model using multistep score distillation (MSD) to map standard Gaussian noise to a learned “guidance-free” noise space, derived from inverting guided high-quality images. Refined noise achieved FID scores comparable to, and in some cases better than, CFG guidance. This method offers AI practitioners a faster and potentially higher-quality alternative to computationally expensive guidance methods for text-to-image diffusion models. |
| Evaluating Language Models as Synthetic Data Generators (Read more on arXiv or HuggingFace) | Seongyun Lee, Vijay Viswanathan, Xiang Yue, Juyoung Suk, seungone | AGORABENCH benchmarks language models’ (LMs) abilities to generate synthetic training data for other LMs. The research aimed to evaluate different LMs as synthetic data generators and understand the characteristics of effective training data generated by LMs. The study employed a controlled setting where various LMs generated 1.26 million training instances using existing data generation methods (instance generation, response generation, quality enhancement) across three domains (math, instruction-following, code), which were then used to fine-tune a student LM (Llama 3.1-8B). GPT-40 achieved the highest average Performance Gap Recovered (PGR) score of 46.8% in instance generation. AI practitioners can utilize AGORABENCH to select appropriate LMs for synthetic data generation based on the specific task and available resources, considering that problem-solving ability does not directly correlate with data generation effectiveness. |
| MV-Adapter: Multi-view Consistent Image Generation Made Easy (Read more on arXiv or HuggingFace) | Ran Yi, Haoran Wang, pookiefoof, bennyguo, huanngzh | MV-Adapter is a plug-and-play adapter enabling pre-trained text-to-image (T2I) diffusion models to generate multi-view consistent images. The objective is to efficiently generate multi-view consistent images while preserving the quality and knowledge of pre-trained T2I models, without full fine-tuning. The key methodology involves duplicating and parallelizing the self-attention layers of the base T2I model to create separate multi-view and image cross-attention layers within the adapter. On camera-guided image-to-multiview generation on the GSO dataset, MV-Adapter achieved 22.131 PSNR (Peak Signal-to-Noise Ratio) with SDXL. This allows AI practitioners to efficiently adapt existing high-quality T2I models for multi-view generation at high resolutions, reducing computational costs and mitigating overfitting risks associated with full model fine-tuning. |
| Negative Token Merging: Image-based Adversarial Feature Guidance (Read more on arXiv or HuggingFace) | Yejin Choi, Ranjay Krishna, Weijia Shi, Lindsey Li, Jaskirat Singh | NegToMe is a training-free method for adversarial guidance in text-to-image diffusion models using reference images. The research aimed to improve adversarial guidance beyond text-based negative prompts by leveraging visual features. The core methodology involves semantically matching and extrapolating source image tokens from their closest counterparts in a reference image during the reverse diffusion process. NegToMe improved output diversity (lower DreamSim score and higher Entropy) while maintaining or improving image quality (FID and IS) across different classifier-free guidance scales. This provides AI practitioners with a simple, efficient technique to enhance control and diversity of generated images using directly image-based references, overcoming limitations of purely text-based negative prompts. |
| Densing Law of LLMs (Read more on arXiv or HuggingFace) | Xu Han, Guoyang Zeng, Weilin Zhao, Jie Cai, xcjthu | Here’s a summary of the AI research paper “Densing Law of LLMs” following the provided guidelines: i) 1-line summary: An empirical law, termed the “Densing Law,” describes the exponential growth of Large Language Model (LLM) capacity density over time. ii) Main research question or objective: To introduce the concept of “capacity density” as a metric for evaluating LLM training quality, considering both effectiveness and efficiency, and to analyze the trend of LLM capacity density. iii) Key methodology used: Capacity density was defined as the ratio of a model’s effective parameter size (minimum parameters needed for equivalent performance) to its actual parameter size. This was estimated using a two-step process: first, fitting a Scaling Law to language modeling loss, and second, fitting a function to relate loss to downstream task performance. Open-source base LLMs released since 2023 were evaluated against five benchmarks. iv) Primary results (include one specific quantitative finding): The maximum capacity density of LLMs doubles approximately every 3.3 months. v) Principal implication for AI practitioners: The Densing Law suggests that achieving comparable performance to state-of-the-art LLMs using significantly fewer parameters is possible within a timeframe of approximately three months, thereby emphasizing the importance of optimizing LLM capacity density for improved efficiency and reduced computational costs in future LLM development. |
| Florence-VL: Enhancing Vision-Language Models with Generative Vision Encoder and Depth-Breadth Fusion (Read more on arXiv or HuggingFace) | Dianqi Li, Haiping Wu, Jianwei Yang, Jiuhai Chen, zhoutianyi | Florence-VL enhances multimodal large language models (MLLMs) using the generative vision model Florence-2. The research aimed to improve vision-language alignment and performance on diverse multimodal tasks by leveraging Florence-2’s enriched visual representations. The key methodology involved a novel “Depth-Breadth Fusion” (DBFusion) that combines visual features extracted from different layers and under multiple prompts of Florence-2, projecting these fused features into a pretrained LLM. Florence-VL 8B achieved 89.9% on MMBench (EN) compared to 67.9% for LLaVA next 8B, demonstrating significant improvements across various benchmarks. This implies that AI practitioners can leverage generative vision models like Florence-2 and fusion techniques like DBFusion to build more robust and versatile MLLMs for tasks requiring detailed image understanding. |
| Infinity: Scaling Bitwise AutoRegressive Modeling for High-Resolution Image Synthesis (Read more on arXiv or HuggingFace) | Yuqi Zhang, Bin Yan, Yi Jiang, Jinlai Liu, Jian Han | Infinity introduces bitwise modeling for autoregressive high-resolution image synthesis. The research aimed to improve the scaling and visual detail representation of discrete generative models for text-to-image synthesis. The core methodology involved a bitwise multi-scale visual tokenizer, an infinite-vocabulary classifier, and a bitwise self-correction mechanism within a visual autoregressive model. On the GenEval benchmark, Infinity achieved an overall score of 0.73, surpassing the SD3-Medium score of 0.62. This work suggests that scaling tokenizer vocabulary and incorporating bitwise modeling can significantly enhance autoregressive models for image generation, providing AI practitioners with a faster, more detailed, and potentially superior alternative to diffusion-based models. |
| Towards Universal Soccer Video Understanding (Read more on arXiv or HuggingFace) | Yanfeng Wang, Ya Zhang, Hao Jiang, haoningwu, Homie0609 | This paper introduces a new framework for multi-modal soccer video understanding. The objective is to develop a comprehensive model adaptable to various soccer video understanding tasks. The researchers constructed SoccerReplay-1988, a dataset of 1,988 soccer matches with rich annotations, and trained MatchVision, a visual-language foundation model, using supervised classification and video-language contrastive learning. MatchVision achieved 80.1% top-1 accuracy on event classification on the SoccerReplay-test benchmark. This work provides AI practitioners with a new dataset and a foundation model for developing more versatile and robust soccer video understanding applications, potentially enabling advancements in automated sports analysis and content generation. |
| HumanEdit: A High-Quality Human-Rewarded Dataset for Instruction-based Image Editing (Read more on arXiv or HuggingFace) | Juncheng Li, Xiangtai Li, Ling Yang, WeiChow, BryanW | HumanEdit is a human-rewarded dataset for instruction-based image editing. The objective was to create a high-quality dataset aligned with human preferences for training and evaluating instruction-guided image editing models, addressing limitations of existing datasets like noisy instructions and low-resolution images. The dataset was created through a four-stage pipeline involving annotator training, image selection, instruction and edited image generation using DALL-E 2, and a two-tiered human quality review process. On the HumanEdit-core subset, the mask-free InstructPix2Pix model achieved a CLIP-I score of 0.8946, while the mask-provided Meissonic model achieved a CLIP-I score of 0.9348. The paper presents quantitative results for multiple baselines across different editing types (add, remove, replace, etc.) but doesn’t explicitly compare them or declare a “best” overall. AI practitioners can use HumanEdit to train and benchmark instruction-based image editing models, especially for high-resolution, photorealistic editing tasks that better align with human expectations than previous datasets. The availability of masks, along with a subset allowing mask-free editing, allows for more flexible and diverse model training and evaluation. |
| Personalized Multimodal Large Language Models: A Survey (Read more on arXiv or HuggingFace) | Zhehao Zhang, Yu Xia, Hanjia Lyu, Junda Wu, Franck-Dernoncourt | This paper surveys techniques for personalizing multimodal large language models (MLLMs). The objective is to categorize and analyze existing methods for adapting MLLMs to individual user preferences across various modalities (text, image, audio, etc.). The authors propose a taxonomy classifying personalization techniques based on instruction, alignment, generation, and fine-tuning across different MLLM applications like text/image generation, recommendation, and retrieval. While specific quantitative results are inconsistently reported across surveyed works, the paper notes ConCon-Chi dataset contains 4008 images and 20 concepts within 101 contexts for evaluating personalized vision-language tasks. AI practitioners can use this taxonomy to understand the landscape of MLLM personalization techniques and identify suitable approaches for specific applications, though further research on standardized evaluation metrics and benchmark datasets is needed. |
| ZipAR: Accelerating Autoregressive Image Generation through Spatial Locality (Read more on arXiv or HuggingFace) | Hong Zhou, Shaoxuan He, Yuanyu He, Feng Chen, Yefei He | ZipAR is a training-free, plug-and-play parallel decoding framework for accelerating auto-regressive visual generation. The research aims to reduce the latency of auto-regressive image generation models which typically decode visual tokens sequentially. ZipAR leverages the spatial locality of images by decoding tokens from different rows in parallel, based on a defined local window size. Experiments demonstrated up to a 91% reduction in forward steps on the Emu3-Gen model with minimal impact on image quality. This allows AI practitioners to significantly accelerate auto-regressive visual generation without retraining or architectural modifications. |
| MRGen: Diffusion-based Controllable Data Engine for MRI Segmentation towards Unannotated Modalities (Read more on arXiv or HuggingFace) | Yanfeng Wang, Weidi Xie, Ya Zhang, Ziheng Zhao, haoningwu | MRGen synthesizes training data for MRI segmentation models targeting modalities without existing mask annotations. The research aims to improve MRI segmentation model performance on unannotated modalities due to the cost and scarcity of annotated data. A two-stage training process involves text-guided pretraining on a large radiology image-text dataset (MedGen-1M) followed by mask-conditioned fine-tuning. On average, MRGen improved Dice Similarity Coefficient (DSC) scores by 25% compared to models trained on source-domain data only. This provides AI practitioners with a method to extend existing segmentation models to new MRI modalities without needing manually annotated data, potentially accelerating development and deployment of robust medical image analysis tools. |
| Discriminative Fine-tuning of LVLMs (Read more on arXiv or HuggingFace) | Ioannis Maniadis Metaxas, Anestis Zaganidis, Alexandros Xenos, Adrian Bulat, Yassine Ouali | This paper introduces VladVA, a novel framework for adapting generative Large Vision-Language Models (LVLMs) for discriminative vision-language tasks. The objective is to enhance LVLMs’ discriminative capabilities while preserving their compositional strengths, addressing the limitations of contrastively-trained VLMs and autoregressive LVLMs. The key methodology involves fine-tuning LVLMs with both contrastive and next-token prediction losses on image-text pairs of variable lengths, combined with parameter-efficient adaptation using soft prompting and LoRA. On Flickr30k, VladVA achieves 85.0% recall@1 for image retrieval, a 5.5% absolute improvement over the baseline LLaVA 1.5-7B model. This work provides AI practitioners with a method to leverage the strengths of generative LVLMs for discriminative tasks like image-text retrieval, potentially leading to more robust and nuanced multimodal systems. |
| Global MMLU: Understanding and Addressing Cultural and Linguistic Biases in Multilingual Evaluation (Read more on arXiv or HuggingFace) | Jian Gang Ngui, David I. Adelani, Clémentine Fourrier, Angelika Romanou, Shivalika Singh | This paper investigates cultural and linguistic biases in the Massive Multitask Language Understanding (MMLU) benchmark and proposes an improved multilingual version. The research aims to understand how cultural biases in translated datasets influence the performance of multilingual language models and to improve the quality of these datasets. A large-scale evaluation of state-of-the-art language models was conducted using subsets of questions annotated as either culturally sensitive or culturally agnostic, alongside an improved, 42-language translated MMLU dataset called Global-MMLU. Analysis found that 28% of the English MMLU questions require culturally sensitive knowledge, with 86.5% of culturally sensitive questions focused on Western culture. AI practitioners should use Global-MMLU and report performance on culturally sensitive and agnostic subsets separately to better understand model capabilities across diverse cultures and languages, and to avoid inadvertently setting multilingual evaluation standards aligned with a single cultural paradigm. |
| Monet: Mixture of Monosemantic Experts for Transformers (Read more on arXiv or HuggingFace) | Jaewoo Kang, Kee-Eung Kim, Young Jin Ahn, affjljoo3581 | Here is a summary of the AI research paper “Monet: Mixture of Monosemantic Experts for Transformers,” following the provided guidelines: i) One-line summary: The MONET architecture integrates sparse dictionary learning into Mixture-of-Experts (MoE) transformer training to achieve parameter-efficient scaling of monosemantic experts and enhance mechanistic interpretability. ii) Main research question/objective: How can the internal computations of large language models (LLMs) be made more interpretable by disentangling polysemantic features and scaling the number of experts in a parameter-efficient manner? iii) Key methodology: The MONET architecture uses a novel expert decomposition method within a Mixture-of-Experts framework, employing product key composition of experts to achieve a square root scaling of total parameters with respect to the number of experts. This is implemented via Horizontal and Vertical Decomposition approaches. iv) Primary results: MONET achieves competitive performance with total parameter-matched dense LLMs on various benchmarks; MONET-VD (Vertical Decomposition) consistently outperforms MONET-HD (Horizontal Decomposition) across benchmarks and model sizes. Specific quantitative results from open-ended LLM benchmarks are provided in Table 2 of the paper. v) Principal implication for AI practitioners: The parameter-efficient scaling of monosemantic experts in MONET enables the creation of highly interpretable LLMs with a significantly increased number of experts. This facilitates robust knowledge manipulation (e.g., domain, language, toxicity control) without sacrificing overall model performance. The methodology offers a novel approach to scaling MoE architectures with enhanced interpretability and control. |
| OmniFlow: Any-to-Any Generation with Multi-Modal Rectified Flows (Read more on arXiv or HuggingFace) | Yusuke Kato, Zichun Liao, Akash Gokul, Konstantinos Kallidromitis, Shufan Li | OmniFlow is a novel generative AI model for any-to-any multi-modal generation. The research aimed to develop a unified model capable of generating various output modalities (text, image, audio) given any input modality combination. The core methodology involves extending rectified flows (RF) to a multi-modal setting, integrating a multi-modal guidance mechanism within a modular architecture inspired by Stable Diffusion 3. On the GenEval benchmark, OmniFlow achieves a score of 0.62 for text-to-image generation. This modular design, allowing for pretraining of individual components and subsequent merging, offers AI practitioners a more efficient and resource-conscious approach to developing and training unified multi-modal generative models, potentially reducing computational overhead compared to training large unified models from scratch. |
| AnyDressing: Customizable Multi-Garment Virtual Dressing via Latent Diffusion Models (Read more on arXiv or HuggingFace) | Zhichao Liao, Fulong Ye, Pengze Zhang, Qichao Sun, Crayon-Shinchan | AnyDressing generates customized images of characters wearing multiple garments based on user-provided garments and text prompts. The research aims to address the limitations of existing virtual dressing methods that struggle with multi-garment combinations and text prompt fidelity. The proposed AnyDressing model uses two primary networks: GarmentsNet, with a Garment-Specific Feature Extractor for parallel encoding of garment textures, and DressingNet, with a Dressing-Attention mechanism and Instance-Level Garment Localization Learning for integrating features and preserving text-image consistency. On a multi-garment evaluation, AnyDressing achieves a CLIP-T score of 0.296, demonstrating improved text consistency. This provides AI practitioners with a more robust and controllable approach for generating virtual dressing images, enabling diverse combinations of attire and improved adherence to user-specified text prompts. |
| KV Shifting Attention Enhances Language Modeling (Read more on arXiv or HuggingFace) | Weipeng Chen, Bingning Wang, Wei Cheng, xumingyu16 | Here’s a concise summary of the AI research paper following your strict guidelines: i) 1-line summary: A novel KV shifting attention mechanism is proposed and empirically shown to improve language model training efficiency and performance, reducing the depth and width requirements of induction heads. ii) Main research question/objective: Can modifications to the transformer’s attention mechanism improve the efficiency and effectiveness of learning induction heads, thus enhancing language modeling performance? iii) Key methodology: A novel “KV shifting attention” mechanism was proposed, decoupling keys and values in the attention mechanism to reduce the structural requirements for depth and width needed for induction heads. This was theoretically analyzed and empirically validated through experiments on both toy and large-scale language models. iv) Primary results: The KV shifting attention demonstrated superior performance to conventional multi-layer transformers, with a 2.9B parameter model achieving an average benchmark score of 38.57 (compared to 36.45 for Vanilla) after 500B training tokens. Specific details regarding the toy model experiments (Figure 1a and 1b) were provided but lacked complete numerical representation in the main text. v) Principal implication for AI practitioners: KV shifting attention offers a method to potentially improve the efficiency of training large language models by reducing computational resources required for induction heads, leading to better performance or faster convergence. Further investigation is needed to assess the applicability and impact across a wider range of architectures and model sizes, and additional numerical results from the small-scale and large-scale experiments would improve the clarity and impact of the conclusions. |
| Marco-LLM: Bridging Languages via Massive Multilingual Training for Cross-Lingual Enhancement (Read more on arXiv or HuggingFace) | Yu Zhao, Tianqi Shi, Chenyang Lyu, Bo Zeng, Lingfeng Ming | Here is a summary of the AI research paper following your guidelines: i) Marco-LLM, a multilingual large language model (LLM), is developed using massive multilingual continual pre-training and post-training to bridge the performance gap between high- and low-resource languages. ii) The main objective is to develop a multilingual LLM that performs exceptionally well in multilingual tasks, including low-resource languages, while maintaining strong performance in high-resource languages like English. iii) The key methodology involves compiling a large-scale multilingual dataset, conducting two-stage continual pre-training using Qwen2 models, and performing extensive multilingual post-training including supervised fine-tuning and preference alignment. iv) Marco-LLM achieved substantial improvements over state-of-the-art LLMs in various multilingual benchmarks, for example, Marco-72B achieved a 93.7% accuracy on CEVAL and 81.2% accuracy on X-MMLU. v) The significant improvement in multilingual understanding and reasoning tasks across various benchmarks, especially for low-resource languages, highlights the efficacy of massive multilingual training and demonstrates the potential to improve LLM capabilities for under-resourced languages. Further investigation of continual learning parameters and data quality will be essential for future model iterations. |
Papers for 2024-12-05
| Title | Authors | Summary |
|---|---|---|
| SNOOPI: Supercharged One-step Diffusion Distillation with Proper Guidance (Read more on arXiv or HuggingFace) | Khoi Nguyen, anhttran1111, termanteus, aengusng, viettmab | SNOOPI enhances one-step text-to-image diffusion model training stability and control via novel guidance techniques. The research aimed to address the instability of Variational Score Distillation (VSD) across different architectures and the lack of negative prompt guidance in one-step diffusion models. The authors introduced Proper Guidance - SwiftBrush (PG-SB), which utilizes a random guidance scale during training, and Negative-Away Steer Attention (NASA), which integrates negative prompts during inference via cross-attention manipulation. Integrating PG-SB and NASA with a PixArt-a backbone achieved a Human Preference Score v2 (HPSv2) of 31.08. This offers AI practitioners a more stable and controllable method for developing efficient one-step text-to-image diffusion models with enhanced image quality and adherence to both positive and negative prompts. |
| Imagine360: Immersive 360 Video Generation from Perspective Anchor (Read more on arXiv or HuggingFace) | liuziwei7, guoyww, mimihe, tongwu2020, jingtan | Imagine360 generates immersive 360° videos from standard perspective videos. The research aimed to develop a framework for transforming perspective videos into 360° equirectangular videos. The core methodology involved a dual-branch video denoising structure with antipodal masking and elevation-aware design, trained on a combined dataset of WEB360 and a newly collected YouTube dataset. Imagine360 achieved a VQA score of 0.8672, outperforming comparison methods like 360DVD and Follow-Your-Canvas. This provides AI practitioners with a new tool for generating high-quality 360° videos from readily available perspective video data, facilitating easier creation of immersive content. |
| Distilling Diffusion Models to Efficient 3D LiDAR Scene Completion (Read more on arXiv or HuggingFace) | An Zhao, slysun, haoranxu, mengcy, SYZhang0805 | ScoreLiDAR, a novel distillation method, accelerates 3D LiDAR scene completion using diffusion models. The research aimed to improve the speed of diffusion-based 3D LiDAR scene completion while maintaining high quality. The method uses Variational Score Distillation (VSD) adapted for 3D data and incorporates a novel Structural Loss to preserve geometric details. On the SemanticKITTI dataset, ScoreLiDAR achieved a 5x speedup, reducing completion time from 30.55 seconds to 5.37 seconds per frame while improving Chamfer Distance by 8%. This allows AI practitioners to utilize diffusion models for real-time or near real-time 3D LiDAR scene completion in applications like autonomous driving where fast processing is crucial. |
| PaliGemma 2: A Family of Versatile VLMs for Transfer (Read more on arXiv or HuggingFace) | mjlm, AlexeyG, yonatanbitton, dkeysers, mitsch | Here’s a summary of the AI research paper following your strict guidelines: i) 1-line summary: PaliGemma 2, a family of versatile vision-language models (VLMs), was developed and evaluated on a broad range of transfer tasks, demonstrating improved performance over its predecessor. ii) Main research question/objective: To investigate the impact of model size and resolution on VLM transfer performance and expand the breadth of transfer tasks beyond those in the original PaliGemma. iii) Key methodology: A family of VLMs was created by combining the SigLIP-So400m vision encoder with various Gemma 2 language models (2B, 9B, and 27B), trained at three resolutions (224px², 448px², 896px²) using a three-stage training process. These models were then fine-tuned on a wide array of transfer tasks including several new tasks such as table and molecular structure recognition. iv) Primary results: PaliGemma 2 achieved state-of-the-art results on many transfer tasks; for example, on ICDAR’15 Incidental and Total-Text, it outperformed the previous state-of-the-art in text detection and recognition (HTS) achieving F1 scores of 75.9 and 74.2, respectively. v) Principal implication for AI practitioners: The release of PaliGemma 2 as open-weight models provides a resource for fine-tuning on various tasks, offering valuable insights into the impact of model scaling on transfer learning and state-of-the-art performance in several domains. The extensive analysis of model size and resolution’s effects on numerous tasks provides a valuable resource for model design choices in VLM development. The specific quantitative results on numerous benchmarks allow for direct comparison with existing models and informed decision-making in selecting appropriate models for various applications. |
| TokenFlow: Unified Image Tokenizer for Multimodal Understanding and Generation (Read more on arXiv or HuggingFace) | sweetrabor, gaozong, xuwang, liqingzju, leo1117 | TokenFlow is a novel unified image tokenizer designed to bridge the gap between multimodal understanding and generation. The central research question is whether a single image tokenizer can derive representations suitable for both multimodal understanding and generation. The key methodology involves a dual-codebook architecture that decouples semantic and pixel-level feature learning while maintaining alignment via shared index mapping, enabling simultaneous access to both feature types. In multimodal understanding benchmarks, TokenFlow surpasses LLaVA-1.5 13B by 7.2% average improvement, marking the first time discrete visual input outperforms this baseline. This improvement significantly impacts AI practitioners by providing a more efficient and performant approach to unify image representations for both understanding and generation tasks within a single framework. |
| Video-3D LLM: Learning Position-Aware Video Representation for 3D Scene Understanding (Read more on arXiv or HuggingFace) | asdfg80, slvjul, zd11024 | Video-3D LLM enhances 3D scene understanding by incorporating 3D positional information into video representations. The research aimed to develop a generalist model for various 3D scene understanding tasks, addressing the limitations of current MLLMs in handling 3D spatial information. The authors developed Video-3D LLM, which leverages a pre-trained Video LLM and integrates 3D position encodings derived from depth images into video features, along with a maximum coverage sampling strategy for efficient frame selection. The model achieved state-of-the-art performance on benchmarks like ScanRefer (58.1% Acc@0.25), Scan2Cap (41.3 BLEU-4@0.5IoU), ScanQA (30.1% EM), and SQA3D (58.6% EM). AI practitioners can utilize this approach to enhance performance in applications requiring 3D spatial reasoning, such as robotics, 3D visual grounding, and question answering. The improvement in accuracy on ScanRefer, by incorporating 3D positional data, highlights the practical benefit for developing more robust 3D scene understanding applications. |
| NVComposer: Boosting Generative Novel View Synthesis with Multiple Sparse and Unposed Images (Read more on arXiv or HuggingFace) | Chengwh, bluestyle97, Yw22, ZyZcuhk, l-li | NVComposer synthesizes novel views from multiple sparse and unposed images without requiring external alignment. The objective is to generate novel views at specified target camera poses from unposed conditional images without explicit pose estimation or pre-reconstruction. The approach uses an image-pose dual-stream diffusion model to generate views and implicitly predict poses, combined with a geometry-aware feature alignment adapter distilling geometric priors from a pre-trained dense stereo model. On the RealEstate10K dataset, NVComposer achieves a PSNR of 22.55 with four input views, outperforming comparison methods. This provides AI practitioners with a more robust and accessible method for generative novel view synthesis, eliminating the need for potentially unstable external alignment pre-processing. |
| VARCO-VISION: Expanding Frontiers in Korean Vision-Language Models (Read more on arXiv or HuggingFace) | SunYoung Park, Daeyoung Kim, kimyoungjune, hojunssss | VARCO-VISION is a novel open-source, Korean-English bilingual vision-language model (VLM). The research aimed to develop a high-performing bilingual VLM and accompanying Korean evaluation benchmarks. The authors employed a four-stage training strategy involving feature alignment pre-training, basic and advanced supervised fine-tuning, and preference optimization using translated and human-validated datasets. VARCO-VISION-14B achieved 82.21% accuracy on the K-MMBench benchmark, outperforming similarly sized open-source models. This release provides AI practitioners with a powerful tool for developing Korean-focused multimodal applications and resources for further research in bilingual VLM training and evaluation. |
| CleanDIFT: Diffusion Features without Noise (Read more on arXiv or HuggingFace) | Björn Ommer, FrankFundel, kolja-b, stefan-baumann, kliyer | CleanDIFT is a novel method for extracting noise-free, timestep-independent features from pre-trained diffusion models. The research aimed to improve the quality and efficiency of diffusion feature extraction by eliminating the need for adding noise to input images. The methodology involved fine-tuning a trainable copy of a diffusion model on clean images while aligning its internal representations with the timestep-dependent features of the original model using projection heads and a cosine similarity loss. On the SPair-71k dataset for zero-shot unsupervised semantic correspondence, CleanDIFT improved PCKbbox accuracy by 1.86 percentage points compared to standard diffusion features. AI practitioners can use CleanDIFT to extract superior, noise-free features from diffusion models more efficiently, eliminating the need for noise or timestep ensembling for various downstream tasks like semantic correspondence, depth estimation, and semantic segmentation. |
| MIDI: Multi-Instance Diffusion for Single Image to 3D Scene Generation (Read more on arXiv or HuggingFace) | zouzx, yhyang-myron, XingqiaoAn, bennyguo, huanngzh | MIDI generates compositional 3D scenes from single images by extending pretrained image-to-3D object generation models to multi-instance diffusion. The objective is to generate multiple spatially correlated 3D instances with accurate relationships from a single image. MIDI employs a novel multi-instance attention mechanism within a denoising transformer, trained on scene-level and single-object data, to model cross-instance interactions and spatial coherence directly during 3D generation. On the BlendSwap dataset, MIDI achieves a scene-level Chamfer Distance of 0.077 and F-Score of 78.21, outperforming other single-image 3D scene generation methods. AI practitioners can use MIDI to create coherent and high-fidelity 3D scenes from single images, potentially impacting applications like 3D content creation and scene understanding. |
| One Shot, One Talk: Whole-body Talking Avatar from a Single Image (Read more on arXiv or HuggingFace) | Boyang Guo, Leipeng Hu, JuyongZhang, YudongGuo, xiangjun-xj | This paper introduces a method for creating animatable, expressive, whole-body talking avatars from a single image. The objective is to reconstruct a 3D talking avatar from a single image that can be animated with realistic gestures and expressions. The method uses pose-guided image-to-video diffusion models to generate pseudo-labels and trains a coupled 3D Gaussian Splatting (3DGS)-mesh hybrid avatar representation with several regularizations. On a self-driven motion reenactment task, the method achieved a peak signal-to-noise ratio (PSNR) of 29.31, outperforming comparison methods. This provides AI practitioners with a new technique to create realistic and controllable talking avatars from limited input data, potentially impacting applications in virtual reality, augmented reality, and telepresence. |
| Mimir: Improving Video Diffusion Models for Precise Text Understanding (Read more on arXiv or HuggingFace) | Dandan Zheng, Kecheng Zheng, Yutong Feng, Shuai Tan, BiaoGong | Mimir is a novel text-to-video generation framework that enhances text comprehension in video diffusion models. The research aims to address the limited text understanding of current video diffusion models, especially when processing short captions or complex motions, by integrating the capabilities of large language models (LLMs). The key methodology involves a “token fuser” that harmonizes the outputs of text encoders and decoder-only LLMs, enabling the model to leverage both learned video priors and advanced text comprehension of LLMs. Mimir achieves 97.68% on Background Consistency in the VBench benchmark, outperforming all other compared models. This implies that AI practitioners can utilize Mimir’s architecture to improve video generation quality and text comprehension, particularly for short, complex prompts. |
| Weighted-Reward Preference Optimization for Implicit Model Fusion (Read more on arXiv or HuggingFace) | Xiaojun Quan, Tianyuan Shi, Longguang Zhong, Fanqi Wan, Ziyi Yang | The paper introduces Weighted-Reward Preference Optimization (WRPO) for fusing heterogeneous large language models (LLMs). The research aims to improve the capabilities of a target LLM by implicitly learning from multiple robust open-source LLMs without vocabulary alignment or distribution merging. WRPO uses a progressive adaptation strategy and weighted reward mechanism within a preference optimization framework, mitigating distributional deviations between source and target LLMs. When applied to LLaMA3-8B-Instruct, WRPO achieves a 55.9% length-controlled win rate against GPT-4-Preview-1106 on AlpacaEval-2. This provides AI practitioners with a more efficient and effective method for integrating strengths from various LLMs into a single model, potentially outperforming larger, computationally expensive ensembles. |
| NitroFusion: High-Fidelity Single-Step Diffusion through Dynamic Adversarial Training (Read more on arXiv or HuggingFace) | Yi-Zhe Song, Kai Zou, Hmrishav Bandyopadhyay, ChenDY | NitroFusion introduces a dynamic adversarial training framework for high-fidelity single-step text-to-image diffusion. The objective is to improve the quality of single-step diffusion models, which typically suffer from quality degradation compared to multi-step models, while maintaining speed advantages. The key methodology involves a dynamic discriminator pool with specialized and periodically refreshed discriminator heads, employing multi-scale and dual-objective (conditional/unconditional) GAN training. NitroFusion achieves an Aesthetic Score of 5.92 and an Image Reward of 0.991 on the COCO-5k validation dataset, exceeding its 8-step teacher model in these metrics. This offers AI practitioners a single model capable of both rapid generation and high-fidelity image synthesis, dynamically adjustable through bottom-up refinement with 1-4 denoising steps. |
Papers for 2024-12-04
| Title | Authors | Summary |
|---|---|---|
| VideoGen-of-Thought: A Collaborative Framework for Multi-Shot Video Generation (Read more on arXiv or HuggingFace) | cqf, tfl01, AI4VR, Jethro37, Cheliosoops | VideoGen-of-Thought (VGoT) is a training-free architecture for generating multi-shot, coherent videos. The research aimed to address the challenge of creating multi-shot videos that maintain narrative logic and visual consistency across different shots. VGoT employs a four-module pipeline: Script Generation, Keyframe Generation, Shot-Level Video Generation, and a novel cross-shot Smooth Mechanism using latent features and reset boundaries. VGoT achieved higher Face Consistency (FC) and Style Consistency (SC) scores, particularly across shots, compared to baseline models (0.2738 cross-shot FC score for VGoT vs. a maximum of 0.0686 for baselines). This provides AI practitioners with a novel method to enhance narrative coherence and cross-shot consistency in generated multi-shot videos, particularly improving transitions between shots for a more natural visual flow. |
| Critical Tokens Matter: Token-Level Contrastive Estimation Enhence LLM’s Reasoning Capability (Read more on arXiv or HuggingFace) | zptu, Thu-redrobot, SihengLi, Chufan, Jiahao004 | This paper introduces cDPO, a token-level contrastive preference optimization framework for enhancing LLM reasoning capabilities. The research investigates the impact of individual tokens, particularly “critical tokens,” on the outcomes of reasoning tasks. The core methodology involves contrastive estimation using separately trained positive and negative models on correct and incorrect reasoning trajectories, coupled with a token-level extension of Direct Preference Optimization (DPO). On the GSM8K benchmark, cDPO achieves an average accuracy of 77.2%, significantly outperforming baseline methods (p < 0.005). This result suggests that AI practitioners can leverage token-level contrastive estimation during preference optimization to improve the accuracy of LLMs on reasoning tasks, specifically by mitigating the negative impact of critical tokens. |
| Free Process Rewards without Process Labels (Read more on arXiv or HuggingFace) | iseesaw, stingning, ganqu, wendili, lievan | This paper introduces a method for deriving process reward models (PRMs) without step-level labels. The research aimed to reduce the cost and complexity of training PRMs compared to outcome reward models (ORMs) and existing PRM training methods. The core methodology involves parameterizing the outcome reward as the log-likelihood ratio of policy and reference language models and training an ORM on response-level data. Experiments on MATH showed that the resulting implicit PRM, when instantiated with cross-entropy loss, outperformed a strong MCTS baseline (Math-Shepherd) by 0.6% while using less than 1/38 of the training data. This implies that AI practitioners can obtain high-performing PRMs at substantially lower cost by leveraging response-level data and this specific reward parameterization, potentially simplifying the development and deployment of reward models for complex reasoning tasks. |
| AV-Odyssey Bench: Can Your Multimodal LLMs Really Understand Audio-Visual Information? (Read more on arXiv or HuggingFace) | shijiay, MoFanCheng, BreakLee, KaituoFeng, kxgong | This paper introduces AV-Odyssey Bench, a benchmark designed to evaluate audio-visual comprehension in Multimodal Large Language Models (MLLMs). The research investigates whether MLLMs genuinely understand audio-visual information, or if their performance relies on surface-level patterns. The benchmark employs 4,555 multiple-choice questions across 26 tasks requiring integration of text, image/video, and audio. On AV-Odyssey, the best-performing model, GPT-40 (audio caption method), achieved only 34.5% accuracy. This indicates current MLLMs struggle with complex audio-visual integration, highlighting a critical area for model and dataset improvement, particularly the integration of audio information within multi-modal contexts. |
| OmniCreator: Self-Supervised Unified Generation with Universal Editing (Read more on arXiv or HuggingFace) | Harry Yang, Lan Wang, sernam, Harold328 | Here’s a concise summary of the AI research paper following your specified guidelines: i) One-line summary: OmniCreator, a self-supervised framework, achieves unified image and video generation and universal text-guided editing by leveraging the original video as a denoising condition. ii) Main research question/objective: To develop a unified framework capable of both text-prompted image and video generation and universal text-guided editing, addressing limitations of existing methods focused on specific editing types or requiring additional controls. iii) Key methodology: A self-supervised approach using original text-video pairs as conditions, with the same video serving as a denoising target, combined with an adapter and query transformer for multimodal fusion and spatiotemporal low-rank adaptations (LoRA) for efficiency. iv) Primary results: OmniCreator exhibits substantial superiority over existing models, achieving an average overall user study score of 4.33 on OmniBench-99 for video editing, compared to scores ranging from 2.00 to 3.33 for other methods. v) Principal implication for AI practitioners: OmniCreator’s self-supervised approach and superior performance on a comprehensive video editing benchmark demonstrates the potential for significant advancements in controllable generative models, particularly regarding unified image/video processing and efficient, flexible editing capabilities. The paper lacks a detailed quantitative evaluation on a standardized image editing benchmark. |
| OCR Hinders RAG: Evaluating the Cascading Impact of OCR on Retrieval-Augmented Generation (Read more on arXiv or HuggingFace) | zichenwen, ouyanglinke, binwang, qintong21, Carkham | OHRBench, a new benchmark for evaluating the impact of OCR on Retrieval-Augmented Generation (RAG) systems, reveals that OCR noise degrades RAG performance. The research investigates how OCR noise affects RAG by creating a dataset of PDFs, ground truth structured data, Q&As, and perturbed data with varying OCR noise levels. The key methodology involves evaluating several OCR solutions and then systematically analyzing the impact of semantic and formatting noise on retrieval and generation components of RAG. Results show even the best OCR solution reduces end-to-end RAG F1-score by at least 2.93 points compared to ground truth, and semantic noise consistently degrades performance across different RAG components. AI practitioners developing RAG systems should prioritize mitigating OCR noise for optimal performance, particularly focusing on semantic accuracy. |
| Scaling Image Tokenizers with Grouped Spherical Quantization (Read more on arXiv or HuggingFace) | Jiangtao Wang, kessel666, briqnn, yifAI, Doreamonzzz | This paper introduces Grouped Spherical Quantization (GSQ) for training image tokenizers. The research aims to address limitations in current image tokenizers related to GAN-based hyperparameters, biased comparisons, and a lack of scaling analysis. GSQ employs spherical codebook initialization, lookup regularization, and latent decomposition to improve training and reconstruction quality. GSQ-GAN achieves a reconstruction FID (rFID) of 0.50 with 16x downsampling on ImageNet at 256x256 resolution. This research suggests that AI practitioners can achieve improved reconstruction quality and efficiency in image tokenizers using GSQ, especially for tasks involving high spatial compression. |
| LSceneLLM: Enhancing Large 3D Scene Understanding Using Adaptive Visual Preferences (Read more on arXiv or HuggingFace) | Sunxy111, Xiaomabufei, senfu, PeihaoChen, Hoyard | LSceneLLM enhances 3D scene understanding in large and complex environments. The research aimed to improve 3D Vision-Language Models’ (3D-VLMs) ability to locate task-relevant visual information in large 3D scenes. The authors developed LSceneLLM, a framework incorporating a coarse scene understanding module and a scene magnifier module that uses LLM’s visual preference for adaptive identification and detailed examination of relevant regions. LSceneLLM outperformed existing methods on the proposed XR-Scene cross-room understanding benchmark and other existing benchmarks; on XR-QA, LSceneLLM achieved a CIDER score of 117.21 compared to 112.80 for the next best method. AI practitioners can use the plug-and-play scene magnifier module to enhance existing 3D-VLMs for improved accuracy in tasks involving large and complex 3D scene understanding. |
| MaskRIS: Semantic Distortion-aware Data Augmentation for Referring Image Segmentation (Read more on arXiv or HuggingFace) | Dongyoon Han, Song Park, Seungho Lee, Minhyun Lee, bhheo | MaskRIS improves Referring Image Segmentation (RIS) by using a novel masking-based data augmentation strategy. The research aimed to develop a more effective data augmentation technique for RIS than conventional methods, which degrade performance due to semantic conflicts. The key methodology involves masking image and text inputs, combined with Distortion-aware Contextual Learning (DCL) to leverage both original and masked data. MaskRIS achieved state-of-the-art performance on RefCOCO, RefCOCO+, and RefCOCOg, increasing overall Intersection-over-Union (oIoU) scores by up to 2.25% compared to previous methods. This implies that AI practitioners working on RIS can significantly enhance model robustness and accuracy by incorporating the MaskRIS data augmentation framework into their training pipelines. |
| A dynamic parallel method for performance optimization on hybrid CPUs (Read more on arXiv or HuggingFace) | Liu Yucheng, Luo Yu, Haihao | This paper introduces a dynamic parallel method for optimizing Large Language Model (LLM) inference on hybrid CPUs. The research aims to address the low inference performance on hybrid CPUs caused by imbalanced hardware capabilities among cores. The proposed method dynamically balances the workload for each core before parallel work begins, integrating a new thread scheduler and CPU runtime with the Neural Speed framework. Results show a 20%-30% improvement in prefill phase latency compared to using OpenMP in Neural Speed, and over 90% of memory bandwidth utilization is achieved for INT4 GEMV on an Ultra-125H. This provides AI practitioners with a more efficient method for running LLM inference on hybrid CPUs, particularly relevant for client-side deployments where these processors are increasingly prevalent. |
| VideoLights: Feature Refinement and Cross-Task Alignment Transformer for Joint Video Highlight Detection and Moment Retrieval (Read more on arXiv or HuggingFace) | Nabeel Mohammed, Md Rizwan Parvez, shafin5, dpaul06 | VideoLights is a novel framework for jointly performing video highlight detection (HD) and moment retrieval (MR). The research aimed to improve joint HD/MR by addressing limitations in cross-task and cross-modal interactions in existing models. The framework utilizes a Feature Refinement and Alignment (FRA) module, Bi-Directional Cross-Modal Fusion (Bi-CMF) network, Unidirectional Joint-Task Feedback Mechanism (Uni-JFM), and leverages LVLMs like BLIP-2. On the QVHighlights dataset, VideoLights-B-pt achieved a state-of-the-art R@0.5 of 70.36% for moment retrieval. This research provides AI practitioners with a new state-of-the-art model and framework for developing more robust and effective video understanding systems for tasks like content management and recommendation. |
Papers for 2024-12-03
| Title | Authors | Summary |
|---|---|---|
| X-Prompt: Towards Universal In-Context Image Generation in Auto-Regressive Vision Language Foundation Models (Read more on arXiv or HuggingFace) | lindahua, TheYJ, yuhangzang, tongwu2020, Zery | X-Prompt enhances in-context image generation in auto-regressive vision-language models. The research aimed to improve auto-regressive VLM performance across diverse seen and unseen image generation tasks within a unified in-context learning framework. The key methodology involved compressing in-context example features into fixed-length tokens, unifying image generation and description tasks, and using a retrieval-augmented image editing strategy. On the GenEval benchmark, X-Prompt with text prediction improved overall text-to-image generation by 0.08 compared to the baseline Chameleon model. This research provides AI practitioners with a method for enhancing the generalizability and efficiency of auto-regressive VLMs in diverse image generation applications, by enabling effective in-context learning with shorter context lengths. |
| GATE OpenING: A Comprehensive Benchmark for Judging Open-ended Interleaved Image-Text Generation (Read more on arXiv or HuggingFace) | LiruiZhao, yefly, xuzhaopan, xiaopengpeng, lyuukuu | OpenING is a new benchmark for evaluating open-ended interleaved image-text generation. The research aimed to create a comprehensive benchmark and robust judge model for open-ended interleaved image-text generation. The authors curated a dataset of 5,400 human-annotated instances across 56 real-world tasks and developed a judge model, IntJudge, trained with a novel reference-augmented generation approach. IntJudge achieved an 82.42% agreement rate with human judgments, outperforming GPT-based evaluators by 11.34%. AI practitioners can use OpenING to evaluate and benchmark new interleaved generation models and IntJudge as a more robust automated evaluation tool compared to GPT-based judges. |
| Switti: Designing Scale-Wise Transformers for Text-to-Image Synthesis (Read more on arXiv or HuggingFace) | Dmitry Baranchuk, Valentin Khrulkov, Mikhail Khoroshikh, Anton Voronov, SpiridonSunRotator | SWITTI is a scale-wise transformer model for text-to-image synthesis designed for improved speed and quality. The research aimed to develop a faster, higher-quality text-to-image generation model using a scale-wise transformer architecture while investigating the role of autoregression and text conditioning across scales. The key methodology involved modifying a scale-wise autoregressive transformer architecture to improve training stability, removing the autoregressive component based on analysis of attention maps, and disabling classifier-free guidance at the highest resolution scales. SWITTI achieves comparable performance to state-of-the-art diffusion models on automated metrics and human evaluations while being up to 7x faster, with a single-step generation time of 9.5 milliseconds for a batch of 8 512x512 images on an NVIDIA A100 80GB GPU. The removal of the autoregressive component and disabling of classifier-free guidance at later stages significantly improved sampling speed while maintaining or slightly enhancing quality, offering practitioners a more efficient model for text-to-image generation. |
| Open-Sora Plan: Open-Source Large Video Generation Model (Read more on arXiv or HuggingFace) | Xinhua Cheng, Yunyang Ge, Lin-Chen, BestWishYsh, LanguageBind | Open-Sora Plan is an open-source project for generating high-resolution, long-duration videos. The objective is to develop a large generation model capable of producing desired videos from various user inputs, including text, images, and structure control signals. The project uses a Wavelet-Flow Variational Autoencoder (WF-VAE), a Joint Image-Video Skiparse Denoiser with 3D attention, and various condition controllers, along with training and inference optimization strategies like a min-max token strategy and adaptive gradient clipping. WF-VAE-L achieves a throughput of 5.55 videos/second when encoding 33-frame 512x512 videos, 7.8 times faster than Allegro with 8 times less memory usage. This project offers AI practitioners a comprehensive framework and efficient methods for developing and implementing high-quality video generation models. |
| TAPTRv3: Spatial and Temporal Context Foster Robust Tracking of Any Point in Long Video (Read more on arXiv or HuggingFace) | Zhaoyang Zeng, Tianhe Ren, Shilong Liu, Hongyang Li, Jinyuan Qu | TAPTRv3 enhances point tracking robustness in long videos using spatial and temporal context. The research aimed to improve the long-video tracking performance of TAPTRv2, which struggles with feature querying due to increasing target variation and scene cuts. The authors introduce Context-aware Cross-Attention (CCA) and Visibility-aware Long-Temporal Attention (VLTA) to enhance spatial and temporal feature querying, respectively, along with a global matching module for scene cut handling. TAPTRv3 achieves state-of-the-art performance on multiple datasets, showing a 9.3 average Jaccard (AJ) improvement over TAPTRv2 on long video datasets (Kinetics, RGB-Stacking, and RoboTAP). This allows AI practitioners to implement more accurate and robust point tracking in long videos for applications such as video editing, SLAM, and robotic manipulation, even without large amounts of real training data. |
| o1-Coder: an o1 Replication for Coding (Read more on arXiv or HuggingFace) | Jinlin Xiao, Jiangming Shu, Yuqi Yang, Shangxi Wu, Yuxiang Zhang | O1-CODER replicates OpenAI’s o1 model, focusing on coding tasks. The objective is to enhance a language model’s System-2 thinking (deliberate, analytical processing) for code generation using reinforcement learning (RL) and Monte Carlo Tree Search (MCTS). The methodology involves training a Test Case Generator, using MCTS to generate reasoning-enhanced code data, and iteratively fine-tuning a policy model with a process reward model. Pseudocode-based code generation with Qwen2.5-Coder-7B achieved an Average Sampling Pass Rate (ASPR) of 74.9% on the MBPP benchmark, significantly exceeding vanilla Qwen2.5-7B’s 49.3% ASPR. This implies that generating accurate pseudocode is crucial for correct code generation, highlighting the importance of methods like RL and MCTS for refining the reasoning process in LLMs for coding tasks. |
| TinyFusion: Diffusion Transformers Learned Shallow (Read more on arXiv or HuggingFace) | Xinchao Wang, Xinyin Ma, Kunjun Li, Gongfan Fang | TinyFusion is a learnable depth pruning method for compressing diffusion transformers. The objective is to create shallower diffusion transformer models with reduced inference costs while maintaining competitive post-fine-tuning performance. The method utilizes a differentiable sampling technique for layer mask selection, co-optimized with a weight update (using LoRA or full fine-tuning) to estimate recoverability. Experiments on DiT-XL show TinyFusion achieves an FID score of 2.86 after pruning to 14 layers and fine-tuning with Masked Knowledge Distillation, using only 7% of the original training cost. This allows AI practitioners to significantly reduce the computational cost of deploying diffusion transformers for image generation without drastically sacrificing generative quality. |
| VLsI: Verbalized Layers-to-Interactions from Large to Small Vision Language Models (Read more on arXiv or HuggingFace) | Yueh-Hua Wu, Yong Man Ro, Yu-Chiang Frank Wang, Ryo Hachiuma, BK-Lee | VLsI is a new family of efficient vision-language models (VLMs) in 2B and 7B sizes. The research aimed to develop smaller VLMs that perform comparably to larger models without architectural changes. The key methodology involves layer-wise distillation using intermediate “verbalizers” that map each layer’s output to natural language, aligning the smaller VLM’s reasoning process with a larger one. VLsI-7B achieved a 17.4% performance improvement over GPT-4V on ten vision-language benchmarks. AI practitioners can utilize VLsI’s layer-wise verbalization technique for efficient VLM distillation, enabling deployment on resource-constrained devices without significant performance degradation. |
| WF-VAE: Enhancing Video VAE by Wavelet-Driven Energy Flow for Latent Video Diffusion Model (Read more on arXiv or HuggingFace) | Liuhan Chen, Yang Ye, Zongjian Li, BestWishYsh, LanguageBind | WF-VAE enhances video reconstruction quality and computational efficiency for latent video diffusion models. The research aimed to address the computational bottlenecks and latent space discontinuities in existing video VAEs, particularly for long, high-resolution videos. The authors introduce Wavelet Flow VAE (WF-VAE), leveraging multi-level wavelet transforms to prioritize low-frequency information and a Causal Cache mechanism for lossless block-wise inference. WF-VAE-L achieves a PSNR of 35.87 and an LPIPS of 0.0175 on the Panda70M dataset with 16 latent channels, outperforming CogVideoX VAE in these metrics. This improvement enables AI practitioners to train and deploy more efficient and higher-quality video generation models, especially for resource-intensive, large-scale applications. |
| SOLAMI: Social Vision-Language-Action Modeling for Immersive Interaction with 3D Autonomous Characters (Read more on arXiv or HuggingFace) | Huaizhong Zhang, Zhengyu Lin, Weiye Xiao, Jianping Jiang, caizhongang | SOLAMI is a novel end-to-end social Vision-Language-Action (VLA) framework for immersive interaction with 3D autonomous characters. The research aimed to create 3D autonomous characters capable of perceiving, understanding, and interacting with humans in immersive environments using multiple modalities. The researchers developed a unified social VLA architecture trained on a synthesized multimodal social interaction dataset (SynMSI) and implemented in a VR interface. SOLAMI achieved a lower inference latency (2.639 seconds) than the LLM+Speech and DLP baseline methods. This lower latency, coupled with improved performance in motion quality and context relevance, indicates that an end-to-end VLA model like SOLAMI can enable more natural and responsive real-time interactions with 3D characters in immersive applications. |
| Long Video Diffusion Generation with Segmented Cross-Attention and Content-Rich Video Data Curation (Read more on arXiv or HuggingFace) | Yuan Zhou, Qiuyue Wang, Yuxuan Cai, hyang0511, Cakeyan | Presto generates 15-second videos with enhanced content richness and long-range coherence. The research aimed to address the challenges of generating long videos with diverse scenarios and consistent storylines. The core methodology involves Segmented Cross-Attention (SCA), dividing hidden states into segments that cross-attend to corresponding sub-captions, and a curated LongTake-HD dataset of long videos with progressive sub-captions. Presto achieved a 78.5% VBench Semantic Score, outperforming state-of-the-art models. This provides AI practitioners with a novel architecture and dataset for generating longer, more coherent, and content-rich videos using diffusion models. |
| Collaborative Instance Navigation: Leveraging Agent Self-Dialogue to Minimize User Input (Read more on arXiv or HuggingFace) | Alessandro Farinelli, Alberto Castellini, Gianni Franchi, e-zorzi, ftaioli | AIUTA enables embodied agents to locate target objects in unknown environments through collaborative dialogue with users. The research addresses the challenge of instance navigation with minimal initial user input. The proposed method, AIUTA (Agent-user Interaction with Uncertainty Awareness), utilizes a self-questioning module with a VLM and LLM to refine object descriptions and an interaction trigger to determine when to query the user. On the CoIN-Bench with simulated users, AIUTA achieved a 14.47% success rate on the Train split, substantially outperforming a zero-shot baseline that lacked user interaction. This work provides a framework for building more practical and user-friendly instance navigation systems by reducing the burden of providing detailed upfront instructions. |
| VLSBench: Unveiling Visual Leakage in Multimodal Safety (Read more on arXiv or HuggingFace) | Jing Shao, Xuanjing Huang, LLLeo612, Max9803, Foreshhh | VLSBench, a new multimodal safety benchmark, is designed to address visual safety information leakage (VSIL) in existing multimodal datasets. The research aimed to understand why textual alignment performs comparably to multimodal alignment on existing multimodal safety benchmarks, suspecting a VSIL problem. The authors constructed VLSBench with 2.4k image-text pairs, preventing leakage from image to text through an automated pipeline involving harmful query generation, detoxification, iterative image generation, and filtration. Multimodal alignment methods outperformed textual alignment methods on VLSBench, with the best close-source model (Gemini-1.5-pro) achieving a 49.78% safety rate. This highlights the need for AI practitioners to prioritize multimodal alignment over textual alignment when addressing safety in multimodal models, especially in scenarios where sensitive visual content is not explicitly described in the text. |
| INCLUDE: Evaluating Multilingual Language Understanding with Regional Knowledge (Read more on arXiv or HuggingFace) | atcbosselut, jjzha, jebish7, shayekh, angelika | INCLUDE benchmarks multilingual LLMs’ understanding of regional knowledge. The study investigates how large language models perform on questions requiring cultural and regional knowledge across diverse languages. Researchers compiled a novel dataset of 197,243 multiple-choice questions from local exams in 44 languages and 15 scripts, avoiding translation artifacts by using original-language sources and annotating questions for regionality and academic domain. GPT-4 achieved the highest overall accuracy of 77.1% on the INCLUDE-BASE subset. AI practitioners should account for regional knowledge variance when developing and evaluating multilingual LLMs and consider that model performance varies considerably based on language and question type, even within a single model. |
| Efficient Track Anything (Read more on arXiv or HuggingFace) | Chenchen Zhu, Lemeng Wu, Xiaoyu Xiang, Chong Zhou, yunyangx | EfficientTAMs are lightweight models for video object segmentation and tracking with reduced computational complexity compared to SAM 2. The research aimed to create more efficient track-anything models with low latency and small model size, suitable for mobile deployment. The methodology involves utilizing a vanilla Vision Transformer (ViT) as the image encoder and introducing an efficient memory module based on coarser representations of memory spatial tokens for cross-attention. On the SA-V test dataset for semi-supervised video object segmentation, EfficientTAM-S achieves 74.5 J&F, comparable to SAM 2, with ~2x speedup on A100 GPUs and ~2.4x parameter reduction. This allows AI practitioners to deploy real-time video object segmentation models on resource-constrained devices, such as mobile phones, broadening the potential applications of this technology. |
| VisOnlyQA: Large Vision Language Models Still Struggle with Visual Perception of Geometric Information (Read more on arXiv or HuggingFace) | Rui Zhang, Ranran Haoran Zhang, Sarkar Snigdha Sarathi Das, Yusen Zhang, ryokamoi | VisOnlyQA, a new dataset, reveals that Large Vision Language Models (LVLMs) struggle with visual perception of geometric information in scientific figures. The research aimed to evaluate the visual perception capabilities of LVLMs independent of reasoning and knowledge. The authors created VisOnlyQA, including real and synthetically generated scientific figures paired with multiple-choice questions about geometric and numerical information, and tested 20 different LVLMs. State-of-the-art models like GPT-40 and Gemini 1.5 Pro achieved only 51.4% and 54.2% accuracy respectively on the real image split, compared to near-perfect human performance (93.5%). The principal implication for AI practitioners is that both training data and model architectures need improvement to enhance the visual perception capabilities of LVLMs, as this weakness significantly limits performance on visual tasks. |
| VISTA: Enhancing Long-Duration and High-Resolution Video Understanding by Video Spatiotemporal Augmentation (Read more on arXiv or HuggingFace) | Wenhu Chen, Cong Wei, Jie Min, hyang0511, wren93 | VISTA improves long and high-resolution video understanding in Large Multimodal Models (LMMs) through data augmentation. The research aimed to address the scarcity of high-quality, long/high-resolution video instruction-following datasets. The key methodology involved spatially and temporally combining videos from existing datasets to create synthetic long and high-resolution video samples, followed by generating corresponding question-answer pairs using a language model (Gemini). Finetuning LMMs on VISTA-400K resulted in an average 3.3% improvement across four long-video understanding benchmarks and a 6.5% gain on the newly introduced HRVideoBench for high-resolution video understanding. This provides AI practitioners with a cost-effective method to improve LMM performance on long and high-resolution video understanding tasks through data augmentation, eliminating the need for costly manual annotation. |
| Steering Rectified Flow Models in the Vector Field for Controlled Image Generation (Read more on arXiv or HuggingFace) | Yezhou Yang, Dimitris N. Metaxas, Song Wen, mpatel57 | FlowChef steers rectified flow models’ denoising trajectories for controlled image generation. The paper investigates how to efficiently guide rectified flow models (RFMs) for tasks like image editing, classifier guidance, and solving linear inverse problems without computationally expensive inversion or backpropagation. The key methodology involves leveraging the smooth vector field dynamics of RFMs and a gradient skipping approach to directly adjust the trajectory during denoising. On linear inverse problems, FlowChef achieves 26.32 PSNR on box inpainting with a 20x20 mask, surpassing baselines on the pixel-space Rectified Flow++ model. This offers AI practitioners a computationally efficient and inversion-free method for controlled image generation using RFMs, potentially improving performance and reducing resource demands for applications like image editing and guided synthesis. |
| PhysGame: Uncovering Physical Commonsense Violations in Gameplay Videos (Read more on arXiv or HuggingFace) | Hangyu Guo, Haoze Zhao, Haoran Tang, Meng Cao, zhangysk | PhysGame introduces a benchmark to evaluate the ability of video LLMs to understand physical commonsense violations in gameplay videos. The research aimed to assess and improve video LLMs’ ability to recognize glitches that defy real-world physics. Researchers created PhysGame, a benchmark with 880 videos of glitches, PhysInstruct, an instruction tuning dataset with 140,057 question-answer pairs, and PhysDPO, a preference optimization dataset with 34,358 pairs using misleading video data. Their proposed PhysVLM model, trained on these datasets, achieved state-of-the-art performance on PhysGame and an overall accuracy of 61.1% on the Video-MME benchmark with subtitles. This work provides a benchmark and resources for training video LLMs capable of robust physical commonsense reasoning, crucial for developing more realistic and reliable AI agents in game development and broader applications. |
| FLOAT: Generative Motion Latent Flow Matching for Audio-driven Talking Portrait (Read more on arXiv or HuggingFace) | Gyoungsu Chae, Dongchan Min, Taekyung Ki | FLOAT generates talking portrait videos from a single source image and audio using a flow matching generative model. The objective is to synthesize realistic talking motions from audio, including lip synchronization, head movements, and facial expressions, while addressing limitations of diffusion-based methods like slow sampling. The key methodology involves modeling talking motion within a learned motion latent space using a transformer-based vector field predictor and decoding the sampled motion latents into video frames. On the HDTF dataset, FLOAT achieves a Fréchet Inception Distance (FID) of 21.100, outperforming compared baselines. This efficient and high-quality approach offers AI practitioners a more effective method for generating realistic and temporally consistent talking portrait videos. |
| A Simple and Provable Scaling Law for the Test-Time Compute of Large Language Models (Read more on arXiv or HuggingFace) | Jingren Zhou, Bolin Ding, Yaliang Li, Xuchen Pan, yanxi-chen | This paper proposes a two-stage algorithm (generation and knockout) for improving the test-time compute of Large Language Models (LLMs). The research aims to boost the success probability of LLMs by increasing test-time compute, specifically addressing the challenge of ensuring high reliability in high-stakes scenarios. The proposed algorithm involves generating multiple candidate solutions and selecting the best one through a knockout tournament with pairwise comparisons. On a subset of the MMLU-Pro benchmark, the algorithm’s accuracy improved from approximately 60% to over 65% for the “engineering” category when scaling the number of initial candidate solutions (N) from 1 to 32 with comparison parameter K=2 using Llama3.1. AI practitioners can leverage this method to enhance LLM reliability for complex tasks by scaling test-time computation with provable performance guarantees, provided the underlying assumptions regarding solution generation and comparison probabilities hold. |
| Towards Cross-Lingual Audio Abuse Detection in Low-Resource Settings with Few-Shot Learning (Read more on arXiv or HuggingFace) | Noel Crespi, Reza Farahbaksh, callmesan | This paper explores cross-lingual few-shot learning for audio abuse detection in low-resource languages. The research objective is to develop a model capable of detecting abusive language in multiple Indian languages using limited labeled data. The methodology involves extracting audio features using pre-trained Wav2Vec and Whisper models, normalizing these features using Temporal Mean or L2-Norm, and classifying them with a Model-Agnostic Meta-Learning (MAML) based few-shot classifier. Whisper with L2-Norm normalization achieved the highest accuracy, reaching 85.22% for Malayalam in the 100-shot setting. AI practitioners can leverage pre-trained audio representations and meta-learning techniques to develop robust abuse detection systems for low-resource languages, even with limited labeled data, highlighting the potential for improved content moderation across diverse linguistic groups. |
Papers for 2024-12-02
| Title | Authors | Summary |
|---|---|---|
| On Domain-Specific Post-Training for Multimodal Large Language Models (Read more on arXiv or HuggingFace) | Xintong Zhang, doubling, edward2021, buaahsh, daixuancheng | This paper investigates domain-specific post-training for adapting general Multimodal Large Language Models (MLLMs) to specialized domains like biomedicine and food. The research aims to improve MLLM performance in specific domains through data synthesis and a novel single-stage training pipeline. A visual instruction synthesizer generates domain-specific tasks from image-caption pairs, filtered by a consistency check, and used for single-stage training alongside image captioning data. AdaMLLM, the resulting adapted MLLM, outperformed general MLLMs across various domain-specific tasks, with a 58.3% average performance on biomedical tasks using PMC-Raw image-caption data and single-stage training. This research provides AI practitioners with a method for efficiently adapting pre-trained MLLMs to specialized domains using readily available image-caption datasets, enabling enhanced performance on domain-specific downstream tasks. |
| Beyond Examples: High-level Automated Reasoning Paradigm in In-Context Learning via MCTS (Read more on arXiv or HuggingFace) | Zengqi Wen, Feihu Che, Shuai Zhang, fmk345, Jinyang23 | HiAR-ICL enhances in-context learning for complex reasoning tasks by focusing on high-level thinking patterns rather than specific examples. The research aims to improve LLM performance on complex reasoning tasks by shifting from example-based in-context learning to a paradigm based on abstract thinking patterns. The core methodology uses Monte Carlo Tree Search (MCTS) to explore reasoning paths and construct “thought cards” representing these patterns, which are then selected based on a cognitive complexity metric. HiAR-ICL achieves 79.6% accuracy on the MATH benchmark using Qwen2.5-7B-Instruct, outperforming GPT-40 (76.6%) and Claude 3.5 (71.1%). This implies AI practitioners can leverage high-level reasoning patterns and MCTS to enhance the performance and generalization of LLMs, especially smaller models, on complex reasoning tasks without extensive demonstration engineering. |
| Timestep Embedding Tells: It’s Time to Cache for Video Diffusion Model (Read more on arXiv or HuggingFace) | MoonQiu, weilllllls, Jeff-Wang, StevenZhang, LiewFeng | TeaCache accelerates video diffusion model inference by selectively caching intermediate model outputs. The research aimed to improve the inference speed of diffusion-based video generation models without compromising visual quality. The method estimates output differences using timestep embedding modulated noisy inputs and a rescaling strategy based on polynomial fitting to determine caching schedules. Experiments showed up to a 4.41x speedup on Open-Sora-Plan with a negligible -0.07% VBench score degradation. This training-free caching strategy offers AI practitioners a way to substantially reduce the computational cost of deploying state-of-the-art video diffusion models. |
| DisCoRD: Discrete Tokens to Continuous Motion via Rectified Flow Decoding (Read more on arXiv or HuggingFace) | Mingu Kang, Minseo Kim, Jisoo Kim, junwann, whwjdqls99 | DisCoRD decodes discrete motion tokens into continuous motion using rectified flow to enhance naturalness while preserving faithfulness to conditioning signals. The research aimed to address the limitations of existing discrete and continuous human motion generation methods, specifically under-reconstruction and frame-wise noise in discrete methods, and cross-modal mapping ambiguity in continuous methods. The core methodology involves training a rectified flow model conditioned on frame-wise features extracted from discrete motion tokens, enabling iterative refinement in continuous space. On HumanML3D, DisCoRD achieved a Fréchet Inception Distance (FID) of 0.032, surpassing existing discrete methods in naturalness. This provides AI practitioners with a method to generate more realistic and faithful human motion from discrete representations, applicable to various motion generation tasks such as text-to-motion and music-to-dance generation. |
| Puzzle: Distillation-Based NAS for Inference-Optimized LLMs (Read more on arXiv or HuggingFace) | nav4, nailon-nvidia, talor-abr, tomer-nv, abercovich | Puzzle is a framework for accelerating LLM inference on specific hardware while preserving model capabilities. The research aimed to optimize large language model architectures for efficient inference on specific hardware while maintaining accuracy. The methodology involved decomposed neural architecture search (NAS) using blockwise local knowledge distillation (BLD), mixed-integer programming for constraint optimization, and global knowledge distillation (GKD). The derived model, Nemotron-51B, achieved a 2.17x inference throughput speedup on a single NVIDIA H100 GPU compared to its parent model, Llama-3.1-70B-Instruct, while preserving 98.4% of its capabilities. This provides AI practitioners with access to state-of-the-art language models optimized for efficient deployment with minimal accuracy trade-offs, enabling wider adoption across various applications and hardware. |
| Trajectory Attention for Fine-grained Video Motion Control (Read more on arXiv or HuggingFace) | Xingang-Pan, Jianlou, PKUWilliamYang, Vicky0522, zeqixiao | This paper introduces trajectory attention for precise camera motion control in video generation. The research aims to improve the precision and consistency of camera motion control in generated videos, addressing limitations of existing methods that struggle with temporal coherence or rely on implicit control mechanisms. The core methodology involves modeling trajectory attention as an auxiliary branch alongside traditional temporal attention in video diffusion models, allowing explicit injection of trajectory information while maintaining the model’s generative capabilities. Experiments on camera motion control for images show the method achieves an Absolute Trajectory Error (ATE) of 0.0396 meters on 25-frame sequences. This provides AI practitioners with a plug-and-play module for enhanced camera motion control in video diffusion models, improving the precision and consistency of generated video motion, particularly valuable for tasks requiring fine-grained control over camera movement. |
| Video Depth without Video Models (Read more on arXiv or HuggingFace) | toshas, PeterTor, peterjohnson, dnarnhofer, Bingxin | RollingDepth estimates temporally consistent video depth using a modified single-image latent diffusion model (LDM). The research aimed to develop accurate and temporally stable video depth estimation without computationally expensive video diffusion models. The key methodology involved adapting a single-image LDM (Marigold) to process short video snippets, incorporating cross-frame self-attention and a robust, optimization-based global alignment algorithm. RollingDepth achieved a 9.6% absolute mean relative error on the PointOdyssey dataset, outperforming existing video and single-image depth models. This implies that AI practitioners can leverage modified single-image LDMs for efficient and accurate video depth estimation, avoiding the computational burden of dedicated video models. |
| AlphaTablets: A Generic Plane Representation for 3D Planar Reconstruction from Monocular Videos (Read more on arXiv or HuggingFace) | bys0318, AlbertHuyb, lshmouse, thuzhaowang, hyz317 | AlphaTablets is a novel 3D plane representation for reconstructing planar surfaces from monocular videos. The research aimed to develop a more accurate and generalizable method for 3D planar reconstruction from monocular video input. The core methodology involved representing 3D planes as rectangles with alpha channels (AlphaTablets), differentiable rasterization for rendering, and a bottom-up pipeline incorporating optimization and a merging scheme. On the ScanNet dataset, the method achieved a 0.456 F-score for 3D geometry reconstruction, outperforming existing methods. This new representation and pipeline offer AI practitioners a more effective and flexible way to reconstruct and edit 3D planar structures from monocular videos, potentially improving applications in scene understanding, robotics, and mixed reality. |
| Look Every Frame All at Once: Video-Ma$^2$mba for Efficient Long-form Video Understanding with Multi-Axis Gradient Checkpointing (Read more on arXiv or HuggingFace) | Hyunjun Kim, dwightro, arkimjh, lakelee | Video-Ma²mba is a novel large multimodal model designed for efficient long-form video understanding. The research aimed to address the challenge of quadratic memory and computational demands of transformer-based models when processing long video sequences. The key methodology involved replacing the transformer backbone with the linear-complexity Mamba-2 architecture and introducing Multi-Axis Gradient Checkpointing (MA-GC) for memory efficiency. Video-Ma²mba achieved a 4.1% improvement on the Video-MME benchmark compared to a 16-frame limited baseline. This implies that AI practitioners can leverage MA-GC within the Mamba-2 framework to process long video sequences (up to 2 hours at 1 FPS on a single GPU) more efficiently than transformer-based models, potentially improving performance in video understanding tasks by capturing more complete temporal information. |
| AC3D: Analyzing and Improving 3D Camera Control in Video Diffusion Transformers (Read more on arXiv or HuggingFace) | willi-menapace, aliaksandr-siarohin, guochengqian, universome, sherwinbahmani | AC3D analyzes and improves 3D camera control within pre-trained video diffusion transformers. The research aims to enable precise 3D camera manipulation in video diffusion models without sacrificing video quality. The key methodology involves analyzing motion spectral volumes, linearly probing internal model representations for camera pose knowledge, and curating a dataset of dynamic videos with static cameras. Results show an 18% improvement in video fidelity (FVD) and 25% improvement in camera steering accuracy compared to the closest baseline. AI practitioners can leverage these insights to develop more precise and efficient camera control mechanisms for text-to-video generation and related applications by understanding how to condition camera pose within video diffusion transformer architectures and tailor training data to enhance scene dynamism while preserving camera control fidelity. |
| FAM Diffusion: Frequency and Attention Modulation for High-Resolution Image Generation with Stable Diffusion (Read more on arXiv or HuggingFace) | Xiatian Zhu, Hai X. Pham, Isma Hadji, Adrian Bulat, Haosen Yang | FAM diffusion introduces two novel modules to improve high-resolution image generation with pre-trained latent diffusion models. The objective is to enable high-resolution image generation without retraining, addressing issues like object repetition and inconsistent local textures seen when upscaling. The key methodology involves a Frequency Modulation (FM) module, operating in the Fourier domain to enhance global structure consistency, and an Attention Modulation (AM) module to improve local texture consistency. FAM diffusion achieves state-of-the-art performance, demonstrating a CLIP score of 32.33 at 4x upscaling with SDXL, and significantly reducing latency compared to patch-based methods. This allows AI practitioners to generate high-quality, high-resolution images from pre-trained models without computationally expensive retraining or significant latency overheads. |
| LLM Teacher-Student Framework for Text Classification With No Manually Annotated Data: A Case Study in IPTC News Topic Classification (Read more on arXiv or HuggingFace) | nljubesi, TajaKuzman | This paper proposes a teacher-student framework using LLMs for multilingual news topic classification without manual annotation. The research aims to develop accurate and computationally efficient multilingual IPTC news topic classifiers for languages lacking annotated training data. The methodology employs GPT-40 to automatically annotate news articles in four languages, creating a training dataset for fine-tuning an XLM-ROBERTa student model. The XLM-ROBERTa model, trained on 15,000 automatically labeled instances, achieves a macro-F1 score of 0.746. This demonstrates the feasibility of using LLM-generated labels to train smaller, more efficient models for multilingual text classification, enabling AI practitioners to build robust classifiers for low-resource languages without extensive manual annotation efforts. |
Papers for 2024-11-29
| Title | Authors | Summary |
|---|---|---|
| Critic-V: VLM Critics Help Catch VLM Errors in Multimodal Reasoning (Read more on arXiv or HuggingFace) | Jingdi Lei, jwu323, ZonglinY, Duke-de-Artois, qq8933 | Critic-V is a framework for enhancing the reasoning capabilities of Vision-Language Models (VLMs). The research aims to address the issue of VLMs generating inaccurate or irrelevant responses in multimodal reasoning tasks. The key methodology involves a Reasoner-Critic architecture, where a Reasoner VLM generates reasoning paths and a Critic VLM provides feedback for refinement using Direct Preference Optimization (DPO) trained on a critique-VQA dataset. Qwen2-VL-7B with Critic-V achieved the highest scores on five out of eight benchmarks, with an 11.8% improvement on MathVista compared to the baseline. This provides AI practitioners with a method to improve the reliability and accuracy of VLMs in reasoning-heavy multimodal applications by integrating an external critic model for real-time feedback during inference. |
| ChatGen: Automatic Text-to-Image Generation From FreeStyle Chatting (Read more on arXiv or HuggingFace) | Hangwei Qian, Weijia Wu, Zhuohang Dang, Changliang Xia, ChengyouJia | ChatGen automates the text-to-image generation process from free-form user input. The research aimed to develop a model that automatically generates prompts, selects appropriate models, and configures arguments for text-to-image generation from freestyle user text, image, or chat history. The authors introduce a multi-stage evolution strategy (ChatGen-Evo) incorporating supervised fine-tuning for prompt generation, ModelTokens for model selection, and in-context learning for argument configuration. ChatGen-Evo achieved a Unified Metric score of 65.9 in supervised settings, surpassing other baselines and demonstrating comparable performance to a much larger 8B parameter model while using only 2B parameters. This work suggests that focusing on stage-wise training for complex automated text-to-image generation tasks can yield significant performance improvements with smaller models, offering a potential path towards more efficient and accessible automated image generation for AI practitioners. |
| TryOffDiff: Virtual-Try-Off via High-Fidelity Garment Reconstruction using Diffusion Models (Read more on arXiv or HuggingFace) | Barbara Hammer, Robin Chan, Petra Bevandic, rizavelioglu | TryOffDiff reconstructs standardized garment images from photos of clothed individuals. The research objective is to generate canonical garment images from real-world photos, a task termed Virtual Try-Off (VTOFF). The key methodology involves adapting Stable Diffusion with SigLIP-based visual conditioning, replacing text prompts with image features. On the modified VITON-HD dataset, TryOffDiff achieves a DISTS score of 22.5, outperforming adapted VTON and pose transfer baselines. The paper mentions no background removal post-processing was applied to TryOffDiff while some form of removal was applied to baseline models; how this affects the comparison remains unclear. This work provides AI practitioners with a novel approach for high-fidelity garment reconstruction, potentially improving e-commerce product imagery and generative model evaluation. |
| Free$^2$Guide: Gradient-Free Path Integral Control for Enhancing Text-to-Video Generation with Large Vision-Language Models (Read more on arXiv or HuggingFace) | Jong Chul Ye, Bryan S Kim, kjm981995 | Free$^2$Guide enhances text-video alignment in diffusion-based generative models without needing reward function gradients. The research aims to improve text alignment in text-to-video generation using non-differentiable reward functions like Large Vision-Language Models (LVLMs). The method approximates guidance by combining path integral control with zeroth-order gradient estimations and enables ensembling multiple reward models. Using GPT-40 with LaVie for text-video alignment showed a 28.6% improvement on the Spatial Relationship metric compared to the baseline LaVie model. This offers AI practitioners a way to leverage powerful black-box LVLMs for improved text-video alignment without needing model fine-tuning or differentiable reward functions, thereby potentially reducing computational overhead. |
| Morph: A Motion-free Physics Optimization Framework for Human Motion Generation (Read more on arXiv or HuggingFace) | Hao Liu, Xin Zhao, Ruibing Hou, Mingshuang Luo, Zhuo Li | Morph enhances the physical plausibility of generated human motion without using real motion data. The research aimed to develop a model-agnostic physics optimization method that doesn’t require costly real motion capture data. A two-stage process trains a Motion Physics Refinement (MPR) module on synthetic noisy motion data from a generator, then uses the refined output to fine-tune the original generator. On the HumanML3D dataset, Morph-MoMask reduced ground penetration errors from 23.152 to 0.0. AI practitioners can use Morph to improve the physical realism of generated motions across diverse motion generation models and tasks (text-to-motion, music-to-dance) without needing expensive real-world motion datasets. |
| LongKey: Keyphrase Extraction for Long Documents (Read more on arXiv or HuggingFace) | Jean Paul Barddal, Cinthia Obladen de Almendra Freitas, Jeovane Honorio Alves, RaduState | LongKey is a novel framework for extracting keyphrases from long documents. The research aimed to address the limitations of existing keyphrase extraction methods in processing long-context documents (greater than 512 tokens). The methodology involves using Longformer for word embeddings, a max-pooling-based keyphrase embedding pooler, and a ranking loss combined with a chunking loss for candidate scoring. On the LDKP10K dataset, LongKey achieved an F1@5 score of 41.81%. The keyphrase embedding pooler significantly contributes to LongKey’s improved performance, offering AI practitioners a more effective technique for extracting keyphrases from lengthy texts, enhancing information retrieval and summarization tasks. |
Papers for 2024-11-28
| Title | Authors | Summary |
|---|---|---|
| ROICtrl: Boosting Instance Control for Visual Generation (Read more on arXiv or HuggingFace) | KevinQHLin, pcma, ynie, 365sleep, guyuchao | Here’s a concise summary of the AI research paper following your strict guidelines: i) ROICtrl enhances diffusion models for precise multi-instance visual generation by introducing regional instance control via ROI-Align and a novel ROI-Unpool operation. ii) The research aimed to improve the accuracy and efficiency of multi-instance visual generation by addressing limitations in associating positional and attribute information with multiple instances in natural language prompts. iii) The key methodology involved using ROI-Align and a novel complementary operation, ROI-Unpool, to enable efficient and accurate manipulation of regions of interest (ROIs) on high-resolution feature maps for visual generation, followed by a learnable attention blending mechanism to integrate instance captions with global captions. iv) ROICtrl achieved a 0.73 instance success rate on the ROICtrl-Bench benchmark, surpassing previous methods in both template-based and free-form instance caption tasks. Specific details on other benchmarks are mentioned but complete numerical results are not provided in the summary. v) The development of ROI-Unpool, a complementary operation to ROI-Align for generative models, offers a significant advancement for AI practitioners working on visual generation. This enables more precise control over multiple instances within generated images, improving the accuracy and computational efficiency of multi-instance image synthesis tasks. Further implications are discussed but quantitative findings are not always fully summarized. |
| Interleaved Scene Graph for Interleaved Text-and-Image Generation Assessment (Read more on arXiv or HuggingFace) | ranjaykrishna, Tim666, lzy8465, Dipsy0830, shuaishuaicdp | This paper introduces ISG, a framework for evaluating interleaved text-and-image generation. The research aims to address the lack of robust evaluation metrics for models generating interleaved text and images. The ISG framework uses a scene graph representation and a four-level (holistic, structural, block, image) evaluation protocol leveraging question-answering feedback. Compositional models achieved a higher holistic score of 6.262 compared to 2.961 for the best unified model, though still lagging behind human performance. AI practitioners developing multimodal generative models should consider compositional architectures and the fine-grained insights provided by ISG for improving model performance and addressing limitations like instruction following and consistency across modalities. |
| CAT4D: Create Anything in 4D with Multi-View Video Diffusion Models (Read more on arXiv or HuggingFace) | Ruiqi Gao, holynski, atrevithick, doinkda, rundi | Here’s a summary of the AI research paper following your strict guidelines: i) CAT4D generates dynamic 3D scenes from monocular video using a multi-view video diffusion model and deformable 3D Gaussian representation. ii) To create 4D (dynamic 3D) scenes from monocular video input, overcoming the limitations of requiring synchronized multi-view video data for accurate 4D reconstruction. iii) A multi-view video diffusion model trained on diverse datasets is used to transform a single monocular video into a multi-view video, enabling robust 4D reconstruction via optimization of a deformable 3D Gaussian representation. A novel sampling strategy is employed to generate nearly-consistent multi-view videos beyond the model’s native output length. iv) The model achieves competitive performance on novel view synthesis and dynamic scene reconstruction benchmarks, demonstrating disentangled camera and time control (quantitative result: 21.97 PSNR, 0.683 SSIM, 0.121 LPIPS on disentangled control experiments using NSFF dataset). v) The disentangled camera and time control demonstrated by the model is a significant achievement for dynamic scene generation from limited input. This approach directly benefits AI practitioners working on video generation, 3D reconstruction, and augmented/virtual reality applications by providing a more robust method for creating dynamic 3D content from readily available monocular video data. The paper notes some ambiguity on the robustness of the method when dealing with highly dynamic scenes, implying a need for further research in that area. |
| Large Language Model-Brained GUI Agents: A Survey (Read more on arXiv or HuggingFace) | Gezelligheid520, liqul, bowenli, shilhe, vyokky | This paper surveys Large Language Model (LLM)-brained GUI agents, intelligent agents operating within GUI environments using LLMs. The objective is to provide a comprehensive overview of this burgeoning field, covering historical evolution, core components, and advanced techniques. The survey analyzes existing frameworks, data collection methods, model training strategies, evaluation benchmarks, and applications of LLM GUI agents. SeeAct, a multimodal LLM GUI agent, achieved a 51.1% task success rate on real-time web tasks. AI practitioners can use this survey as a guide for constructing LLM-powered GUI agents and as a reference for advancing research in this domain, particularly in optimizing model performance for complex, real-world GUI interactions. |
| MARVEL-40M+: Multi-Level Visual Elaboration for High-Fidelity Text-to-3D Content Creation (Read more on arXiv or HuggingFace) | Sankalp Sinha, mzafzal, saali14, alootikki, SadilKhan | This paper introduces MARVEL-40M+, a large-scale, multi-level annotated dataset for text-to-3D content generation. The objective is to address the limitations of existing text-to-3D datasets in size, diversity, and annotation depth, hindering high-fidelity 3D model generation. A multi-stage annotation pipeline combining multi-view VLMs (InternVL2), LLMs (Qwen 2.5), and filtered human metadata creates five levels of descriptions for over 8.9 million 3D assets. Evaluation shows MARVEL-40M+ achieves a 72.41% win rate against existing datasets in image-text alignment as judged by GPT-4. AI practitioners can leverage MARVEL-40M+ to train and evaluate more robust and higher-fidelity text-to-3D generation models, benefiting applications in gaming, AR, and VR by providing a significantly richer and larger training resource. |
| Collaborative Decoding Makes Visual Auto-Regressive Modeling Efficient (Read more on arXiv or HuggingFace) | Xinchao Wang, Gongfan Fang, horseee, Zigeng | Here’s a summary of the AI research paper following your strict guidelines: i) One-line summary: Collaborative Decoding (CoDe) improves Visual Auto-Regressive (VAR) model efficiency by partitioning multi-scale inference between a large and a small model, resulting in significant speed and memory reductions with minimal quality loss. ii) Main research question/objective: How can the efficiency of Visual Auto-Regressive (VAR) image generation models be improved, particularly addressing memory consumption and computational redundancies associated with long token sequences? iii) Key methodology: A novel decoding strategy called Collaborative Decoding (CoDe) is proposed. CoDe divides the multi-scale inference process into a “drafter” (large model generating low-frequency content) and a “refiner” (small model generating high-frequency details). Model-specific fine-tuning is also applied. iv) Primary results: CoDe achieves a 1.7x speedup and reduces memory usage by approximately 50% compared to the original VAR model, with only a negligible increase in FID (from 1.95 to 1.98). A 2.9x speedup was also achieved under different drafting steps. v) Principal implication for AI practitioners: CoDe offers a practical method to significantly enhance the efficiency of VAR models for image generation, reducing both computational cost and memory requirements without substantial quality degradation. This is particularly relevant for deploying high-resolution image generation models on resource-constrained platforms. |
| DiffusionDrive: Truncated Diffusion Model for End-to-End Autonomous Driving (Read more on arXiv or HuggingFace) | Haoran Yin, xinggangw, bojiang-bentoml, csy71, LegendBC | Here is a summary of the AI research paper following your strict guidelines: i) DiffusionDrive, a truncated diffusion model, achieves real-time end-to-end autonomous driving performance superior to existing methods. ii) To develop a real-time, high-quality, multi-mode end-to-end autonomous driving policy that addresses the limitations of existing methods (mode collapse and computational cost). iii) A truncated diffusion policy incorporating prior multi-mode anchors, an efficient cascade diffusion decoder, and a reduced number of denoising steps. iv) On the NAVSIM navtest split, DiffusionDrive achieved 88.1 PDMS without post-processing, exceeding the state-of-the-art. v) The significant speed improvement (45 FPS on an NVIDIA 4090 GPU) and high performance using a ResNet-34 backbone demonstrate the potential of truncated diffusion models for real-time autonomous driving applications. This finding directly impacts the feasibility of deploying diffusion models in resource-constrained real-world scenarios. |
| DreamCache: Finetuning-Free Lightweight Personalized Image Generation via Feature Caching (Read more on arXiv or HuggingFace) | Diego Valsesia, emagli, mosams, u-michieli, Ema97x | DreamCache is a finetuning-free, lightweight approach for personalized image generation. The research aimed to develop an efficient and high-quality personalized image generation method overcoming limitations of existing approaches. DreamCache employs a feature caching mechanism with lightweight, trained conditioning adapters to dynamically modulate generated image features. The method achieved state-of-the-art image and text alignment with only 25M additional parameters; specifically, DreamCache achieved a DINO score of 0.767 on the SD 2.1 backbone with a single reference image. This efficient personalization approach significantly reduces computational costs and memory demands, making it suitable for resource-constrained devices and real-time applications. |
| Identity-Preserving Text-to-Video Generation by Frequency Decomposition (Read more on arXiv or HuggingFace) | Yunyuan Ge, LiuhanChen, hexianyi, Jinfa, BestWishYsh | Here’s a summary of the AI research paper following your strict guidelines: i) One-line summary: ConsisID, a tuning-free diffusion transformer-based model, generates high-fidelity, identity-preserving videos by controlling identity features in the frequency domain. ii) Main research question/objective: To develop a tuning-free identity-preserving text-to-video generation model that maintains consistent human identity in generated videos and addresses limitations of existing Diffusion Transformer (DiT) based models. iii) Key methodology: Frequency decomposition of identity features into high-frequency (intrinsic) and low-frequency (global) components, injected into different DiT layers; hierarchical training strategy combining coarse-to-fine training, dynamic mask loss, and dynamic cross-face loss. iv) Primary results: ConsisID outperforms ID-Animator across multiple metrics, achieving a FaceSim-Arc score of 0.73 versus ID-Animator’s 0.32. (Note: other quantitative metrics (FID, CLIPScore, FaceSim-Cur) are also reported). v) Principal implication for AI practitioners: The frequency decomposition approach and hierarchical training strategy offer a tuning-free method for identity-preserving video generation using DiT models, improving efficiency and generalization compared to previous tuning-based methods. This is significant as it reduces the computational cost and improves the applicability of DiT for identity-preserving video generation. |
| Omegance: A Single Parameter for Various Granularities in Diffusion-Based Synthesis (Read more on arXiv or HuggingFace) | Xiaoming Li, cavanloy, OAOA, itsmag11 | Here’s a summary of the AI research paper following your strict guidelines: i) One-line summary: A single parameter, ω (omega), is introduced to control the granularity of diffusion-based image and video synthesis without model retraining or architectural changes. ii) Main research question/objective: How can the granularity (level of detail) in diffusion-based image and video synthesis be effectively controlled without requiring model retraining or significant architectural modifications? iii) Key methodology: A single parameter, ω, scales the predicted noise during each denoising step in the reverse diffusion process. This parameter can be applied globally, spatially using an omega mask, or temporally using an omega schedule. iv) Primary results: A user study demonstrated 93.94% accuracy in controlling granularity using omega scaling. v) Principal implication for AI practitioners: Omegance offers a simple, efficient method for controlling the granularity of diffusion models. This allows for flexible and nuanced control over generated outputs without the need for model retraining, making it highly relevant for various image and video synthesis applications and potentially reducing development time and computational costs. |
| UniPose: A Unified Multimodal Framework for Human Pose Comprehension, Generation and Editing (Read more on arXiv or HuggingFace) | Shiguang Shan, Hong Chang, Heylon, flow2023, LiyiGang | Here’s a summary of the AI research paper following your strict guidelines: i) UniPose: A unified multimodal framework for human pose comprehension, generation, and editing using LLMs. ii) To build a general-purpose framework for human pose comprehension, generation, and editing across multiple modalities (images, text, 3D poses). iii) A multimodal LLM framework employing a pose tokenizer to unify representation of 3D poses and text, a mixture of visual encoders (CLIP and pose-specific), and a mixed-attention mechanism within the LLM. iv) UniPose achieved competitive performance across various pose-relevant tasks, outperforming existing methods on the Pose-Diff task (UniPose achieved 67.9, 81.8, and 88.6 on Top-1, Top-2, and Top-3 R-precision, respectively, while PoseFix achieved 64.6, 77.1, and 83.0, respectively). v) The successful unification of pose comprehension, generation, and editing tasks within a single multimodal LLM framework offers a powerful tool for AI practitioners developing human-centric applications, improving zero-shot generalization and enabling efficient task adaptation. Further analysis of the model’s performance on different subsets of the task and its ability to generalize to unseen data is required. |
| Draft Model Knows When to Stop: A Self-Verification Length Policy for Speculative Decoding (Read more on arXiv or HuggingFace) | Xingyu Chen, Tian Liang, zptu, Jiahao004, Geralt-Targaryen | Here’s a summary of the AI research paper following your strict guidelines: i) This paper proposes SVIP, a self-verification length policy for speculative decoding that dynamically adjusts draft sequence lengths based on draft token entropy. ii) The main objective is to improve the inference speed of large language models (LLMs) using speculative decoding by addressing the issue of fixed draft lengths in conventional methods. iii) SVIP employs a difficulty-aware dynamic draft length policy that determines draft sequence lengths based on an approximation of a theoretical lower bound of the draft token acceptance rate, using draft model entropy. iv) SVIP achieved up to a 20% wall-time speedup on SpecBench compared to baseline speculative decoding methods. v) The impactful finding, a significant wall-time speedup, directly implies that AI practitioners can leverage SVIP for more efficient LLM inference, particularly in applications demanding high throughput, like chatbots or long-form text generation. The paper does not, however, provide details on memory usage implications of the method. |
| VideoLLM Knows When to Speak: Enhancing Time-Sensitive Video Comprehension with Video-Text Duet Interaction Format (Read more on arXiv or HuggingFace) | Jiansheng Wei, Jianxin Liang, Xiaojun Meng, Yueqian Wang, ColorfulAI | Here’s a summary of the AI research paper following the provided guidelines: i) One-line summary: This paper introduces a novel video-text duet interaction format for VideoLLMs, improving time-sensitive video comprehension by enabling real-time, localized responses. ii) Main research question/objective: How can the interaction format between users and VideoLLMs be improved to enhance time-sensitive video comprehension tasks, such as live-streaming understanding and temporal video grounding? iii) Key methodology: A video-text duet interaction format was developed, where video playback is continuous, and both user and model can insert text messages at any point. A new dataset, MMDuetIT, was created to train VideoLLMs for this format. The Multi-Answer Grounded Video Question Answering (MAGQA) task was introduced for benchmarking. iv) Primary results: Using the video-text duet format, the MMDuet model achieved a 76% CIDEr score on the YouCook2 dense video captioning task. v) Principal implication for AI practitioners: The video-text duet interaction format offers a significant advancement in VideoLLM design for real-time, context-aware responses to time-sensitive tasks. This approach directly addresses limitations of existing whole-video interaction formats which require pre-processing entire videos before generating any output and thus cannot handle real-time scenarios. The significant improvement on the YouCook2 dataset (76% CIDEr) shows the effectiveness of this new interaction paradigm. |
| Adaptive Blind All-in-One Image Restoration (Read more on arXiv or HuggingFace) | Javier Vazquez-Corral, Shaolin Su, Luis Herranz, davidserra9 | Here’s a summary of the AI research paper following your strict guidelines: i) 1-line summary: An adaptive blind all-in-one image restoration model (ABAIR) is proposed that addresses multiple degradations, generalizes to unseen degradations, and efficiently incorporates new ones. ii) Main research question or objective: How to create a blind all-in-one image restoration model that effectively handles multiple and composite degradations, generalizes well to unseen degradations, and can easily incorporate new degradations without extensive retraining? iii) Key methodology used: A three-phase approach: (1) pre-training a baseline model on a large dataset with synthetic degradations and a segmentation head; (2) adapting the baseline model to specific degradations using independent low-rank adapters (LoRA); (3) adaptively combining adapters via a lightweight degradation estimator. iv) Primary results (include one specific quantitative finding): The ABAIR model outperforms state-of-the-art methods by a 2.91dB average PSNR improvement on a five-degradation image restoration task. v) Principal implication for AI practitioners: The modular design with low-rank adapters enables efficient adaptation to new degradation types with minimal retraining, reducing computational costs and improving model flexibility for real-world applications where degradation types are often unknown or composite. |
| Make-It-Animatable: An Efficient Framework for Authoring Animation-Ready 3D Characters (Read more on arXiv or HuggingFace) | Houqiang Li, Wengang Zhou, Kai Ma, Jinxu Xiang, jasongzy | Here is a summary of the AI research paper following your strict guidelines: i) 1-line summary: A data-driven framework, Make-It-Animatable, rapidly generates animation-ready 3D character models from various input representations, achieving significant speed improvements over existing methods. ii) Main research question/objective: To develop an efficient and generalizable framework for automatically creating animation-ready 3D character models, regardless of their initial pose, shape, or representation (mesh or 3D Gaussian splats). iii) Key methodology: A unified framework incorporating a particle-based shape autoencoder, coarse-to-fine shape representation, and a structure-aware transformer for bone modeling and blend weight generation. iv) Primary results: The framework processes each character in approximately one second; on the Mixamo dataset, the method achieved 82.5% IoU in skeleton prediction compared to RigNet’s 53.5%. v) Principal implication for AI practitioners: The Make-It-Animatable framework provides a highly efficient and flexible solution for generating animation-ready 3D characters suitable for real-time applications such as virtual reality and gaming; the sub-second processing time represents a substantial advancement over existing methods. |
| ChatRex: Taming Multimodal LLM for Joint Perception and Understanding (Read more on arXiv or HuggingFace) | Yihao Chen, Yuda Xiong, Yuqin Yang, Gen luo, Qing Jiang | ChatRex enhances multimodal large language models (MLLMs) for joint perception and understanding tasks. The research addresses the poor perception performance of existing MLLMs due to modeling conflicts and limited training data. The key methodology involves a decoupled architecture, treating object detection as a retrieval task based on proposals from a universal proposal network and utilizing a new multi-granularity dataset, Rexverse-2M. ChatRex achieved 48.5 mAP on COCO object detection, comparable to specialized object detectors. This suggests MLLMs can be significantly improved for fine-grained perception tasks, broadening their applicability for AI practitioners working on tasks requiring both visual understanding and accurate object detection. |
| Training and Evaluating Language Models with Template-based Data Generation (Read more on arXiv or HuggingFace) | yifAI | Here’s a summary of the AI research paper following the specified guidelines: i) This paper introduces Template-based Data Generation (TDG) to create a large-scale mathematical dataset for training and evaluating large language models (LLMs). ii) The main objective was to address the scarcity of high-quality, large-scale datasets for training LLMs on complex mathematical reasoning tasks. iii) The key methodology employed was TDG, using GPT-4 to automatically generate parameterized meta-templates for synthesizing a vast array of high-quality math problems and solutions. This involved a simultaneous generation and verification process. iv) The primary result is the creation of TemplateMath Part I: TemplateGSM, a dataset containing over 7 million synthetically generated grade school math problems, each with code-based and natural language solutions. v) The principal implication for AI practitioners is the availability of a large-scale, high-quality mathematical dataset (TemplateGSM) that addresses a significant barrier in training LLMs for sophisticated mathematical reasoning, potentially enabling significant advancements in LLM capabilities for mathematical problem-solving. |
Papers for 2024-11-27
| Title | Authors | Summary |
|---|---|---|
| ShowUI: One Vision-Language-Action Model for GUI Visual Agent (Read more on arXiv or HuggingFace) | Shiwei Wu, Zhengyuan Yang, Difei Gao, Linjie Li, Kevin Qinghong Lin | ShowUI is a vision-language-action model designed for building GUI visual agents. The research aimed to develop a lightweight, efficient model for GUI automation tasks like navigation and grounding by addressing challenges in visual modeling, action integration, and training data curation. The key methodologies included UI-Guided Visual Token Selection for efficient visual processing, Interleaved Vision-Language-Action Streaming to unify different modalities, and a curated dataset with a rebalancing strategy. ShowUI achieved 75.1% accuracy on zero-shot screenshot grounding using a 2B parameter model trained on 256K data. This implies that AI practitioners can leverage ShowUI’s efficient architecture and training methods to build performant GUI agents with limited computational resources and training data. |
| Star Attention: Efficient LLM Inference over Long Sequences (Read more on arXiv or HuggingFace) | Boris Ginsburg, Fei Jia, Shantanu Acharya | Star Attention is a block-sparse attention mechanism for efficient inference of transformer-based LLMs on long sequences. The research aimed to reduce the computational cost and improve the speed of LLM inference on long sequences. The two-phase method processes context with blockwise-local attention using anchor blocks, followed by global attention for query and response tokens to all cached key-value vectors. Star Attention achieved up to 11x speedup versus Ring Attention while maintaining 95-100% accuracy on the RULER benchmark with sequence lengths up to 128K. This allows AI practitioners to utilize LLMs with significantly longer context lengths while maintaining high accuracy and drastically reduced inference time and computational cost. |
| Rethinking Token Reduction in MLLMs: Towards a Unified Paradigm for Training-Free Acceleration (Read more on arXiv or HuggingFace) | Honggang Chen, Donglin Wang, Pengxiang Ding, Xuyang Liu, Yuhang Han | This paper introduces a unified “filter-correlate-compress” paradigm for training-free token reduction in Multimodal Large Language Models (MLLMs). The research aims to accelerate MLLM inference by reducing visual token quantity while preserving essential information, without requiring retraining. The proposed FiCoCo method suite, implementing this paradigm, decomposes token reduction into three distinct pipeline stages: filtering redundant tokens, correlating discarded information to retained tokens, and compressing the token set. Experimental results on LLaVA-1.5-7B show up to an 82.4% FLOPs reduction with minimal performance impact, outperforming other training-free methods. This offers AI practitioners a plug-and-play method for significantly improving the inference efficiency of MLLMs, facilitating practical deployment of these computationally demanding models. |
| MME-Survey: A Comprehensive Survey on Evaluation of Multimodal LLMs (Read more on arXiv or HuggingFace) | Xinyu Fang, Bo Li, Shukang Yin, Chaoyou Fu, yifanzhang114 | This paper surveys evaluation methods for Multimodal Large Language Models (MLLMs). The objective is to provide a comprehensive overview of MLLM evaluation to aid researchers in selecting appropriate benchmarks and developing better evaluation methods. The paper categorizes benchmarks by evaluated capabilities (foundational, behavioral, application-focused), summarizes benchmark construction processes, and discusses evaluation methods (human, LLM/MLLM, script-based) and metrics. MME-RealWorld, the largest manually annotated benchmark, contains 29K question-answer pairs and achieves a maximum accuracy of only 60% with state-of-the-art MLLMs on several real-world tasks. AI practitioners should consider the limitations of current MLLMs on complex real-world tasks when designing applications and prioritize benchmark selection and development based on specific application requirements. |
| TEXGen: a Generative Diffusion Model for Mesh Textures (Read more on arXiv or HuggingFace) | Ying-Tian Liu, Yuan-Chen Guo, Xin Yu, Lp256, yuanze1024 | TEXGen is a generative diffusion model for synthesizing high-resolution textures for 3D meshes. The research aimed to develop a feed-forward model for generalizable mesh texturing, avoiding test-time optimization common in previous methods. A novel hybrid 2D-3D network architecture, combining UV space convolutions with 3D point cloud attention, was employed. The model achieved a FID score of 34.53 and KID score of 11.94 × 10⁻⁴ on multi-view renderings of textured meshes, outperforming existing methods. This provides AI practitioners with a fast and effective method for generating high-quality textures for diverse 3D models, eliminating the need for computationally expensive per-object optimization. |
| Pathways on the Image Manifold: Image Editing via Video Generation (Read more on arXiv or HuggingFace) | David Bensaïd, Roy Velich, Daniel Silver, Gal Yona, Noam Rotstein | Frame2Frame (F2F) reformulates image editing as a video generation task to improve edit accuracy and image preservation. The research aims to overcome limitations of existing text-guided diffusion models for image editing, such as difficulty adhering to complex edit instructions and loss of source image fidelity. F2F uses a three-step process: generating temporal editing captions from source image and edit prompt using a VLM (ChatGPT-40), generating a video sequence with a pretrained video diffusion model (CogVideoX) conditioned on the temporal caption, and selecting the optimal edited frame using a VLM. On the TEdBench benchmark, F2F achieved a CLIP score of 0.63 for target edit accuracy, outperforming competing methods. This approach offers AI practitioners a novel method for high-fidelity image manipulation by leveraging the temporal coherence of video generation models, though the computational cost and potential for unintended camera motion effects are noted as limitations. |
| SketchAgent: Language-Driven Sequential Sketch Generation (Read more on arXiv or HuggingFace) | Judith E Fan, Alex Zhao, Kristine Zheng, Tamar Rott Shaham, Yael Vinker | SketchAgent generates sketches from text prompts using a sequential, stroke-based approach guided by multimodal large language models (LLMs). The objective is to create a language-driven sketching system capable of generating diverse, dynamic sketches and supporting human-computer collaborative sketching. The methodology involves prompting a frozen multimodal LLM to generate string-based drawing actions on a numbered grid canvas, which are then converted into Bézier curves and rendered. Using Claude3.5-Sonnet as the backbone LLM, SketchAgent achieved a Top-1 CLIP zero-shot classification accuracy of 23% on a 50-category QuickDraw sketch generation task. This sequential approach, leveraging off-the-shelf LLMs, offers AI practitioners a new method for developing interactive and dynamic sketch generation systems, eliminating the need for training or fine-tuning specialized models. |
| Learning 3D Representations from Procedural 3D Programs (Read more on arXiv or HuggingFace) | Zezhou Cheng, Xuweiyi Chen | This paper investigates learning 3D representations from procedurally generated data rather than semantically rich datasets. The research explores whether self-supervised learning methods can effectively learn 3D representations from synthetic shapes created via procedural programs and how these compare to representations learned from real-world 3D models. The study uses Point-MAE, a masked autoencoding framework, to train on a synthetic dataset of 150K procedurally generated 3D point clouds and compares performance with Point-MAE trained on ShapeNet. On ScanObjectNN’s PB-T50-RS benchmark, Point-MAE trained on synthetic shapes achieves 85.46% accuracy, compared to 85.18% for Point-MAE trained on ShapeNet. This suggests that procedurally generated data can be a viable alternative to real-world datasets for self-supervised 3D representation learning, potentially mitigating challenges related to data acquisition and copyright for AI practitioners working with 3D data. |
| SAR3D: Autoregressive 3D Object Generation and Understanding via Multi-scale 3D VQVAE (Read more on arXiv or HuggingFace) | XIngang Pan, Tengfei Wang, Shangchen Zhou, Yushi Lan, Yongwei Chen | SAR3D is a novel framework for fast 3D object generation and detailed understanding. The research sought to determine if autoregressive models could be effectively applied to both fast 3D object generation and detailed understanding. The key methodology involves a multi-scale 3D Vector-Quantized Variational Autoencoder (VQVAE) to tokenize 3D objects and a next-scale prediction training approach for autoregressive modeling. SAR3D achieves 3D object generation in 0.82 seconds on an A6000 GPU. This fast generation speed, coupled with the model’s ability to facilitate detailed 3D understanding through LLM finetuning, offers AI practitioners a more efficient method for both creating and interpreting 3D content. |
| DreamMix: Decoupling Object Attributes for Enhanced Editability in Customized Image Inpainting (Read more on arXiv or HuggingFace) | Ping Hu, Liqian Ma, Lu Zhang, Pengxiang Li, Yicheng Yang | DreamMix is a diffusion-based generative model for subject-driven image inpainting that allows editing object attributes while preserving identity. The research aimed to improve the editability of inserted objects in subject-driven image inpainting while maintaining identity preservation. The key methodology involves a disentangled inpainting framework with local content generation and global context harmonization, an attribute decoupling mechanism, and a textual attribute substitution module. In user studies, DreamMix received a 55% preference for identity preservation and a 74% preference for attribute editing. This provides AI practitioners with a more controllable and effective tool for customized image inpainting applications, enhancing both object insertion accuracy and text-driven attribute editing. |
| VLRewardBench: A Challenging Benchmark for Vision-Language Generative Reward Models (Read more on arXiv or HuggingFace) | Yifan Song, Xuqing Yang, Zhihui Xie, Yuancheng Wei, Lei Li | VL-RewardBench is introduced as a challenging benchmark for evaluating vision-language generative reward models (VL-GenRMs). The research aimed to create a robust benchmark to assess the reliability and effectiveness of VL-GenRMs in aligning and evaluating multimodal AI systems. The benchmark was constructed using an AI-assisted annotation pipeline incorporating ensemble filtering with small LVLMs for general and hallucination tasks, and AI-aided preference labeling for complex reasoning tasks, across datasets like WildVision, VLFeedback, and MMMU-Pro. Evaluation across 16 LVLMs revealed that even GPT-4o achieved only 62.4% macro-average accuracy on the benchmark, with many smaller models performing near chance levels. The strong correlation (Pearson’s r > 0.9) between VL-RewardBench performance and downstream Best-of-N sampling accuracy on MMMU-Pro provides AI practitioners with a reliable metric for selecting and developing effective VL-GenRMs for practical alignment tasks. |
| SALOVA: Segment-Augmented Long Video Assistant for Targeted Retrieval and Routing in Long-Form Video Analysis (Read more on arXiv or HuggingFace) | Yong Man Ro, Hosu Lee, Hyunjun Kim, Junho Kim | SALOVA enhances long-form video understanding in Large Multi-modal Models (LMMs) by retrieving relevant video segments. The research aimed to improve LMM comprehension of lengthy videos, addressing limitations in context length and memory overhead. The key methodology involved a novel video-LLM framework with a dynamic routing mechanism and spatio-temporal projector to retrieve relevant segments based on user queries, trained on a newly created “SceneWalk” dataset of densely captioned long videos. SALOVA-Qwen (7B) achieved 55.6% accuracy on the Video-MME long video benchmark, surpassing other open-sourced models with similar parameter sizes. This targeted retrieval approach offers AI practitioners a more efficient and contextually aware method for processing long videos, minimizing information loss and improving response relevance in LMMs. |
| Low-Bit Quantization Favors Undertrained LLMs: Scaling Laws for Quantized LLMs with 100T Training Tokens (Read more on arXiv or HuggingFace) | Haitao Mi, Zhisong Zhang, Thomas Hartvigsen, Tao Ge, Xu Ouyang | This paper investigates the impact of low-bit quantization on large language models (LLMs) at different training levels. The research aims to understand how quantization-induced degradation (QiD) relates to training tokens, model size, and bit width. The researchers analyzed over 1500 quantized LLM checkpoints from the Pythia suite, using GPTQ for 2-, 3-, and 4-bit quantization and measuring QiD on the RefinedWeb dataset. They derived scaling laws, finding that a 70B parameter LLM requires over 17 trillion training tokens to achieve a QiD greater than 0.2 with 4-bit quantization. AI practitioners should consider an LLM’s training level when evaluating or applying low-bit quantization, as fully trained models exhibit significantly higher QiD, posing challenges for deployment. |
| MolReFlect: Towards In-Context Fine-grained Alignments between Molecules and Texts (Read more on arXiv or HuggingFace) | Jingdi Le, Wei Liu, Yunqing Liu, Jiatong Li, qq8933 | MolReFlect improves molecule-caption translation in LLMs by focusing on fine-grained alignments between molecular sub-structures and textual phrases. The research aimed to address the challenge of aligning molecules and their corresponding captions with greater granularity and explainability than existing methods. A teacher-student framework was used, where a larger teacher LLM extracts fine-grained alignments, which are then refined and used to fine-tune a smaller student LLM via Chain-of-Thought In-Context Molecule Tuning (CoT-ICMT). On the ChEBI-20 dataset, MolReFlect with Mistral-7B achieved a BLEU-4 score of 0.608 for molecule-to-caption generation, outperforming the previous best score by 4.6%. This work highlights the importance of fine-grained alignments for improving the accuracy and explainability of LLMs in molecule-caption translation, enabling more effective application in molecule discovery and related tasks. |
| Visual Counter Turing Test (VCT^2): Discovering the Challenges for AI-Generated Image Detection and Introducing Visual AI Index (V_AI) (Read more on arXiv or HuggingFace) | Abhilekh Borah, Sainath Reddy Sankepally, Subhankar Ghosh, Shashwat Bajpai, Nasrin Imanpour | This paper introduces a benchmark and a metric for evaluating AI-generated image detection and quality. The research aims to assess the effectiveness of current AI-generated image detection (AGID) methods and propose a new evaluation framework. The researchers created the Visual Counter Turing Test (VCT²) benchmark dataset (~130K images) using prompts from Twitter and MS COCO and tested 15 state-of-the-art AGID methods. Results show significant limitations in existing AGID methods, with Midjourney 6 generated images achieving a 93.65 on the newly proposed Visual AI Index (VAI), exceeding the average real image VAI score of 85.61. This indicates a need for AI practitioners to develop more robust AGID techniques capable of detecting high-quality synthetic images generated by advanced models like Midjourney 6, as current methods are proving insufficient. |
| AnchorCrafter: Animate CyberAnchors Saling Your Products via Human-Object Interacting Video Generation (Read more on arXiv or HuggingFace) | Xiaodong Cun, Yong Zhang, Juan Cao, Ziyao Huang, Ziyi Xu | AnchorCrafter generates realistic anchor-style product promotion videos by animating human images with objects and motion controls. The research aimed to address the limitations of existing pose-guided human video generation methods in depicting realistic human-object interactions (HOI). The system uses a diffusion-based video generation model with novel HOI-appearance perception, HOI-motion injection, and HOI-region reweighting loss components. AnchorCrafter achieved a 0.848 Object-IoU, significantly higher than comparison methods, demonstrating improved object motion accuracy. This work provides AI practitioners with a tool for creating realistic and controllable product promotion videos with animated human presenters interacting naturally with products, advancing the field of video generation for e-commerce and related applications. |
Papers for 2024-11-26
| Title | Authors | Summary |
|---|---|---|
| Material Anything: Generating Materials for Any 3D Object via Diffusion (Read more on arXiv or HuggingFace) | Qing Wang, Ziwei Liu, Tengfei Wang, xanderhuang | Material Anything generates physically-based rendering (PBR) materials for 3D objects under diverse lighting and texture conditions. The objective is to create a robust, automated method for generating realistic PBR materials for any 3D object, regardless of its initial texture or lighting. The method uses a two-stage pipeline: an image-space material diffusion model with a confidence mask to handle various lighting scenarios, followed by UV-space material refinement for consistency. On a dataset of textured objects, Material Anything achieves a CLIP score of 89.70, demonstrating improved alignment with text prompts compared to existing methods. This provides AI practitioners with a unified framework for efficient, high-quality PBR material generation, potentially streamlining workflows in applications like game development, virtual reality, and product visualization. |
| Large-Scale Text-to-Image Model with Inpainting is a Zero-Shot Subject-Driven Image Generator (Read more on arXiv or HuggingFace) | Sungroh Yoon, Heeseung Kim, Jooyoung Choi, Chaehun Shin | Diptych Prompting performs zero-shot subject-driven text-to-image generation through diptych inpainting with a large-scale text-to-image model. The research aimed to develop a zero-shot method for subject-driven text-to-image generation that improves subject alignment compared to existing encoder-based image prompting methods. The key methodology involved arranging a reference image in the left panel of a diptych, masking the right panel, and using a text prompt describing the desired context for inpainting the right panel with FLUX, while enhancing cross-attention between panels and removing the reference image background. In a human preference study focusing on subject alignment, Diptych Prompting achieved a 77.9% win rate compared to existing methods. This provides AI practitioners with a novel, effective technique for zero-shot, subject-driven image generation using the inpainting capabilities of large-scale text-to-image models. |
| From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judge (Read more on arXiv or HuggingFace) | Chengshuai Zhao, Alimohammad Beigi, Liangjie Huang, Bohan Jiang, Dawei Li | This paper surveys the emerging field of using large language models (LLMs) as judges for various AI tasks. The paper aims to provide a comprehensive overview of LLM-based judgment to advance the field. The authors categorize and analyze existing LLM-as-a-judge methods based on input (point-wise, pair/list-wise) and output (score, ranking, selection) formats, and propose a taxonomy spanning judging attributes, methodologies (tuning, prompting), and applications (evaluation, alignment, retrieval, reasoning). In a benchmark by Zheng et al. (2023), GPT-4 achieved near-human performance when judging open-ended text generation. AI practitioners can leverage LLMs as automated judges for enhanced evaluations, alignment procedures, retrieval tasks, and complex reasoning pipelines, potentially achieving human-level performance in judging open-ended text generation. |
| Knowledge Transfer Across Modalities with Natural Language Supervision (Read more on arXiv or HuggingFace) | Marco Grangetto, Emanuele Aiello, luca-molinaro, carloalbertobarbano | This paper introduces Knowledge Transfer, a method for teaching pre-trained visual models novel concepts using only textual descriptions. The research aims to determine if leveraging pre-existing visual knowledge within a model, combined with textual descriptions, can enable the model to learn new visual concepts without visual examples. The core methodology involves synthesizing images via model inversion based on textual descriptions of novel concepts, and then fine-tuning the visual encoder with a contrastive loss (InfoNCE) to align visual and textual features. In experiments on rare image concepts, CLIP ViT-B/32 achieved 100% accuracy on “Gyroscope” after Knowledge Transfer, compared to 0% baseline accuracy. This demonstrates the potential for AI practitioners to efficiently introduce new concepts into pre-trained visual models without the need for extensive labeled image datasets, facilitating rapid model adaptation and reducing data acquisition costs. |
| MH-MoE:Multi-Head Mixture-of-Experts (Read more on arXiv or HuggingFace) | Furu Wei, Shuming Ma, Xun Wu, Shaohan Huang | This paper presents a novel implementation of Multi-Head Mixture-of-Experts (MH-MoE) for improved efficiency and performance. The objective is to maintain FLOPS and parameter parity with standard Sparse Mixture-of-Experts (SMoE) models while leveraging the multi-head mechanism of MH-MoE. The key methodology involves adding a “heads” dimension and two linear projection layers, adjusting the intermediate dimension and number of experts to maintain FLOPS parity. Experiments on language models show that MH-MoE achieves a perplexity of 10.51 on the RedPajama dataset with 3 heads and 100,000 training steps, outperforming standard SMoE (10.90) and fine-grained SMoE (10.74). This implies that AI practitioners can leverage this MH-MoE implementation to improve the performance and efficiency of large language models by using a multi-head attention structure within the MoE framework. |
| DreamRunner: Fine-Grained Storytelling Video Generation with Retrieval-Augmented Motion Adaptation (Read more on arXiv or HuggingFace) | Mohit Bansal, Jaehong Yoon, Han Lin, Jialu Li, Zun Wang | DREAMRUNNER generates long-form, multi-scene storytelling videos with fine-grained control over object motions and appearances. The research addresses the challenge of creating coherent and dynamic storytelling videos with complex object interactions and transitions. The methodology involves hierarchical story planning with an LLM, retrieval-augmented test-time adaptation for learning motion and subject priors, and a novel spatial-temporal region-based 3D attention and prior injection module (SR3AI) for video generation. On the DreamStorySet benchmark, DREAMRUNNER achieved a 13.1% relative improvement in character consistency (CLIP score) compared to VLogger. This improvement in character consistency offers AI practitioners a more effective method for generating realistic and coherent characters in long-form video content, contributing to more engaging and believable storytelling. |
| Factorized Visual Tokenization and Generation (Read more on arXiv or HuggingFace) | Zheng Zhang, Pichao Wang, Ziteng Gao, Jianxiong Gao, Zechen Bai | FQGAN improves visual tokenization for image generation by factorizing large codebooks. The research aims to address the instability and performance saturation of traditional VQ-based tokenizers when scaling codebook size. The core methodology involves decomposing a large codebook into smaller sub-codebooks, applying disentanglement regularization, and integrating representation learning with pre-trained vision models like CLIP and DINOv2. FQGAN achieves state-of-the-art reconstruction FID (rFID) of 0.24 on ImageNet 256x256 validation set with an 8x downsampling ratio and a factorized 3x16,384 codebook. This indicates that AI practitioners can use FQGAN to achieve significantly improved image reconstruction quality and potentially better downstream generation performance when using VQ-based tokenizers. |
| O1 Replication Journey – Part 2: Surpassing O1-preview through Simple Distillation, Big Progress or Bitter Lesson? (Read more on arXiv or HuggingFace) | Yuxiang Zheng, Yixiu Liu, Xuefeng Li, Haoyang Zou, Zhen Huang | This paper examines replicating OpenAI’s O1 model capabilities, particularly focusing on knowledge distillation. The research aims to evaluate if simple distillation from O1’s API, combined with supervised fine-tuning, can surpass O1-preview performance. The key methodology involved distilling O1’s API responses for long-thought chains and fine-tuning a base language model (Qwen2.5-Math-72B) on this distilled data. Their distilled and fine-tuned 72B parameter model outperformed O1-preview on the AIME2024 (American Invitational Mathematics Examination) dataset, scoring 13/30 compared to O1-preview’s 12/30. The primary implication for AI practitioners is that while distillation offers rapid performance gains, over-reliance on it may hinder the development of novel AI techniques and potentially create a technological dependency, limiting future breakthroughs. |
| GMAI-VL & GMAI-VL-5.5M: A Large Vision-Language Model and A Comprehensive Multimodal Dataset Towards General Medical AI (Read more on arXiv or HuggingFace) | Zhe Chen, Bin Fu, Wei Li, Yanzhou Su, foreverbeliever | GMAI-VL, a large vision-language model, achieves state-of-the-art results on multimodal medical tasks using the new GMAI-VL-5.5M dataset. The research aimed to improve general medical AI (GMAI) by addressing the lack of specialized medical knowledge in existing large vision-language models. Researchers created the GMAI-VL-5.5M dataset by converting 219 specialized medical imaging datasets into 5.5 million image-text pairs using an annotation-guided data generation methodology and a three-stage training process (shallow alignment, deep alignment, instruction tuning) for the GMAI-VL model. GMAI-VL achieved an average accuracy of 88.48% on the OmniMedVQA benchmark. This provides AI practitioners with a high-performing, specialized model and a comprehensive multimodal dataset for developing and evaluating general medical AI applications. |
| One Diffusion to Generate Them All (Read more on arXiv or HuggingFace) | Aniruddha Kembhavi, Christopher Clark, Sangho Lee, Tuan Pham, Duong H. Le | OneDiffusion is a unified diffusion model for bidirectional image synthesis and understanding across diverse tasks. The research aimed to develop a single diffusion model capable of performing multiple image-related tasks without task-specific modules or training. The core methodology involves modeling all inputs and outputs as a sequence of “views” with varying noise levels during training, enabling flexible conditioning and generation at inference. On the GenEval benchmark for text-to-image generation at 1024x1024 resolution, OneDiffusion achieved a score of 0.65. This unified approach offers AI practitioners a more versatile and scalable solution for image-related tasks, potentially simplifying model development and deployment by eliminating the need for multiple specialized models. |
| VisualLens: Personalization through Visual History (Read more on arXiv or HuggingFace) | Zhaojiang Lin, Yi Lu, Kai Sun, Deqing Fu, Wang Bill Zhu | VisualLens is a novel approach for personalized recommendations leveraging a user’s task-agnostic visual history. The research investigates whether visual history can improve personalized recommendations. The methodology involves retrieving relevant images from the user’s history, generating a preference profile using image embeddings, captions, and extracted aspect words, and matching this profile to candidate items using a multimodal LLM. VisualLens achieved 82-91% Hit@10 on created benchmarks, outperforming state-of-the-art methods like UniMP by ~10% and GPT-40 by up to 4.6% on Hit@3. This suggests AI practitioners can leverage users’ visual data, such as photos from reviews or social media, to significantly enhance personalization in recommendation systems, even outperforming large language models. |
| Cautious Optimizers: Improving Training with One Line of Code (Read more on arXiv or HuggingFace) | Qiang Liu, Bo Liu, Lizhang Chen, Kaizhao Liang | Cautious Optimizers improve the training speed of momentum-based optimizers with a simple, single-line code modification. The research aims to develop a faster and more stable optimizer for large model training that requires minimal implementation effort. The core methodology involves introducing a mask that selectively applies updates based on alignment between the proposed update direction and the current gradient. On the LLaMA 1B language model, the Cautious AdamW variant achieved a 1.47x speedup compared to standard AdamW. This allows AI practitioners to train large models more efficiently with virtually no code changes or computational overhead, potentially enabling faster experimentation and model development cycles. |
| The Impossible Test: A 2024 Unsolvable Dataset and A Chance for an AGI Quiz (Read more on arXiv or HuggingFace) | Forrest McKee, David Noever | This research evaluates large language models’ (LLMs) ability to acknowledge uncertainty on unsolvable problems. The research sought to determine how well LLMs admit ignorance rather than generate incorrect responses to fundamentally unsolvable questions. Twelve state-of-the-art LLMs, both open and closed-source, were tested on a curated dataset of 675 unsolvable graduate-level problems using multiple-choice questions that included “I don’t know” as a correct answer. The best-performing models achieved 62-68% accuracy in admitting “I don’t know,” with GPT-4 demonstrating higher uncertainty acknowledgement on more challenging problems (35.8%) compared to simpler problems (20.0%). This finding highlights the importance of incorporating uncertainty recognition into LLM training and evaluation frameworks, prompting AI practitioners to develop methods for LLMs to distinguish between solvable and unsolvable problems as a potential marker for advanced reasoning capabilities and a critical aspect of responsible AI development. |
| SplatFlow: Multi-View Rectified Flow Model for 3D Gaussian Splatting Synthesis (Read more on arXiv or HuggingFace) | Soonwoo Kwon, Jin-Young Kim, Jiho Jang, Byeongjun Park, Hyojun Go | SplatFlow is a novel framework for text-driven 3D Gaussian Splatting (3DGS) scene generation and editing. The research aims to create a unified framework for generating and editing 3DGS scenes from text prompts, addressing the limitations of existing specialized methods. The core methodology involves a multi-view rectified flow (RF) model trained to generate multi-view consistent images, depths, and camera poses, along with a Gaussian Splatting Decoder (GSDecoder) to convert these into 3DGS representations. On the MVImgNet dataset, SplatFlow achieves a FID score of 34.85, outperforming the Director3D baseline (FID 39.55). This provides AI practitioners with a more versatile and efficient tool for generating and editing complex 3D scenes directly from text prompts, simplifying content creation pipelines. |
| Predicting Emergent Capabilities by Finetuning (Read more on arXiv or HuggingFace) | Sergey Levine, Dan Klein, Eric Wallace, sea-snell | This paper investigates predicting the emergence of capabilities in large language models (LLMs). The research asks: can few-shot emergent capabilities in future, larger LLMs be predicted by finetuning current, smaller LLMs? The core methodology involves finetuning smaller LLMs with varying amounts of data, fitting a parametric “emergence law” to model how the point of emergence shifts with data, and extrapolating this law to the few-shot setting. On MMLU, the method predicts emergence using models trained with ~10²² FLOPS, while the smallest post-emergence model required ~5 * 10²² FLOPS, enabling prediction 4-5x in advance in terms of FLOPS. This allows AI practitioners to potentially assess the future capabilities and emergent behavior of larger LLMs before they are trained, informing architectural choices and resource allocation. |
| SegBook: A Simple Baseline and Cookbook for Volumetric Medical Image Segmentation (Read more on arXiv or HuggingFace) | Zhongying Deng, Haoyu Wang, Yanjun Li, Ying Chen, Jin Ye | This paper benchmarks the transfer learning capabilities of full-body CT pre-trained models for volumetric medical image segmentation. The research investigates under what conditions pre-trained models can effectively transfer to diverse downstream medical image segmentation tasks across varying modalities, targets, and dataset sizes. The study employs STU-Net, a scalable U-Net architecture, pre-trained on the TotalSegmentor dataset and fine-tuned on 87 public datasets. Fine-tuning improved average Dice Similarity Coefficient (DSC) by 2.80% for the STU-Net-huge model across all datasets. This research demonstrates the efficacy of full-body CT pre-training for cross-modality and cross-target transfer in medical image segmentation, offering AI practitioners pre-trained models and a benchmark for developing and evaluating transfer learning techniques for volumetric medical image analysis. |
| From CISC to RISC: language-model guided assembly transpilation (Read more on arXiv or HuggingFace) | Abdulrahman Mahmoud, Rania Hossam, Chaimaa Abi, Ahmed Heakl | CRT, a lightweight LLM-based transpiler, automatically converts x86 assembly code to ARM and RISC-V assembly. The research aimed to develop a direct translation method between x86 (CISC) and ARM/RISC-V (RISC) architectures that preserves correctness without virtualization overhead. The methodology involved training various small-scale LLMs on a dataset of 500k C programs compiled to x86 and ARM/RISC-V, employing an extended tokenizer and hardware-informed training optimizations. The transpiler achieved 79.25% translation accuracy from x86 to ARMv5 and 88.68% accuracy from x86 to RISC-V64. This demonstrates the potential of using LLMs for efficient cross-architecture assembly transpilation, offering AI practitioners a new approach to code portability across diverse hardware ISAs without reliance on dynamic binary translation or emulation. |
| Best of Both Worlds: Advantages of Hybrid Graph Sequence Models (Read more on arXiv or HuggingFace) | Bryan Perozzi, Clayton Sanford, Mahdi Karami, Ali Parviz, Ali Behrouz | This paper investigates the strengths and weaknesses of different sequence models for graph-structured data. The research aims to determine which sequence models and tokenization strategies are most effective for various graph tasks. The authors introduce a unifying framework, Graph Sequence Model (GSM), and analyze sequence model performance on tasks including counting, connectivity, and shortest path. Results show no single sequence model or tokenizer consistently outperforms others across all tasks; for instance, a hybrid model combining Mamba and Transformer layers improved performance in most cases. This suggests AI practitioners should carefully select tokenization and sequence models based on the specific graph task, considering factors like local vs. global information needs and node ordering. |
Papers for 2024-11-25
| Title | Authors | Summary |
|---|---|---|
| Style-Friendly SNR Sampler for Style-Driven Generation (Read more on arXiv or HuggingFace) | Sungroh Yoon, Heeseung Kim, Yeongtak, chaehun, jychoi | This paper introduces a Style-friendly SNR sampler to improve style learning in text-to-image diffusion models during fine-tuning. The research aims to address the limitations of existing fine-tuning methods, which often fail to capture new artistic styles due to the use of object-centric objectives and noise distributions. The key methodology involves adjusting the noise level sampling during fine-tuning by biasing the signal-to-noise ratio (SNR) distribution towards higher noise levels (lower log-SNR values) where style features are observed to emerge. Experiments using FLUX-dev on the StyleDrop dataset showed a DINO image similarity score of 0.461 for the proposed method compared to 0.373 for the standard SD3 sampler, demonstrating improved style alignment. The Style-friendly SNR sampler enables more effective style template learning for personalized content creation, allowing AI practitioners to fine-tune text-to-image diffusion models for higher-fidelity style-driven generation. |
| TÜLU 3: Pushing Frontiers in Open Language Model Post-Training (Read more on arXiv or HuggingFace) | Hamish Ivison, Shengyi Huang, Valentina Pyatkin, Jacob Morrison, Nathan Lambert | TÜLU 3 is a family of open-source, state-of-the-art language models fine-tuned for enhanced post-training capabilities. The research aimed to develop a robust, open post-training recipe for language models that rivals closed, proprietary methods. Key methodologies included supervised fine-tuning, preference tuning with Direct Preference Optimization (DPO), and a novel Reinforcement Learning with Verifiable Rewards (RLVR) approach. TÜLU 3 70B outperformed Llama 3.1 Instruct 70B by 3.2 points on an aggregate evaluation suite. The primary implication for AI practitioners is the availability of a comprehensive, open-source recipe and accompanying resources (data, code, evaluation framework) to reproduce and adapt state-of-the-art post-training techniques for their own language models. |
| A Flexible Large Language Models Guardrail Development Methodology Applied to Off-Topic Prompt Detection (Read more on arXiv or HuggingFace) | Shaun Khoo, shingurding, gabrielchua | This paper introduces a data-free methodology for developing LLM guardrails, focusing on off-topic prompt detection. The research aimed to create a method for developing effective LLM guardrails in pre-production environments where real-world user data is unavailable. The key methodology involved using LLMs to generate synthetic datasets of on-topic and off-topic prompts and then training classifier models on this data. Fine-tuned cross-encoder and bi-encoder models achieved an F1 score of 0.99 on a synthetic dataset generated by GPT-40. This methodology enables AI practitioners to deploy LLM applications with pre-built safety measures for off-topic prompt detection even before real-world data becomes available, minimizing potential misuse from the outset. |
| OminiControl: Minimal and Universal Control for Diffusion Transformer (Read more on arXiv or HuggingFace) | Xinchao Wang, Qiaochu Xue, Xingyi Yang, Songhua Liu, Zhenxiong Tan | OminiControl integrates image conditions into Diffusion Transformers (DiTs) for diverse control tasks. The research aimed to develop a parameter-efficient method for both spatially and non-spatially aligned image control in DiTs. The key methodology involves reusing the model’s VAE encoder for processing condition images and integrating them as tokens within the DiT’s multi-modal attention mechanism. On the Canny-to-image task, OminiControl achieved a 0.38 F1-Score, significantly outperforming Stable Diffusion 1.5 based ControlNet (0.34) and T2I-Adapter (0.22), as well as Flux.1-based ControlNetPro (0.21). This allows AI practitioners to utilize a unified and efficient approach for implementing diverse image-based control within DiT architectures, simplifying implementation and reducing parameter overhead compared to previous specialized methods. |
| Large Multi-modal Models Can Interpret Features in Large Multi-modal Models (Read more on arXiv or HuggingFace) | Ziwei Liu, Bo Li, Yifei Shen, Kaichen Zhang | This paper presents a framework for interpreting and steering the internal representations of large multimodal models (LMMs). The research aims to understand the internal neural representations of LMMs, particularly how they encode semantic information. The key methodology involves training a Sparse Autoencoder (SAE) on LLaVA-NeXT data integrated into a specific LMM layer and interpreting learned features using a larger LMM (LLaVA-OV-72B) in a zero-shot manner. Results show the SAE features can steer LMM behavior, with some features exhibiting IOU scores above 0.5 with ground truth segmentation masks based on automatically generated explanations. This framework allows AI practitioners to better understand and potentially control the behavior of LMMs, including mitigating hallucinations and prompting desired outputs by manipulating specific internal features. |
| VideoEspresso: A Large-Scale Chain-of-Thought Dataset for Fine-Grained Video Reasoning via Core Frame Selection (Read more on arXiv or HuggingFace) | Xiu Su, Le Zhuo, Hairong Shi, Wei Huang, Songhao Han | VideoEspresso is a new dataset and framework for improving video reasoning capabilities of Large Vision Language Models (LVLMs). The research aimed to address the scarcity of high-quality, large-scale datasets for video reasoning tasks. The key methodology involved a semantic-aware pipeline to construct a VideoQA dataset with multimodal Chain-of-Thought (CoT) annotations, coupled with a Hybrid LVLMs Collaboration framework for reasoning. The proposed method outperformed existing baselines on 12 out of 14 video reasoning tasks, achieving 34.1% average accuracy, surpassing the top open-source model (InternVL2) by 5.4% and the closed-source model (GPT-40) by 7.7%. This dataset and framework provide AI practitioners with new resources and methods for developing and evaluating LVLMs with enhanced video reasoning capabilities, leading to more cost-effective and accurate performance. |
| Efficient Long Video Tokenization via Coordinated-based Patch Reconstruction (Read more on arXiv or HuggingFace) | Pieter Abbeel, Jinwoo Shin, Sihyun Yu, Huiwon Jang, younggyoseo | CoordTok, a novel video tokenizer, efficiently encodes long videos into a compact set of tokens by reconstructing patches based on sampled coordinates. The research aimed to develop a more efficient video tokenizer that leverages temporal coherence and scales to long video clips. The key methodology involved encoding videos into factorized triplane representations and training a decoder to reconstruct patches corresponding to randomly sampled (x,y,t) coordinates. CoordTok encodes a 128-frame, 128x128 resolution video into 1280 tokens, achieving similar reconstruction quality as baselines requiring 6144 or 8192 tokens. This efficient tokenization enables AI practitioners to train memory-intensive video generation models, like diffusion transformers, on significantly longer video sequences than previously feasible. |
| Novel View Extrapolation with Video Diffusion Priors (Read more on arXiv or HuggingFace) | Shijian Lu, Ling Shao, KunhaoLiu | ViewExtrapolator leverages stable video diffusion (SVD) to refine artifact-prone novel views rendered by radiance fields or point clouds, enabling novel view extrapolation beyond training views. The research aims to improve novel view extrapolation, where synthesized views are far outside the range of training views, which is a weakness of current radiance field methods. The key methodology involves rendering a video transitioning from a training view to the extrapolated view, then refining it with SVD by modifying its denoising process and using guidance and resampling annealing. On the LLFF-Extra dataset, ViewExtrapolator achieves a 0.378 LPIPS score compared to 0.429 for the baseline DRGS method. The paper does not specify if tuning SVD was required and if results improved further by fine-tuning SVD model. AI practitioners can utilize ViewExtrapolator as a post-processing method to significantly improve the visual quality of novel view extrapolations generated from existing 3D rendering techniques like radiance fields or point clouds. It should be noted that performance degrades with dynamic videos and extreme novel view angles. |
| MyTimeMachine: Personalized Facial Age Transformation (Read more on arXiv or HuggingFace) | David W. Jacobs, Annie N. Wang, Bang Gong, Jiaye Wu, Luchao Qi | MyTimeMachine (MyTM) personalizes facial age transformation using a few subject-specific images and a global aging prior. The research aimed to develop a personalized age transformation method that accurately reflects an individual’s appearance at a target age. MyTM leverages a novel Adapter Network trained on a personal photo collection (~50 images) to modify the latent features of a global age transformation network (SAM). In age regression evaluations, MyTM achieved an 11.7% improvement in identity preservation (IDsim = 0.67) compared to the best-performing baseline (FADING). AI practitioners can use MyTM to generate more accurate and personalized age-transformed faces, crucial for applications like visual effects in film or age progression for forensic investigations. |
| BALROG: Benchmarking Agentic LLM and VLM Reasoning On Games (Read more on arXiv or HuggingFace) | Maciej Wolczyk, Ulyana Piterbarg, Samuel Coward, Bartłomiej Cupiał, pagli98 | BALROG benchmarks the agentic capabilities of large language models (LLMs) and vision-language models (VLMs) in complex game environments. The research aims to evaluate LLMs’ and VLMs’ long-horizon reasoning and decision-making capabilities in dynamic settings. The benchmark uses six reinforcement learning environments: BabyAI, Crafter, TextWorld, Baba Is AI, MiniHack, and NetHack, with varying complexities and textual and visual observation modalities. GPT-4 achieved the highest average progression across all environments in the language-only setting at 32.34%. The significant performance gap between simpler and more complex games, as well as the drop in performance when using visual observations, highlights the need for AI practitioners to focus on improving VLMs’ vision-based decision-making and LLMs’ long-horizon planning abilities for more effective agent development. |
| One to rule them all: natural language to bind communication, perception and action (Read more on arXiv or HuggingFace) | Giuseppe Boccignone, Dimitri Ognibene, colo286 | This paper presents a novel architecture for robot task planning using Large Language Models (LLMs). The research aims to enable robots to understand natural language commands and autonomously generate actionable plans in dynamic environments. The core methodology involves a modified ReAct framework integrating LLMs with a semantic mapping system using scene graphs and feedback loops for real-time adaptation. In preliminary tests on simple robotic requests, the system achieved a 90% success rate. AI practitioners can leverage this approach to develop more robust and adaptable robots capable of understanding and executing complex tasks in real-world settings using natural language instructions. |
| WildLMa: Long Horizon Loco-Manipulation in the Wild (Read more on arXiv or HuggingFace) | Ge Yang, Sai Aneesh Suryadevara, Xuanbin Peng, Yuchen Song, Ri-Zhao Qiu | WildLMa is a framework for enabling quadruped robots to perform long-horizon loco-manipulation tasks in real-world environments. The research aims to develop a system that allows quadruped robots to perform complex, long-horizon manipulation tasks in unstructured environments. The methodology involves adapting a learned low-level whole-body controller for VR teleoperation, creating a library of generalizable visuomotor skills via imitation learning and heuristics (WildLMa-Skill), and using an LLM-based planner to coordinate skills for long-horizon tasks (WildLMa-Planner). WildLMa achieved a 71.2% average success rate across tabletop grasping, button pressing, and ground grasping tasks, exceeding baseline imitation learning methods by at least 20%. This work provides AI practitioners with a practical framework and techniques for developing robust and generalizable loco-manipulation skills for quadruped robots, potentially enabling real-world deployment for tasks such as cleaning or fetching objects. |
Papers for 2024-11-22
| Title | Authors | Summary |
|---|---|---|
| Enhancing the Reasoning Ability of Multimodal Large Language Models via Mixed Preference Optimization (Read more on arXiv or HuggingFace) | Yangzhou Liu, Yue Cao, Wenhai Wang, Zhe Chen, Weiyun Wang | This paper introduces Mixed Preference Optimization (MPO) to improve multimodal reasoning in Large Language Models (LLMs). The research aims to address the limited multimodal reasoning capabilities and distribution shift issues observed in open-source Multimodal LLMs (MLLMs), particularly with Chain-of-Thought (CoT) prompting. The authors develop MPO, combining supervised fine-tuning loss with preference, quality, and generation losses, and create MMPR, a large-scale multimodal reasoning preference dataset, using automated pipelines. InternVL2-8B-MPO, trained with MPO, achieves 67.0% accuracy on MathVista, an 8.7 point improvement over the baseline InternVL2-8B and comparable to the much larger InternVL2-76B. This suggests that MPO and MMPR can significantly improve the reasoning performance of smaller MLLMs, offering a potential pathway for developing more efficient and capable models for AI practitioners. |
| Marco-o1: Towards Open Reasoning Models for Open-Ended Solutions (Read more on arXiv or HuggingFace) | Tianqi Shi, Hao Wang, Bo Zeng, Huifeng Yin, Yu Zhao | Marco-01 is a large language model developed to enhance reasoning abilities for complex problem-solving. The research aims to determine if an OpenAI-style model can generalize to domains lacking clear standards and quantifiable rewards. The model uses Chain-of-Thought (CoT) fine-tuning, Monte Carlo Tree Search (MCTS), and a reflection mechanism. Marco-01 achieved a 90.40% accuracy on the English MGSM dataset, a +6.17% improvement over the baseline Qwen2-7B-Instruct. This indicates that combining CoT, MCTS, and reflection mechanisms can significantly improve the reasoning abilities of LLMs, offering AI practitioners new techniques for developing models capable of tackling complex, open-ended problems. |
| OpenScholar: Synthesizing Scientific Literature with Retrieval-augmented LMs (Read more on arXiv or HuggingFace) | Amanpreet Singh, Weijia Shi, Rulin Shao, jacquelinehe, akariasai | OpenScholar is a retrieval-augmented language model for synthesizing scientific literature. The research investigated whether large language models can effectively assist scientists in synthesizing the growing body of scientific literature. The study developed OpenScholar, a specialized retrieval-augmented LM that synthesizes citation-backed responses by retrieving from a datastore of 45 million open-access papers and iteratively refining outputs using self-feedback. OpenScholar-8B outperformed GPT-40 by 5% and PaperQA2 by 7% in correctness on the ScholarQABench benchmark. AI practitioners can leverage OpenScholar and similar retrieval-augmented LMs to access, synthesize, and cite scientific literature more effectively and accurately. |
| Multimodal Autoregressive Pre-training of Large Vision Encoders (Read more on arXiv or HuggingFace) | Michal Klein, Philipp Dufter, Xiujun Li, Mustafa Shukor, efini | AIMv2, a family of vision encoders, is pre-trained using a multimodal autoregressive objective. The research aims to develop a scalable and effective pre-training method for vision encoders that generalizes well to diverse downstream tasks. The method involves training a vision transformer encoder with a causal multimodal decoder that autoregressively generates image patches and text tokens from a unified multimodal sequence of image and text embeddings. The AIMv2-3B model achieved 89.5% top-1 accuracy on ImageNet-1k with a frozen trunk after high-resolution fine-tuning. This offers AI practitioners a straightforward, scalable, and high-performing vision encoder for various vision and multimodal applications, including zero-shot image recognition and multimodal instruction tuning. |
| Ultra-Sparse Memory Network (Read more on arXiv or HuggingFace) | Defa Zhu, Qiyang Min, Taoer, xyzed, FetchFortune | UltraMem, a novel architecture employing large-scale, ultra-sparse memory layers, aims to improve inference efficiency in large language models. The research sought to reduce inference latency while maintaining or exceeding the performance of Mixture of Experts (MoE) models, addressing MoE’s high memory access costs. The key methodology involves using Tucker decomposition for query-key retrieval within a memory layer and implicit value expansion to reduce memory access during training. Experiments show UltraMem achieves up to 6x faster inference than MoE with the same parameter count and computational cost at a batch size of 64. This allows AI practitioners to deploy larger language models with improved inference speed in resource-constrained environments and potentially improve scaling properties for even larger models. |
| Hymba: A Hybrid-head Architecture for Small Language Models (Read more on arXiv or HuggingFace) | Zijia Chen, Wonmin Byeon, Shizhe Diao, Yonggan Fu, Xin Dong | Hymba, a family of small language models (SLMs), integrates transformer attention and state space models (SSMs) within a hybrid-head parallel architecture for enhanced efficiency and performance. The research aimed to develop more efficient and performant SLMs by combining the strengths of attention mechanisms and SSMs while mitigating their individual weaknesses. The key methodology involved fusing attention and SSM heads in parallel within the same layer, incorporating learnable meta tokens, optimizing KV cache usage, and scaling model size and training data. Hymba-1.5B outperforms Llama-3.2-3B (a 3B parameter model) by 1.32% on average accuracy across commonsense reasoning tasks, while requiring an 11.67× smaller cache size and achieving 3.49× higher throughput. This result signifies that AI practitioners can achieve comparable or better performance with significantly smaller and more efficient SLMs using hybrid architectures like Hymba, potentially enabling broader deployment on resource-constrained devices. |
| Natural Language Reinforcement Learning (Read more on arXiv or HuggingFace) | Mengyue Yang, Haotian Fu, Ziyu Wan, Xidong Feng, Benjamin-eecs | This paper introduces Natural Language Reinforcement Learning (NLRL), a novel RL paradigm that uses natural language to represent core RL components. The objective is to improve reinforcement learning efficiency, stability, and interpretability by leveraging natural language and large language models (LLMs). The core methodology involves redefining RL principles (objectives, policy, value function, Bellman equation) as language-based constructs and implementing them with LLMs via prompting and gradient-based training. In Tic-Tac-Toe experiments, NLRL achieved higher win rates against baseline models, including a traditional PPO agent, reaching a win rate of 0.9. NLRL offers AI practitioners a new framework for building more interpretable and potentially more efficient RL agents by integrating the strengths of large language models into the reinforcement learning process, although the paper’s empirical evaluation focuses on relatively simple environments. |
| Insight-V: Exploring Long-Chain Visual Reasoning with Multimodal Large Language Models (Read more on arXiv or HuggingFace) | Winston Hu, Jingkang Yang, Hai-Long Sun, Zuyan, THUdyh | Insight-V is a system for enhancing visual reasoning in Multimodal Large Language Models (MLLMs). The research aimed to improve long-chain visual reasoning in MLLMs, addressing the lack of robust datasets and training strategies. A two-step pipeline generated structured reasoning data: a progressive strategy created diverse reasoning paths, and multi-granularity assessment ensured data quality; a multi-agent system, consisting of reasoning and summarization agents, was trained using supervised fine-tuning and iterative Direct Preference Optimization. Insight-V improved the performance of LLaVA-NeXT by an average of 7.0% across seven visual reasoning benchmarks. This suggests AI practitioners can significantly enhance MLLM visual reasoning capabilities by using specialized data generation pipelines and multi-agent system architectures with iterative DPO training. |
| Stable Flow: Vital Layers for Training-Free Image Editing (Read more on arXiv or HuggingFace) | Kfir Aberman, Egor Nemchinov, Ohad Fried, Or Patashnik, omriav | Stable Flow leverages the reduced diversity of flow-based diffusion models for consistent, training-free image editing. The research aimed to identify crucial layers in Diffusion Transformer (DiT) models for effective image editing without retraining. The methodology involved systematically bypassing individual DiT layers during image generation and measuring the perceptual impact using DINOv2, identifying “vital layers” essential for image formation. Injecting features from a source image into the vital layers of the edited image’s generation trajectory resulted in a CLIP image-text direction similarity score of 0.14, higher than other compared methods. This allows AI practitioners to perform various image edits, including non-rigid transformations and object manipulation, using a single, training-free mechanism by targeting these vital layers in flow-based DiT models. |
| UnifiedCrawl: Aggregated Common Crawl for Affordable Adaptation of LLMs on Low-Resource Languages (Read more on arXiv or HuggingFace) | Tae-Sun Chung, Akhil Kedia, Bethel Melesse Tessema | UnifiedCrawl improves Large Language Model (LLM) performance on low-resource languages using consumer-grade hardware. The research aimed to improve LLM performance in low-resource languages given data scarcity and limited compute resources. The authors developed UnifiedCrawl, a method to efficiently extract monolingual data from the Common Crawl corpus, and fine-tuned multilingual LLMs using quantization and low-rank adapters (QLoRA). Fine-tuning a 4.5B parameter XGLM model with UnifiedCrawl-Amharic data using QLoRA resulted in a 45% perplexity reduction from 35.6 to 19.6 compared to the original XGLM model. This demonstrates that using UnifiedCrawl and QLoRA allows practitioners to adapt large, pre-trained multilingual LLMs for low-resource languages using readily available hardware, promoting wider accessibility and affordability. |
| MagicDriveDiT: High-Resolution Long Video Generation for Autonomous Driving with Adaptive Control (Read more on arXiv or HuggingFace) | Zhenguo Li, Lanqing Hong, Bo Xiao, Kai Chen, Ruiyuan Gao | MagicDriveDiT generates high-resolution, long street-view videos for autonomous driving applications with precise control. The objective is to synthesize realistic and controllable high-resolution, long street-view videos suitable for autonomous driving applications. The paper uses a DiT-based diffusion model with flow matching, spatial-temporal conditional encoding, and a progressive bootstrapping training strategy incorporating variable video lengths and resolutions. MagicDriveDiT achieves a Frechet Video Distance (FVD) score of 94.84, significantly lower than baseline models, on the nuScenes dataset. AI practitioners working with autonomous driving systems can leverage MagicDriveDiT to create high-quality, controllable synthetic video datasets for training and testing perception models, potentially reducing reliance on real-world data collection. |
| Do I Know This Entity? Knowledge Awareness and Hallucinations in Language Models (Read more on arXiv or HuggingFace) | Neel Nanda, Senthooran Rajamanoharan, Oscar Obeso, Javier Ferrando | This paper investigates the mechanisms behind hallucinations in large language models, specifically focusing on entity recognition. The research aims to understand how language models determine whether they possess knowledge about a given entity and how this relates to hallucination. The researchers use sparse autoencoders (SAEs) to identify directions in the representation space of the model that correlate with known and unknown entities. They find that manipulating these “entity recognition” directions can causally influence the model’s refusal to answer or its tendency to hallucinate, achieving nearly 100% refusal for unknown entities when steering with the discovered latent direction. Steering with unknown entity latents disrupts the factual recall mechanism by reducing attention paid to entity tokens by downstream attention heads. This finding suggests that AI practitioners can potentially leverage and manipulate these latent directions to control hallucination and refusal behaviors in language models, directly impacting the reliability and factuality of generated text. |
| Patience Is The Key to Large Language Model Reasoning (Read more on arXiv or HuggingFace) | Yijiong Yu | This paper proposes a method to improve large language model reasoning by encouraging more detailed reasoning processes. The research aims to enhance complex problem-solving in LLMs without requiring extensive, costly training data. The key methodology involves using preference optimization (DPO) to train a model to favor detailed reasoning processes (positive examples) over concise answers (negative examples). Results demonstrate a 6.7% improvement on the GSM8k benchmark. This suggests AI practitioners can significantly improve LLM performance on complex tasks by training for more patient and thorough reasoning, even with limited data, though at the cost of increased inference time. |
Papers for 2024-11-21
| Title | Authors | Summary |
|---|---|---|
| SageAttention2 Technical Report: Accurate 4 Bit Attention for Plug-and-play Inference Acceleration (Read more on arXiv or HuggingFace) | Jun Zhu, Jia Wei, Pengle Zhang, Haofeng Huang, jt-zhang | SageAttention2 accelerates attention computation in transformer models using 4-bit quantization. The objective is to improve the efficiency of attention computation, particularly for long sequences, while maintaining accuracy comparable to full-precision attention. The key methodology involves quantizing Q and K matrices to INT4 using a per-warp granularity, P and V matrices to FP8 with per-channel granularity for V, and employing smoothing techniques for Q, K, and V to minimize quantization error. SageAttention2 achieves a peak performance of 485 TOPS on RTX4090, surpassing FlashAttention2 by about 3x. AI practitioners can use SageAttention2 as a plug-and-play module to significantly accelerate inference in various transformer-based models, including those for large language processing, image generation, and video generation, with negligible end-to-end metric loss. |
| VBench++: Comprehensive and Versatile Benchmark Suite for Video Generative Models (Read more on arXiv or HuggingFace) | Jiashuo Yu, Yinan He, Xiaojie Xu, Fan Zhang, Ziqi Huang | VBench++ is a comprehensive benchmark suite for evaluating text-to-video (T2V) and image-to-video (I2V) generative models. The research aimed to create a more effective and human-aligned evaluation framework for video generation models than existing metrics. The methodology involved designing a suite of 16 evaluation dimensions covering video quality, condition consistency, and trustworthiness, along with tailored prompts and evaluation methods, and collecting human preference annotations. VBench++ evaluations showed a high Spearman’s correlation with human preferences (e.g., ρ = 0.9651 for Subject Consistency). AI practitioners can use VBench++ to gain detailed insights into the strengths and weaknesses of different video generation models across various dimensions, enabling more informed model selection, training, and development for specific applications. |
| VideoAutoArena: An Automated Arena for Evaluating Large Multimodal Models in Video Analysis through User Simulation (Read more on arXiv or HuggingFace) | Mohan Kankanhalli, Jing Ma, Dongxu Li, teowu, Ziyang | VideoAutoArena automates the evaluation of large multimodal models (LMMs) for video analysis using simulated users. The research aimed to develop a more scalable and user-centric evaluation method for LMMs compared to traditional benchmarks. The key methodology involves using LMMs to simulate user personas, generate open-ended questions about videos, conduct pairwise model comparisons (battles), automatically judge responses using GPT-40, and rank models using an ELO rating system. GPT-40 achieved 87.29% agreement with human judges in selecting the better response. This automated arena provides AI practitioners with a cost-effective and scalable method for evaluating and comparing LMMs in user-centric video analysis tasks. |
| Is Your LLM Secretly a World Model of the Internet? Model-Based Planning for Web Agents (Read more on arXiv or HuggingFace) | Cheng Chang, Kai Zhang, Boyu Gou, Boyuan Zheng, Yu Gu | WEB-DREAMER uses LLMs as world models for planning in web navigation. The research investigates whether large language models (LLMs) can function as effective world models for web navigation, addressing safety and complexity challenges. The study uses a model-based planning approach where an LLM simulates potential action outcomes in natural language and selects the highest-scoring action. On VisualWebArena, WEB-DREAMER achieved a 23.6% success rate, a 33.3% relative improvement over the reactive baseline. This suggests that incorporating LLM-based world models enables safer and more efficient planning for web agents compared to reactive agents and potentially opens new possibilities for online planning in place of less scalable tree search methods. |
| SAMURAI: Adapting Segment Anything Model for Zero-Shot Visual Tracking with Motion-Aware Memory (Read more on arXiv or HuggingFace) | Jenq-Neng Hwang, Hsiang-Wei Huang, Cheng-Yen Yang, Nitre, wchai | SAMURAI enhances the Segment Anything Model 2 (SAM 2) for zero-shot visual object tracking. The research aims to improve SAM 2’s visual object tracking performance, particularly in crowded scenes and during occlusions, without retraining or fine-tuning. The key methodology involves integrating motion information via a Kalman Filter and a motion-aware memory selection mechanism to improve mask selection and memory management within the SAM 2 architecture. SAMURAI achieves a 7.1% AUC gain on the LaSOText dataset and a 3.5% AO gain on GOT-10k compared to the baseline SAM2.1. This improvement offers AI practitioners a more robust and accurate real-time, zero-shot visual tracking method readily adaptable across various datasets and potentially other tracking frameworks. |
| Stylecodes: Encoding Stylistic Information For Image Generation (Read more on arXiv or HuggingFace) | CiaraRowles | Stylecodes encodes image styles into compact strings for style-conditioned image generation. The research aimed to develop an open-source method for controlling the style of diffusion-based image generation, enabling easy sharing and collaboration. The authors developed Stylecodes, a system combining an attention-based autoencoder and a ControlNet-style UNet decoder to encode image style as a 20-digit base64 code and condition a frozen Stable Diffusion 1.5 model. Experiments showed that Stylecodes effectively enforces the encoded style, allowing generation of images matching the style of a source image given different text prompts; the dataset size was 35,000 image-style-prompt entries. AI practitioners can use Stylecodes for easily shareable and collaborative style control in image generation, though the paper does not specify the quality of style transfer compared to other methods nor specify metrics for evaluation. The training cost for the control model was a limitation, especially for larger diffusion models. |
| When Precision Meets Position: BFloat16 Breaks Down RoPE in Long-Context Training (Read more on arXiv or HuggingFace) | Cunxiao Du, Tongyao Zhu, Chao Du, Qian Liu, haonan3 | This paper investigates the impact of BFloat16 precision on Rotary Positional Embedding (RoPE) in long-context language model training. The authors aim to determine if BFloat16 precision degrades the relative positional encoding properties of RoPE and how this affects long-context performance. They introduce AnchorAttention, a modified attention mechanism that treats the first token as a shared anchor with a fixed position ID, and compare its performance to full attention and intra-document attention. Results on the RULER benchmark show AnchorAttention significantly improves long-context performance, exceeding full attention by 17.47 percentage points on the LLAMA-2-7B model with 128K context window. AI practitioners training LLMs with long contexts should consider using AnchorAttention with BFloat16 to improve performance and reduce training time. |
| ORID: Organ-Regional Information Driven Framework for Radiology Report Generation (Read more on arXiv or HuggingFace) | Dongnan Liu, Ziyong Feng, Xiang An, Tiancheng Gu, Kaichengalex | The paper introduces ORID, a framework for generating radiology reports from X-ray images by leveraging organ-regional information. The objective is to improve the accuracy and believability of automated radiology report generation. ORID uses a LLaVA-Med-RRG model fine-tuned on an organ-level instruction dataset, an organ-based cross-modal fusion module, and an organ importance coefficient analysis module based on a graph neural network. On the IU-Xray dataset, ORID achieved a BLEU@1 score of 0.501, outperforming state-of-the-art methods. This implies that AI practitioners working on medical report generation can leverage organ-specific information and cross-modal fusion techniques to enhance the precision and clinical relevance of generated reports. |
Papers for 2024-11-20
| Title | Authors | Summary |
|---|---|---|
| Continuous Speculative Decoding for Autoregressive Image Generation (Read more on arXiv or HuggingFace) | Fei Li, Qi Yang, Kun Ding, Robert Zhang, MarkWang | This paper introduces Continuous Speculative Decoding (CSpD), a novel method for accelerating autoregressive image generation. The objective is to reduce the computational overhead of continuous-valued autoregressive image generation models while maintaining output quality. CSpD adapts the speculative decoding algorithm from discrete to continuous token space by using denoising trajectory alignment, token pre-filling, and acceptance-rejection sampling to address inconsistencies between draft and target models. Experiments on MAR models for ImageNet 256x256 generation demonstrated a speedup of up to 2.33x. This provides AI practitioners with a technique to significantly accelerate inference for continuous autoregressive image generation models without requiring model retraining or architectural changes, enabling faster generation with comparable quality. |
| Soft Robotic Dynamic In-Hand Pen Spinning (Read more on arXiv or HuggingFace) | Jeffrey Ichnowski, Christopher G. Atkeson, Jean Oh, Uksang Yoo, Yunchao Yao | SWIFT is a system for learning dynamic in-hand manipulation tasks with soft robotic hands, using pen spinning as a case study. The research aimed to enable a soft robotic hand to autonomously learn to grasp and dynamically spin a pen using only real-world data. A self-supervised, trial-and-error approach employing Covariance Matrix Adaptation Evolution Strategy (CMA-ES) optimized grasp location and servo parameters for a three-fingered soft hand. After optimization, SWIFT achieved a 100% success rate across three pens with different weight distributions. This demonstrates the potential for soft robots to perform complex dynamic manipulation tasks without precise object models or simulated training, which can inform the development of more robust and adaptable real-world robotic manipulation systems. |
| RedPajama: an Open Dataset for Training Large Language Models (Read more on arXiv or HuggingFace) | Shane Adams, Yonatan Oren, Quentin Anthony, Daniel Fu, Maurice Weber | RedPajama releases two datasets, V1 and V2, aiming to address transparency and data access challenges in large language model training. The research aimed to create open and versatile datasets for training and analyzing LLMs, specifically focusing on data composition and filtering strategies. RedPajama-V1 reproduced the LLaMA training dataset and RedPajama-V2 created a new web-based dataset with quality signals. Decoder-only transformer models with up to 1.6 billion parameters trained on filtered subsets of RedPajama-V2 showed varying performance on NLP benchmarks, with the Gopher+fuzzy deduplication filter achieving the highest aggregate scores. This allows practitioners to leverage the RedPajama datasets and associated quality signals to curate and experiment with data subsets for training large language models, fostering development of more transparent and potentially higher-performing LLMs. |
| Building Trust: Foundations of Security, Safety and Transparency in AI (Read more on arXiv or HuggingFace) | Huamin Chen, Mark Bestavros, Emily Fox, Garth Mollett, huzaifas-sidhpurwala | The paper explores security and safety implications of publicly available AI models. The objective is to propose strategies for enhancing security, safety, and transparency in the development and operation of public AI models. The paper reviews current security and safety scenarios, highlighting challenges like a lack of standardized processes for lifecycle management and vulnerability remediation. A key finding is generative AI’s steeper adoption curve compared to other technologies, with a projected 124.7 million US users by year four of its release, compared to 116.9 million smartphone users by year four. A primary implication for AI practitioners is the need to adopt a holistic approach to AI risk management, encompassing both security (protecting systems from threats) and safety (preventing unintended harm from model operation), possibly through the creation of frameworks such as a “Hazards Exposure eXchange (HEX)” format and an “Adjunct panel” mirroring similar concepts used in traditional software security. The paper lacks precise details about the proposed HEX format and Adjunct panel, hindering full comprehension of their function. |
| Evaluating Tokenizer Performance of Large Language Models Across Official Indian Languages (Read more on arXiv or HuggingFace) | D. J. Bora, tamang0000 | This paper evaluates the tokenization performance of various large language models (LLMs) across 22 official Indian languages. The research aimed to compare the efficiency of different tokenizers used by 12 LLMs in processing these languages. Normalized Sequence Length (NSL) was used as the primary evaluation metric, calculated as the ratio of tokenized sequence lengths between a given tokenizer and a baseline. The SUTRA tokenizer achieved the lowest average NSL across 14 out of the 22 languages. This finding indicates that the SUTRA tokenizer is particularly efficient for Indian languages and highlights the importance of tokenizer selection for multilingual LLM performance. |
Papers for 2024-11-19
| Title | Authors | Summary |
|---|---|---|
| BlueLM-V-3B: Algorithm and System Co-Design for Multimodal Large Language Models on Mobile Devices (Read more on arXiv or HuggingFace) | wolf1110, AJZhou, liuyangbian, yina0, lucky-lance | BlueLM-V-3B is a 3B parameter multimodal large language model designed for efficient deployment on mobile devices. The research aimed to develop an MLLM that performs well on mobile hardware despite memory and computational limitations. The authors co-designed the model architecture and system, featuring a relaxed aspect ratio matching method for dynamic image resolution, batched image encoding, and token downsampling. On the MediaTek Dimensity 9300 processor, BlueLM-V-3B achieves a generation speed of 24.4 tokens/s with 4-bit LLM weight quantization and a memory usage of 2.2GB. This work enables AI practitioners to deploy performant MLLMs on resource-constrained mobile devices, facilitating broader access to complex multimodal AI capabilities on personal devices. |
| Generative World Explorer (Read more on arXiv or HuggingFace) | Daniel Khashabi, Alan Yuille, Tianmin Shu, jienengchen, TaiMingLu | Genex enables embodied agents to mentally explore 3D environments and update beliefs without physical movement. The research aimed to develop a framework for imaginative exploration in physical worlds to improve decision-making in partially observable environments. A video diffusion model conditioned on egocentric panoramic view and movement direction generates future observations, enabling belief revision. On the Genex-DB dataset, Genex achieved a 69.5 FVD score for video generation quality and below 0.1 latent MSE for long-range imaginative exploration consistency. This work introduces a novel approach for AI practitioners to integrate generative video into partially observable decision processes, offering potential for enhanced planning and multi-agent interaction in embodied AI systems by enabling belief updates based on imagined, rather than physically experienced, observations. |
| AnimateAnything: Consistent and Controllable Animation for Video Generation (Read more on arXiv or HuggingFace) | Rong Zhang, Hong Li, Chi Wang, Guojun Lei, yikaiw | AnimateAnything introduces a two-stage pipeline for generating controllable and consistent videos from images and various control signals. The research aims to address the challenge of integrating diverse control signals like camera trajectories, text prompts, and user motion annotations for precise video manipulation. The key methodology involves converting all visual control signals into a unified optical flow representation, which then guides a video diffusion model. On the OpenVid dataset, AnimateAnything achieved an Aesthetic Quality score of 0.600, outperforming comparison methods. This unified optical flow approach offers AI practitioners a more robust and flexible method for controlling video generation, potentially improving applications like film production and virtual reality. |
| Drowning in Documents: Consequences of Scaling Reranker Inference (Read more on arXiv or HuggingFace) | Michael Carbin, Matei Zaharia, Erik Lindgren, Mathew Jacob, mrdrozdov | This paper investigates the impact of scaling the number of reranked documents on retrieval quality. The research questions how the performance of state-of-the-art rerankers changes when scoring progressively more documents, including the entire dataset. The authors evaluate open and closed-source rerankers on eight academic and enterprise information retrieval benchmarks, measuring Recall@10 and Recall@100 at various reranking depths (K). Results show Recall@10 drops dramatically for many rerankers as K increases beyond 100, often falling below the performance of standalone retrievers; for example, average Recall@10 across enterprise datasets using voyage-rerank-lite-1 decreased from 0.7 to roughly 0.2 as K increased from 100 to 5000. AI practitioners should carefully consider the number of documents (K) provided to rerankers as excessively large K can significantly degrade performance, and listwise reranking with LLMs may offer increased robustness. |
| Comprehensive and Practical Evaluation of Retrieval-Augmented Generation Systems for Medical Question Answering (Read more on arXiv or HuggingFace) | Thien Huu Nguyen, Chien Van Nguyen, Nghia Trung Ngo, Franck-Dernoncourt | This paper introduces MedRGB, a benchmark for evaluating retrieval-augmented generation (RAG) systems in medical question answering. The research aimed to assess the performance of RAG systems in practical medical scenarios, including handling noise, integrating multiple information sources, and resisting factual errors. The methodology involved creating multiple test scenarios (standard RAG, sufficiency, integration, and robustness) and evaluating state-of-the-art and open-source LLMs across these scenarios using four medical QA datasets supplemented with noise and adversarial information. Results revealed that Llama-3-70b achieved the highest noise detection accuracy in the sufficiency test, but all models struggled with factual error detection in the robustness test, with GPT-3.5 having the highest detection rate despite the lowest performance. The key implication for AI practitioners is the need for specialized modules and improved model robustness beyond target accuracy when developing reliable medical RAG systems, as current models have limited ability to handle noise and misinformation within retrieved content. |
| SlimLM: An Efficient Small Language Model for On-Device Document Assistance (Read more on arXiv or HuggingFace) | Viet Dac Lai, Seunghyun Yoon, Phat T. Nguyen, Thang M. Pham, Franck-Dernoncourt | SlimLM models are optimized for on-device document assistance tasks. The research aimed to develop efficient small language models (SLMs) for document processing on mobile devices, addressing the trade-off between model size, performance, and resource constraints. The key methodology involved pre-training SlimLM models (ranging from 125M to 1B parameters) on the SlimPajama-627B dataset and fine-tuning them on DocAssist, a specialized dataset for summarization, question suggestion, and question answering. SlimLM-1B achieved a ROUGE-L score of 0.48, approaching the performance of the larger Qwen2-1.5B-Instruct model. The primary implication for AI practitioners is the ability to deploy performant document processing capabilities directly on mobile devices, potentially reducing server costs and enhancing user privacy. |
| SmoothCache: A Universal Inference Acceleration Technique for Diffusion Transformers (Read more on arXiv or HuggingFace) | Haomiao Jiang, Joshua Geddes, mnandwana, helloterran, josephliu-roblox | SmoothCache is a model-agnostic inference acceleration technique for Diffusion Transformers (DiT). The research aimed to develop a universal caching scheme to speed up DiT inference across various modalities without compromising generation quality. The methodology involved leveraging layer-wise representation errors from a small calibration set to adaptively cache and reuse key features during inference. Experiments showed up to a 71% speedup while maintaining or improving generation quality on models like DiT-XL, Open-Sora, and Stable Audio Open. This technique offers AI practitioners a simple, training-free method to significantly reduce DiT inference latency, potentially enabling real-time applications. |
| Top-$nσ$: Not All Logits Are You Need (Read more on arXiv or HuggingFace) | Liusheng Huang, Hongli Xu, Jianchun Liu, tomorrowdawn | Top-ησ, a novel sampling method for large language models (LLMs), operates directly on pre-softmax logits by leveraging a statistical threshold. The research aims to improve LLM reasoning task performance by developing a sampling method that filters irrelevant tokens more effectively than existing approaches. The key methodology involves separating logits into noisy and informative regions based on their statistical properties, specifically by capturing a region extending n standard deviations (σ) below the maximum logit value. On the GSM8K dataset, top-ησ achieves 74.61% accuracy at a temperature of 3.0, while other comparable sampling methods fail completely. AI practitioners can utilize top-ησ to potentially improve the performance and stability of LLMs in reasoning tasks, especially at higher temperatures, where traditional sampling methods often degrade. The paper mentions an incomplete preprint version, stating some experimental results and appendices will be added later. |
| StableV2V: Stablizing Shape Consistency in Video-to-Video Editing (Read more on arXiv or HuggingFace) | Dong Liu, Yunwei Lan, Kaidong Zhang, Rui Li, Chang Liu | StableV2V is a novel video editing method that aims to maintain shape consistency between user prompts and edited video content. The paper addresses the problem of existing video editing methods often producing results inconsistent with user-desired shapes, especially when prompts introduce significant shape changes. The key methodology involves a three-stage pipeline: a prompted first-frame editor, an iterative shape aligner (ISA) that simulates and refines the depth map of edited frames based on source video motion, and a conditional image-to-video generator that propagates edited content. On the DAVIS-EDIT benchmark, StableV2V achieves a DOVER score of 67.78/70.80 for text-based editing, outperforming comparable methods. This implies that AI practitioners can leverage StableV2V’s shape-consistent editing approach to develop more robust and user-intuitive video editing tools, particularly for tasks involving significant shape transformations. |
| LLäMmlein: Compact and Competitive German-Only Language Models from Scratch (Read more on arXiv or HuggingFace) | Andreas Hotho, Julia Wunderle, Jan Pfister | This paper introduces LLäMmlein, two German-only decoder-only LLMs (120M and 1B parameters) trained from scratch. The objective was to create high-performing, transparent German language models and address the performance gap of existing German LLMs compared to English models. The methodology involved preprocessing a filtered RedPajama V2 dataset, training a custom German tokenizer, and pretraining the models using a TinyLlama framework. LLäMmlein 1B achieved state-of-the-art performance on the EuroParl token classification task within the SuperGLEBer benchmark with a score of 0.732. The open-sourcing of the models, code, and data provides AI practitioners with resources for further German NLP research, including domain adaptation and the creation of a dedicated German instruction dataset. |
| Awaker2.5-VL: Stably Scaling MLLMs with Parameter-Efficient Mixture of Experts (Read more on arXiv or HuggingFace) | Nanyi Fei, Hongpeng Lin, Guoxing Yang, Yanqi Dai, Jinqiang Long | Awaker2.5-VL is a Mixture of Experts (MoE) architecture designed to address the “multi-task conflict” issue in Multimodal Large Language Models (MLLMs). The research aimed to improve MLLM performance on diverse tasks by mitigating interference between different data distributions and representations. The key methodology involves a sparsely activated MoE structure with Low-Rank Adaptation (LoRA) experts and a simplified routing strategy based on instruction embeddings. On the MME-Realworld-CN benchmark, Awaker2.5-VL achieved an overall score of 62.7, surpassing all other compared models. This indicates that incorporating MoE with LoRA and a stable routing strategy can be an effective approach for scaling MLLMs and improving performance across diverse multimodal tasks, offering a potential solution to the multi-task conflict issue. |
| FitDiT: Advancing the Authentic Garment Details for High-fidelity Virtual Try-on (Read more on arXiv or HuggingFace) | Chengming Xu, Qingdong He, Donghao Luo, Xiaobin Hu, Boyuan Jiang | FitDiT is a novel Diffusion Transformer (DiT)-based model for high-fidelity image-based virtual try-on. The research aims to address the challenges of preserving rich texture details and achieving accurate size-aware fitting in virtual try-on applications. The key methodology involves customizing a DiT architecture with structure slimming, garment condition modulation, garment feature injection, a dilated-relaxed mask strategy, and frequency-domain learning. FitDiT achieved a 71.6% reduction in KID error compared to the second-best method on the unpaired VITON-HD dataset, indicating improved garment texture preservation. This improvement in texture fidelity using the DiT architecture provides AI practitioners developing virtual try-on applications with a more effective model for generating realistic and detailed synthesized images of people wearing clothes. |
| Adaptive Decoding via Latent Preference Optimization (Read more on arXiv or HuggingFace) | Jason Weston, Asli Celikyilmaz, Ping Yu, Ilia Kulikov, Shehzaad Dhuliawala | This paper introduces Adaptive Decoding, a method for dynamically adjusting the sampling temperature of large language models (LLMs) during text generation. The research aims to address the suboptimality of fixed temperature decoding for tasks requiring varying levels of creativity and factual accuracy. The core methodology involves adding an ADAPTIVEDECODER module to the LLM, trained using Latent Preference Optimization (LPO) to learn optimal temperature values for different prompts or tokens. Results on the UltraMathStories dataset, a combination of math, creative writing, and general instruction-following tasks, show that Adaptive Decoding outperforms all fixed temperature decoding strategies. This implies that AI practitioners can leverage Adaptive Decoding to improve LLM performance on diverse tasks without manual temperature tuning, automating the balance between creative and factual generation. |
Papers for 2024-11-18
| Title | Authors | Summary |
|---|---|---|
| LLaVA-o1: Let Vision Language Models Reason Step-by-Step (Read more on arXiv or HuggingFace) | LiYuan, sunlichao137, Yibing, Pengjin, Xkev | LLaVA-01 is a vision-language model designed for improved multi-stage, structured reasoning. The research aimed to enhance visual reasoning capabilities in VLMs, particularly for complex tasks requiring systematic analysis. The authors fine-tuned Llama-3.2-11B-Vision-Instruct on a new 100k sample dataset with structured reasoning annotations (LLaVA-01-100k) and introduced stage-level beam search for inference. LLaVA-01 outperformed the base Llama model by 6.9% on average across six multimodal reasoning benchmarks and surpassed some larger, closed-source models. This indicates that training with structured reasoning data and employing stage-level beam search can significantly improve the performance and scalability of VLMs for reasoning-intensive tasks. |
| GaussianAnything: Interactive Point Cloud Latent Diffusion for 3D Generation (Read more on arXiv or HuggingFace) | doubling, hongfz16, ZhaoyangLyu, sczhou, yslan | GaussianAnything introduces a novel framework for 3D generation using a point cloud-structured latent space and cascaded diffusion. The objective is to develop a scalable and interactive 3D generation method addressing challenges in input formats, latent space design, and output representations of existing 3D diffusion models. The method employs a 3D VAE encoding multi-view posed RGB-D-N renderings into a point cloud-structured latent space, followed by cascaded latent diffusion modeling using DiT and flow matching. On the Objaverse dataset, GaussianAnything achieved a Minimum Matching Distance (MMD) of 15.48%, outperforming other image-conditioned methods. The proposed point cloud-structured latent space enables geometry-texture disentanglement and interactive 3D editing, offering AI practitioners a new approach for controllable 3D content creation. |
| The Dawn of GUI Agent: A Preliminary Case Study with Claude 3.5 Computer Use (Read more on arXiv or HuggingFace) | Mingyu Ouyang, AnalMom, QuStar, SiyuanH | This paper presents a preliminary case study of Claude 3.5 Computer Use, a new API-based GUI agent. The research explores Claude 3.5’s capability in real-world desktop environments across web search, workflow, productivity software, and video game domains. The methodology involves curating and testing Claude 3.5 on 20 designed tasks across 12 software or websites, analyzing its planning, action execution, and critic feedback. Claude 3.5 successfully completed 14 out of 20 tasks (70% success rate). The results highlight Claude 3.5’s potential for automating desktop tasks but also reveal limitations related to scrolling-based navigation, text selection accuracy, and contextually aware navigation that AI practitioners should consider when deploying such models in real-world applications. |
| Number it: Temporal Grounding Videos like Flipping Manga (Read more on arXiv or HuggingFace) | Vito328, zhouzhouyi, tms28k, kaleidudu, Liang0223 | NumPro enhances Video Temporal Grounding (VTG) in Video Large Language Models (Vid-LLMs) using frame number overlays. The research aims to improve Vid-LLM performance on VTG tasks, specifically addressing their difficulty in pinpointing event timestamps despite strong visual comprehension. The core methodology involves augmenting video frames with numerical identifiers, enabling Vid-LLMs to associate visual content with temporal information through a “manga-like” numbered panel approach. NumPro-FT, fine-tuned on a NumPro-enhanced dataset, achieves a new state-of-the-art on Charades-STA, surpassing previous SOTA by 11.8% on R@0.3. This provides AI practitioners with a simple, yet effective method to significantly boost VTG performance in Vid-LLMs without requiring complex architectural modifications or extensive retraining. |
Papers for 2024-11-15
| Title | Authors | Summary |
|---|---|---|
| MagicQuill: An Intelligent Interactive Image Editing System (Read more on arXiv or HuggingFace) | Qiuyu Wang, Hao Ouyang, wwen1997, bruceyyu, LiuZichen | MagicQuill is an interactive image editing system built upon diffusion models that allows users to make edits using brushstrokes, which are interpreted by a multimodal large language model (MLLM). The research aimed to develop a robust, open-source, interactive, and precise image editing system that simplifies the process of making detailed image edits. The system combines a dual-branch Editing Processor (inpainting and control branches) with a Painting Assistor (MLLM for prompt prediction) and an Idea Collector (user interface for brushstroke input). Compared to baselines, MagicQuill achieved improved edge alignment and color fidelity with a lower LPIPS score of 0.0667 and a higher PSNR of 27.282 on a constructed test dataset. The paper does not report standard deviations for these or other metrics, making statistical significance unclear. It is unclear how ground truth images were obtained for this evaluation. AI practitioners can leverage this architecture to develop more user-friendly and precise image editing tools, integrating MLLMs to understand user intent from freehand input and enhance generative control in diffusion-based editing. However, the paper does not adequately discuss the generalizability of the Draw&Guess dataset and the robustness of the trained MLLM across diverse user sketch styles and potential ambiguities. |
| LLaMA-Mesh: Unifying 3D Mesh Generation with Language Models (Read more on arXiv or HuggingFace) | Jun Zhu, Hang Su, Yikai Wang, Jonathan Lorraine, Zhengyi Wang | LLaMA-Mesh enables large language models (LLMs) to generate 3D meshes directly from text prompts. The research aimed to unify 3D mesh generation and text generation within a single LLM framework. The key methodology involved representing 3D mesh vertex coordinates and face definitions as plain text within the OBJ file format, enabling direct integration with the LLM without vocabulary expansion. LLaMA-Mesh achieved mesh generation quality comparable to specialized models while retaining language capabilities, scoring 61.74 on MMLU (5-shot) compared to the baseline LLaMA3.1 (8B) score of 66.07. This allows AI practitioners to leverage the text-based knowledge embedded in LLMs for 3D content creation, opening up new possibilities for language-driven 3D modeling. |
| Cut Your Losses in Large-Vocabulary Language Models (Read more on arXiv or HuggingFace) | Philipp Krähenbühl, Vladlen Koltun, Alexander Hertzberg, Brody Huval, erikwijmans | Cut Cross-Entropy (CCE) reduces memory footprint of cross-entropy loss in large language models. The authors aimed to address the disproportionately large memory consumption of cross-entropy loss computation in large language models, especially those with extensive vocabularies. CCE computes cross-entropy without materializing the full logit matrix, instead calculating logits on-the-fly and leveraging sparsity in the softmax gradient. Using CCE with the Gemma 2 (2B) model, memory for loss computation decreased from 24GB to 1MB, and overall classifier head memory from 28GB to 1GB. This allows practitioners training LLMs to significantly increase batch size during training or train larger models on existing hardware due to reduced memory requirements. |
| ClinicalBench: Can LLMs Beat Traditional ML Models in Clinical Prediction? (Read more on arXiv or HuggingFace) | Zhongwei Wan, Che Liu, Shan Chen, Jian Yu, canyuchen | ClinicalBench benchmarks LLMs and traditional ML models on clinical prediction tasks. The research investigates whether LLMs can outperform traditional ML models in clinical prediction. The benchmark uses two clinical databases (MIMIC-III and MIMIC-IV) and evaluates performance on three common clinical prediction tasks (length-of-stay, mortality, and readmission) with various LLMs (general-purpose and medical) and traditional ML models, using prompting and fine-tuning strategies. Across all tasks and datasets, traditional ML models generally outperformed LLMs, with XGBoost achieving a Macro F1-score of 67.94% on length-of-stay prediction in MIMIC-III, substantially higher than LLMs. AI practitioners should exercise caution when applying LLMs to clinical prediction tasks, as they currently do not demonstrate superiority over established ML methods, despite strong performance on medical question answering benchmarks. |
| Hermes: A Large Language Model Framework on the Journey to Autonomous Networks (Read more on arXiv or HuggingFace) | Merouane Debbah, Antonio De Domenico, Ali Maatouk, Fadhel Ayed, nicopi | Hermes is a chain-of-agent LLM framework for modeling and automating cellular network operations using “blueprints” for constructing Network Digital Twins (NDTs). The research investigates whether LLMs can effectively model network behavior and advance network autonomy. The key methodology involves a three-phase process where a “Designer” LLM agent creates a blueprint for a NDT, a “Coder” agent translates it into Python code, and a feedback loop refines the blueprint based on numerical evaluation. When using GPT-40 as the LLM, Hermes achieved a success rate of 82.5% in modeling power control and energy saving tasks, compared to 25% for chain-of-thought and 55% for Hermes-coder (without the Designer). The success rate varies based on the complexity of the modeling task and with the specific LLMs being employed and increases substantially with the inclusion of domain specific models in the model repository. This indicates that integrating structured blueprints with domain expertise enhances LLM reliability in network modeling tasks and paves the way for more robust autonomous network operations using LLMs. |
| Sharingan: Extract User Action Sequence from Desktop Recordings (Read more on arXiv or HuggingFace) | Kehong Yuan, Jue Zhang, Xiaoting Qin, Yi Ren, Yanting Chen | Sharingan introduces two VLM-based methods to extract user action sequences from desktop recordings: Direct Frame-Based (DF) and Differential Frame-Based (DiffF). The research aims to determine the efficacy of VLMs in extracting user actions from desktop video recordings. Both methods use VLMs (GPT and Gemini series) to process video frames, with DiffF incorporating explicit frame difference detection. On the ACTONE dataset, the DF approach with GPT-40 achieved 70-80% accuracy in identifying operation types, with extracted sequences being replayable via RPA. This work enables AI practitioners to explore desktop video as a data source for RPA, automated tutorial generation, and user behavior analysis. |
Papers for 2024-11-14
| Title | Authors | Summary |
|---|---|---|
| Large Language Models Can Self-Improve in Long-context Reasoning (Read more on arXiv or HuggingFace) | Mo Yu, Lemao Liu, Zesen Cheng, Cheng Yang, Siheng99 | SEALONG, a novel self-improvement method for LLMs, enhances long-context reasoning. The research investigates LLMs’ capacity for self-improvement in reasoning over extended text. The methodology involves sampling multiple output reasoning trajectories, scoring them using Minimum Bayes Risk (MBR), and fine-tuning via supervised learning or preference optimization. Llama-3.1-8B-Instruct improved by 4.2 points using SEALONG, outperforming prior methods relying on expert-generated data. This self-improvement technique allows LLMs to enhance their long-context reasoning abilities without external annotations, offering a scalable path towards more advanced reasoning capabilities for AI practitioners. |
| EgoVid-5M: A Large-Scale Video-Action Dataset for Egocentric Video Generation (Read more on arXiv or HuggingFace) | Guosheng Zhao, Jiayu Wang, Feng Liu, Kang Zhao, Xiaofeng Wang | EgoVid-5M is a 5-million-clip dataset designed for training egocentric video generation models. The research aimed to create a high-quality dataset to address the challenges of generating egocentric videos due to dynamic viewpoints, action diversity, and scene complexity. The researchers annotated EgoVid-5M with fine-grained kinematic control data using Visual Inertial Odometry and high-level textual descriptions via a multimodal large language model, and then implemented a data cleaning pipeline addressing text-video and frame-frame consistency, motion smoothness, and video clarity. Training a DynamiCrafter model on EgoVid-1M-3 (a subset of EgoVid-5M) resulted in an improved CD-FVD score compared to models trained on alternative cleaning strategies. AI practitioners can now leverage EgoVid-5M and its associated metadata to train and evaluate egocentric video generation models, potentially advancing applications in virtual/augmented reality and gaming. |
| Direct Preference Optimization Using Sparse Feature-Level Constraints (Read more on arXiv or HuggingFace) | Hanqi Yan, Minjun Zhu, Hongbo Zhang, Chak Tou Leong, Qingyu Yin | FPO (Feature-level constrained Preference Optimization) improves large language model (LLM) alignment by using sparse feature-level constraints. The research aimed to develop a more efficient and controllable method for aligning LLMs to human preferences than existing methods like RLHF and DPO. FPO leverages pre-trained Sparse Autoencoders (SAEs) and introduces feature-level constraints within a Direct Preference Optimization (DPO) framework, minimizing mean squared error (MSE) between sparse activations. On the AlpacaEval-2 benchmark, FPO achieved a win rate improvement of up to 5.08% compared to baseline methods. This provides AI practitioners with a more efficient and stable method for aligning LLMs, potentially reducing computational costs and improving generation quality. |
| CamemBERT 2.0: A Smarter French Language Model Aged to Perfection (Read more on arXiv or HuggingFace) | Benoît Sagot, Éric de la Clergerie, Rian Touchent, Francis Kulumba, Wissam Antoun | This paper introduces CamemBERT 2.0, two updated French language models: CamemBERTav2 (DeBERTaV3 architecture, Replaced Token Detection objective) and CamemBERTv2 (RoBERTa architecture, Masked Language Modeling objective). The objective is to address temporal concept drift and improve performance on various natural language processing (NLP) tasks. Both models were trained on a larger, more recent 275B token dataset with an updated tokenizer designed to better capture French linguistic nuances. CamemBERTav2 achieved an F1 score of 93.4% on named entity recognition (NER) using the FTB dataset, significantly outperforming the original CamemBERT (89.97%). AI practitioners can leverage these updated, open-source models for improved performance in various French NLP applications, including specialized domains like biomedicine, highlighting the importance of continuous model updates and data freshness in mitigating concept drift. |
| Can sparse autoencoders be used to decompose and interpret steering vectors? (Read more on arXiv or HuggingFace) | Adam Mahdi, Yushi Yang, Harry Mayne | This paper investigates why directly applying sparse autoencoders (SAEs) to steering vectors yields misleading decompositions. The research aims to understand why SAEs provide inaccurate interpretations of steering vectors, which are used to control the behavior of large language models. The methodology involves decomposing steering vectors for “corrigibility” in a language model using SAEs and comparing them to decompositions of zero vectors and model activations. The primary results show that the L2-norm of the corrigibility steering vector is substantially smaller than that of typical model activations, and that 51.2% of relevant features show stronger activations on negative example prompts. This implies that SAE interpretations of steering vectors are often dominated by the encoder bias and fail to capture meaningful negative projections in feature directions, hindering their direct use for interpreting how these vectors influence language model behavior. |
Papers for 2024-11-13
| Title | Authors | Summary |
|---|---|---|
| SAMPart3D: Segment Any Part in 3D Objects (Read more on arXiv or HuggingFace) | Xiaoyang Wu, Liangjun Lu, Yuan-Chen Guo, Yukun Huang, Yunhan Yang | SAMPart3D is a zero-shot 3D part segmentation framework. The objective is to segment 3D objects into semantic parts at multiple granularities without predefined part labels or text prompts. The methodology involves a two-stage 2D-to-3D distillation process from DINOv2 and SAM, followed by semantic querying with Multimodal Large Language Models (MLLMs). On the PartObjaverse-Tiny dataset, SAMPart3D achieved 53.7% mean Intersection over Union (mIoU) for class-agnostic part segmentation. This provides AI practitioners with a scalable and flexible method for zero-shot 3D part segmentation, facilitating applications like part-level editing and interactive segmentation. |
| JanusFlow: Harmonizing Autoregression and Rectified Flow for Unified Multimodal Understanding and Generation (Read more on arXiv or HuggingFace) | Chengyue Wu, Wen Liu, Xiaokang Chen, Xingchao Liu, Yiyang Ma | JanusFlow is a unified multimodal model for image understanding and generation. The research aimed to create a single model capable of both image understanding and generation using rectified flow within an autoregressive LLM framework. The key methodology involved integrating rectified flow with an LLM, decoupling vision encoders for understanding and generation, and aligning their representations during training. On the MJHQ FID-30k benchmark, JanusFlow achieved a score of 9.51, outperforming other 1.3B parameter models. This provides AI practitioners with a more efficient and versatile vision-language model architecture that requires fewer parameters than alternative approaches while achieving state-of-the-art or comparable performance. |
| Stronger Models are NOT Stronger Teachers for Instruction Tuning (Read more on arXiv or HuggingFace) | Radha Poovendran, Luyao Niu, Fengqing Jiang, Zhangchen Xu, yuchenlin | This paper investigates the impact of response generator model selection on instruction-tuned LLM performance. The research questions which models are the most effective response generators for instruction tuning and how to determine effective response generators without instruction tuning. The authors fine-tuned five base LLMs on instruction datasets generated by 20 different response generators and evaluated them on AlpacaEval 2 and Arena-Hard benchmarks. Gemma-2-9b-it and Qwen2.5-72B-Instruct emerged as the two best response generators, outperforming larger models and even GPT-4 in some cases (e.g., average performance of 13.92% and 16.15% on Llama-3.1-Minitron-4B, respectively, compared to 5.72% for GPT-4). The proposed Compatibility-Adjusted Reward (CAR) metric, accounting for both response quality and compatibility with the base model, outperformed baseline metrics in predicting response generator effectiveness. AI practitioners should prioritize response generators with high compatibility with the base LLM, as measured by CAR, rather than solely relying on benchmark performance, to maximize the effectiveness of instruction tuning. |
| Wavelet Latent Diffusion (Wala): Billion-Parameter 3D Generative Model with Compact Wavelet Encodings (Read more on arXiv or HuggingFace) | Derek Cheung, Arianna Rampini, Pradyumna Reddy, Aliasghar Khani, adityasanghi | WaLa introduces a novel framework for generating high-quality 3D shapes from various input modalities. The objective is to address the computational challenges of large-scale 3D generative models while preserving fine details and complex geometries. The key methodology involves encoding 3D shapes into compact wavelet-based latent representations using a VQ-VAE, achieving a 2,427x compression ratio, and training a billion-parameter diffusion model on this latent space. On the Google Scanned Objects (GSO) dataset, WaLa achieved an Intersection over Union (IoU) of 0.978 for point cloud to mesh reconstructions. WaLa offers AI practitioners a highly efficient and versatile method for generating high-resolution 3D shapes from various modalities, including text, sketches, and images, within seconds, which was previously computationally infeasible. |
Papers for 2024-11-12
| Title | Authors | Summary |
|---|---|---|
| Add-it: Training-Free Object Insertion in Images With Pretrained Diffusion Models (Read more on arXiv or HuggingFace) | Gal Chechik, Lior Wolf, Dvir Samuel Yuval Atzmon, Rinon Gal, Yoad Tewel | Add-it is a training-free method for inserting objects into images based on text prompts. The objective is to develop a method for adding objects to images based on textual instructions that preserves image context and structure while placing objects naturally within the scene. The method leverages pretrained text-to-image diffusion models, incorporating a weighted extended self-attention mechanism that balances information from a source image, a target image, and a text prompt, alongside a novel Subject-Guided Latent Blending mechanism and a structure transfer step. On the Additing Affordance Benchmark, which evaluates the plausibility of object placement, Add-it achieves an affordance score of 0.828, significantly outperforming other methods. Human evaluations on the Emu Edit Benchmark favored Add-it outputs in 80% of cases. AI practitioners can leverage Add-it to enhance existing text-to-image models for object insertion tasks without requiring additional training or fine-tuning of these large models, thereby enabling more realistic image editing applications. |
| OmniEdit: Building Image Editing Generalist Models Through Specialist Supervision (Read more on arXiv or HuggingFace) | Xinrun Du, Weiming Ren, Zheyang Xiong, Cong Wei, wenhu | OmniEdit is an instruction-based image editing model trained using specialist supervision. The research aims to address limitations in existing instruction-guided image editing models, such as biased editing capabilities and poor data quality. The key methodology involves training a generalist editing model supervised by seven specialist models, utilizing importance sampling based on large multimodal model (LMM) scoring, and introducing a novel diffusion-transformer architecture called EditNet. OMNI-EDIT achieved a 0.20 higher accuracy compared to the strongest baseline CosXL-Edit on the proposed OMNI-EDIT-BENCH dataset. This implies that AI practitioners can leverage specialist models and LMM-based scoring during training to develop more generalized and robust image editing models capable of performing diverse editing tasks on images with varying resolutions and aspect ratios. |
| Chinese SimpleQA: A Chinese Factuality Evaluation for Large Language Models (Read more on arXiv or HuggingFace) | Hui Huang, Yingshui Tan, Jiaheng Liu, Shilong Li, Yancheng He | Chinese SimpleQA is a benchmark to evaluate the factuality of large language models (LLMs) in answering short, fact-seeking questions in Chinese. The research aimed to create a comprehensive Chinese benchmark for evaluating LLM factuality. The methodology involved automated question-answer pair generation from knowledge sources, followed by human verification and filtering for difficulty and adherence to static answer criteria. Only two closed-source LLMs (o1-preview and Doubao-pro-32k) surpassed the 60% accuracy threshold. The benchmark highlights the need for continued improvement in Chinese LLM factuality and provides a resource for evaluating and enhancing performance in Chinese knowledge domains. |
| Edify Image: High-Quality Image Generation with Pixel Space Laplacian Diffusion Models (Read more on arXiv or HuggingFace) | Tiffany Cai, Yogesh Balaji, Maciej Bala, Yuval Atzmon, NVIDIA | Edify Image is a family of diffusion models for generating high-quality, photorealistic images. The research aimed to develop diffusion models capable of generating high-resolution images with precise controllability. The key innovation is the Laplacian Diffusion Model, a multi-scale approach where image frequency bands are attenuated at varying rates during a cascaded diffusion process. The two-stage text-to-image model can generate images at 1K resolution, and an upsampler further refines these to 4K. AI practitioners can leverage these models for various applications like text-to-image synthesis, upsampling, and image editing with ControlNets, leveraging the novel Laplacian diffusion approach for enhanced control over image generation at multiple scales. |
| IOPO: Empowering LLMs with Complex Instruction Following via Input-Output Preference Optimization (Read more on arXiv or HuggingFace) | Yongbin Li, Fei Huang, Cheng Fu, Haiyang Yu, Xinghua Zhang | IOPO enhances large language models’ (LLMs) ability to follow complex instructions. The research aims to improve LLMs’ handling of intricate, multi-constraint instructions. The authors introduce a new benchmark, TRACE, and an alignment method called Input-Output Preference Optimization (IOPO), which considers both input and output preferences. IOPO demonstrated an 8.15% improvement on in-domain data and a 6.29% improvement on out-of-domain data compared to Supervised Fine-Tuning (SFT) regarding complex instruction following. This finding provides AI practitioners with a novel alignment technique to optimize LLMs for applications requiring nuanced instruction understanding and adherence. |
| M-Longdoc: A Benchmark For Multimodal Super-Long Document Understanding And A Retrieval-Aware Tuning Framework (Read more on arXiv or HuggingFace) | Maojia Song, Chaoqun Liu, Hou Pong Chan, Liying Cheng, Yew Ken Chia | M-LongDoc introduces a benchmark and retrieval-aware tuning framework for multimodal long document understanding. The research aims to improve large multimodal models’ ability to understand and answer questions on lengthy, complex multimodal documents. A retrieval-aware tuning approach is proposed, incorporating distracting content from different modalities and pages during training. Experiments show a 4.6% relative improvement in answer correctness using this tuning method compared to baseline open-source models. This improved performance enables more efficient and accurate processing of lengthy multimodal documents, benefiting AI practitioners developing document understanding applications. |
| Watermark Anything with Localized Messages (Read more on arXiv or HuggingFace) | Matthijs Douze, Teddy Furon, Alain Durmus, Pierre Fernandez, Tom Sander | The Watermark Anything Model (WAM) performs localized image watermarking, enabling segmentation of watermarked areas and extraction of multiple messages. The research aimed to develop a watermarking method robust to image manipulations like splicing and inpainting, even with small watermarked areas. A two-stage training process was employed: initial training for robustness at low resolution followed by fine-tuning for imperceptibility and multiple watermark handling using a JND map. WAM achieved over 85% mIoU for detection of watermarked areas when hiding five 32-bit messages in 10% areas of an image, even after horizontal flips and contrast adjustments. AI practitioners can utilize WAM for robust localization of watermarked areas and extraction of distinct messages from within a single image, enabling novel applications like verification of content origin and detection of AI-generated objects within images. |
| Counterfactual Generation from Language Models (Read more on arXiv or HuggingFace) | Ryan Cotterell, Anej Svete, vesteinn, Shauli | This paper introduces a framework for generating true counterfactual strings from language models. The research aimed to understand and mitigate the unintended side effects of common language model intervention techniques. The key methodology involved formulating language models as Generalized Structural-equation Models (GSEMs) using the Gumbel-max trick, enabling counterfactual reasoning. Results showed that even “minimal” interventions like MEMIT and linear steering induce significant semantic shifts in generated text, with instruction tuning interventions showing the most unintended side-effects (sharing only 24% of tokens with original strings on average). This implies that AI practitioners should carefully evaluate the potential for unintended consequences, even with seemingly targeted interventions, and consider the proposed GSEM framework for analyzing and mitigating these effects. |
| Game-theoretic LLM: Agent Workflow for Negotiation Games (Read more on arXiv or HuggingFace) | Julie Chen, Alfonso Amayuelas, Lingyao Li, Ollie Liu, Wenyue Hua | This paper investigates the rationality of Large Language Models (LLMs) in strategic decision-making within game-theoretic scenarios. The research objective is to evaluate LLM rationality in both complete and incomplete information games and explore methods to enhance it. The authors design and implement game-theory-inspired workflows, including dominant strategy search and backward induction, to guide LLM reasoning. In “Deal or No Deal”, Claude-3.5 Sonnet with workflow achieved a 95.45% agreement rate. A key implication for AI practitioners is that incorporating structured, game-theoretic workflows into LLM agents can significantly improve their negotiation performance and strategic decision-making in complex, multi-agent environments, but the choice of whether to use a workflow is itself a strategic decision. |
| Golden Touchstone: A Comprehensive Bilingual Benchmark for Evaluating Financial Large Language Models (Read more on arXiv or HuggingFace) | Yiyan Qi, Zhouchi Lin, Huanyi Su, Junxi Liu, Xiaojun Wu | Golden Touchstone is a bilingual benchmark for evaluating financial large language models (LLMs). The research aimed to create a comprehensive, bilingual benchmark to evaluate FinLLMs on a wider range of tasks and in both English and Chinese. The benchmark includes 22 datasets across eight core financial NLP tasks, and performance was assessed for several LLMs including GPT-40, Llama-3, and a newly developed model, Touchstone-GPT, trained using continuous pre-training and financial instruction tuning. Llama-3 achieved the highest Weighted-F1 score (0.5116) on the English stock movement prediction task, though all models underperformed on this challenging task. This suggests that current LLMs struggle with complex financial prediction tasks and that benchmarks like Golden Touchstone are crucial for directing further research and model development in financial AI. |
| Ablation is Not Enough to Emulate DPO: How Neuron Dynamics Drive Toxicity Reduction (Read more on arXiv or HuggingFace) | Adam Mahdi, Harry Mayne, Filip Sondej, Yushi Yang | This paper investigates the mechanisms by which Direct Preference Optimization (DPO) reduces toxicity in language models. The research aims to determine how DPO’s internal mechanisms lead to toxicity reduction in language models, challenging the existing explanation that it primarily dampens the most toxic MLP neurons. The study uses ablation of toxic neurons, activation patching, and projection of neuron activation changes onto a toxicity probe in GPT-2 medium. Results show that dampening toxic neurons accounts for only 31.8% of the total toxicity reduction, with a significant portion coming from promoting anti-toxicity via other neuron groups and noisy adjustments across many neurons. This suggests for AI practitioners that mitigating toxicity in LLMs requires a more nuanced approach than simply targeting the most toxic neurons, and that a more holistic understanding of neuron dynamics is essential for effective toxicity reduction. |
| KMM: Key Frame Mask Mamba for Extended Motion Generation (Read more on arXiv or HuggingFace) | Feng Chen, Qi Chen, Akide Liu, Zeyu Zhang, Ha0Tang | This paper introduces Key Frame Mask Mamba (KMM) for generating extended human motion sequences from text. The research aims to address limitations of existing methods, specifically memory decay and weak text-motion alignment, in generating long and complex motions from text prompts. The core methodology involves a novel key frame masking strategy based on local density and a contrastive learning approach for text-motion alignment within the Mamba architecture. On the BABEL dataset, KMM achieved a 57% improvement in Frechet Inception Distance (FID) compared to previous state-of-the-art methods. This implies that AI practitioners can leverage KMM to generate higher-quality, more text-aligned extended motion sequences, potentially benefiting applications in animation, gaming, and virtual reality. |
Papers for 2024-11-11
| Title | Authors | Summary |
|---|---|---|
| Balancing Pipeline Parallelism with Vocabulary Parallelism (Read more on arXiv or HuggingFace) | Min Lin, Penghui Qi, Man Tsung Yeung, ufotalent | This paper proposes Vocabulary Parallelism to address computational and memory imbalances caused by vocabulary layers in pipeline parallel training of large language models. The research aims to mitigate pipeline bubbles and memory bottlenecks arising from uneven workload distribution across pipeline stages due to vocabulary layers. The core methodology involves partitioning vocabulary layers across all pipeline devices, grouping computations into pipeline passes, and minimizing communication barriers within these layers. Results show up to a 51% improvement in throughput compared to naive approaches, and near-perfect memory balance when combined with the V-Half scheduling strategy. This allows AI practitioners training large language models with pipeline parallelism to achieve significantly improved throughput and reduced memory consumption, particularly in large vocabulary scenarios, enabling training of larger models or using larger batch sizes. |
| StdGEN: Semantic-Decomposed 3D Character Generation from Single Images (Read more on arXiv or HuggingFace) | Kaiwen Xiao, Zhongkai Wu, Wang Zhao, Yanning Zhou, Yuze He | StdGEN is a novel pipeline for generating semantically decomposed 3D characters from single images. The research aimed to create a method for generating high-quality, decomposable 3D characters from single images, addressing limitations of existing methods in decomposability, quality, and optimization time. The pipeline utilizes a Semantic-aware Large Reconstruction Model (S-LRM), a multi-view diffusion model, and an iterative multi-layer surface refinement module. On the Anime3D++ dataset, StdGEN achieved a CLIP similarity score of 0.935 for 3D character generation from arbitrary pose images. The decomposable nature of the generated 3D characters and the speed of generation (within minutes) offer AI practitioners a valuable tool for efficient character creation, editing, and animation in various 3D applications. |
| DELIFT: Data Efficient Language model Instruction Fine Tuning (Read more on arXiv or HuggingFace) | Marina Danilevksy, Lucian Popa, Krishna Killamsetty, ishikaa | DELIFT is a novel algorithm for optimizing data selection across different fine-tuning stages of Large Language Models (LLMs). The research aimed to create a unified framework for efficient data selection across all fine-tuning stages of LLMs, optimizing performance and data efficiency. DELIFT uses a pairwise utility metric combined with submodular optimization techniques to select data subsets. In experiments, DELIFT reduced fine-tuning data size by up to 70% without compromising performance, sometimes even exceeding full-dataset performance. This allows AI practitioners to significantly reduce computational costs and training time for LLMs without sacrificing performance, potentially increasing accessibility of LLM fine-tuning in resource-constrained environments. |
| Parameter-Efficient Fine-Tuning of Large Language Models for Unit Test Generation: An Empirical Study (Read more on arXiv or HuggingFace) | Jingyue Li, andstor | This paper investigates the effectiveness of parameter-efficient fine-tuning (PEFT) methods for training large language models (LLMs) to generate unit tests. The primary research question is how well PEFT methods perform on unit test generation compared to full fine-tuning and in relation to resource utilization. The study evaluates LoRA, (IA)³, and prompt tuning against full fine-tuning across ten LLMs of varying sizes using the METHODS2TEST and HumanEval-X datasets, measuring syntactic correctness, CodeBLEU similarity, and code coverage. LoRA achieved the highest CodeBLEU scores in five out of ten models and was the only method to improve CodeBLEU for CodeLlama-7B. AI practitioners can leverage PEFT, especially LoRA, to efficiently fine-tune LLMs for unit test generation, potentially matching or exceeding the performance of full fine-tuning while significantly reducing computational costs. |
| LLM2CLIP: Powerful Language Model Unlock Richer Visual Representation (Read more on arXiv or HuggingFace) | Yuqing Yang, Xufang Luo, Aoqi Wu, Weiquan Huang, Yif29 | LLM2CLIP enhances visual representations by integrating large language models (LLMs) into CLIP training. The research aimed to determine if LLMs could improve multimodal representation learning, addressing CLIP’s limitations with complex and long text. The key methodology involved caption contrastive fine-tuning of the LLM and a novel training process where the fine-tuned LLM guides CLIP’s visual encoder. LLM2CLIP boosted the performance of the SOTA EVA02 model by 16.5% on long and short-text retrieval tasks. This implies that AI practitioners can leverage LLM2CLIP to significantly improve the performance of existing and future multimodal models relying on CLIP, especially in tasks involving complex or long textual descriptions. |
| Improving the detection of technical debt in Java source code with an enriched dataset (Read more on arXiv or HuggingFace) | Rick Kazman, Davide Di Ruscio, Phuong T. Nguyen, Anh M. T. Bui, Nam Le Hai | This paper presents a novel dataset and methods for improving the detection of technical debt (TD) in Java source code. The research aimed to determine if manually classified comments and source code context enhance the detection of self-admitted technical debt (SATD). The authors curated a dataset, TESORO, by extracting SATD comments and corresponding source code from Java projects, then manually classifying TD types. Experiments using pre-trained language models (PLMs) like CodeBERT and RoBERTa showed that adding TESORO to training data improved SATD detection F1-scores by up to 14.59%. This suggests AI practitioners can significantly improve the performance of their TD detection models by incorporating source code context and leveraging datasets like TESORO for training. |
Papers for 2024-11-08
| Title | Authors | Summary |
|---|---|---|
| OpenCoder: The Open Cookbook for Top-Tier Code Large Language Models (Read more on arXiv or HuggingFace) | Jiaran Hao, Jason Klein Liu, Tianhao Cheng, Siming Huang, Zenithwang | OpenCoder is a top-tier, open-source code large language model (LLM) with reproducible datasets and training pipelines. The research aimed to create a high-performing, fully transparent code LLM and investigate data curation strategies for such models. Key methodologies included code-optimized data cleaning and deduplication, recall of code-related text corpora, and use of high-quality synthetic data in annealing and supervised fine-tuning stages. OpenCoder-8B achieved a zero-shot pass@1 rate of 68.9% on HumanEval. The transparent, reproducible nature of OpenCoder provides a powerful model and robust foundation for researchers and practitioners to accelerate and reproduce advancements in code AI. |
| ReCapture: Generative Video Camera Controls for User-Provided Videos using Masked Video Fine-Tuning (Read more on arXiv or HuggingFace) | David E. Jacobs, Nikhil Karnad, Shiran Zada, Roni Paiss, David Junhao Zhang | ReCapture enables generating novel camera trajectories for existing user-provided videos while preserving scene content and dynamics. The research aims to develop a method for generating videos with new camera trajectories from single user-provided videos without needing paired training data. The method uses masked video fine-tuning with spatial and temporal Low-Rank Adaptations (LoRAs) applied to a pre-trained video diffusion model, conditioned on an intermediate “anchor video” generated via either point cloud rendering or multi-view diffusion. On the Kubric-4D dataset, ReCapture achieves a PSNR of 20.92, outperforming existing 4D reconstruction and generative methods. This provides AI practitioners with a technique to manipulate camera motion in existing videos without requiring extensive 4D datasets or explicit 3D scene representations, facilitating applications in video editing and content creation. |
| BitNet a4.8: 4-bit Activations for 1-bit LLMs (Read more on arXiv or HuggingFace) | Furu Wei, Shuming Ma, Hongyu Wang | BitNet a4.8 introduces a hybrid quantization and sparsification strategy enabling 4-bit activations for 1-bit Large Language Models (LLMs). The research aimed to reduce the inference cost of 1-bit LLMs while maintaining performance comparable to higher-precision models like BitNet b1.58. The method involves using 4-bit activations for inputs to attention and feed-forward network layers, sparsifying intermediate states with 8-bit quantization, and a two-stage training recipe from 8-bit to 4-bit activations. For a 7B parameter model, BitNet a4.8 achieved similar performance to BitNet b1.58 on downstream tasks, while having only 55% activated parameters (3.4B). This allows AI practitioners to deploy and infer large language models more efficiently with reduced computational and memory requirements by leveraging 4-bit activations and sparsity. |
| DimensionX: Create Any 3D and 4D Scenes from a Single Image with Controllable Video Diffusion (Read more on arXiv or HuggingFace) | Zilong Chen, Fangfu Liu, Shuo Chen, Wenqiang Sun, yikaiw | DimensionX generates 3D and 4D scenes from a single image using controllable video diffusion. The research aims to create photorealistic 3D and 4D scenes from single images using controllable video diffusion, addressing the limited spatial and temporal control in existing video diffusion models. The key methodology is ST-Director, which decouples spatial and temporal factors in video diffusion by learning dimension-aware LoRAs from specifically curated datasets, enabling control over individual dimensions and their combination. On the Tank and Temples dataset for sparse-view 3D generation, DimensionX achieves 20.42 PSNR, 0.668 SSIM, and 0.185 LPIPS, outperforming baseline methods. This provides AI practitioners with a more controllable and effective approach for generating 3D and 4D content from limited input data, enabling applications in various fields like virtual reality and content creation. |
| Mixture-of-Transformers: A Sparse and Scalable Architecture for Multi-Modal Foundation Models (Read more on arXiv or HuggingFace) | Ning Dong, Srinivasan Iyer, Liang Luo, Lili Yu, WxWx | Mixture-of-Transformers (MoT) accelerates multi-modal foundation model pretraining by decoupling non-embedding parameters by modality. The paper investigates whether modality-specific parameterization in transformers can improve multi-modal pretraining efficiency without compromising performance. MoT isolates parameters like feed-forward networks, attention matrices, and layer normalization by modality while maintaining global self-attention across all input tokens. This creates separate transformer towers for each modality. In the Chameleon 7B text and image generation setting, MoT matched dense model performance using only 55.8% of the FLOPs. Across various multi-modal datasets and training setups (Chameleon, Chameleon+Speech, Transfusion), MoT consistently reduced training FLOPs and wall-clock time, particularly for image generation. Further analysis comparing MoT against Mixture-of-Experts and analyzing modality separation effects via Leave-One-Out analysis is provided, but the methodology used in these analyses is not fully clear. AI practitioners can use MoT to significantly reduce computational costs and training time for large multi-modal foundation models without significant performance degradation, especially in image-related tasks. |
| Thanos: Enhancing Conversational Agents with Skill-of-Mind-Infused Large Language Model (Read more on arXiv or HuggingFace) | Ho-Jin Choi, Kyeongjin Oh, Junyoung Youn, Dokyong Lee, Young-Jun Lee | THANOS enhances LLM-based conversational agents by infusing them with a “skill-of-mind” process. The research aims to improve the quality and social appropriateness of LLM responses in interactive dialogue settings by incorporating conversational skills. A new skill-of-mind-annotated dataset, MULTIFACETED SKILL-OF-MIND, containing roughly 100K conversations, was created and used to fine-tune LLaMA models of varying sizes (1B, 3B, and 8B parameters). THANOS 8B achieved an average of 29.7% accuracy on skill classification across multiple datasets, a substantial improvement over baseline LLM-based agents. AI practitioners can use THANOS and the MULTIFACETED SKILL-OF-MIND dataset to develop more socially adept and engaging conversational agents by grounding response generation in relevant conversational skills. |
| TIP-I2V: A Million-Scale Real Text and Image Prompt Dataset for Image-to-Video Generation (Read more on arXiv or HuggingFace) | Yi Yang, Wenhao Wang | TIP-I2V is a novel million-scale dataset of user-provided text and image prompts for image-to-video generation. The research aimed to create a dedicated dataset for studying user prompts in image-to-video generation, which was lacking previously. The dataset was curated by collecting text and image prompts from Pika Discord channels, along with generated videos from five state-of-the-art image-to-video models. The authors found significant semantic differences between TIP-I2V prompts and those in existing text-to-video (VidProM) and text-to-image (DiffusionDB) datasets, with TIP-I2V focusing on animating existing image content. In benchmark evaluations using TIP-I2V, the early commercial model Pika outperformed the latest open-source model, CogVideoX-5B, in 8 out of 10 evaluation dimensions. This finding indicates that AI practitioners should consider real-world user prompt data when developing and evaluating image-to-video models. |
| DynaMem: Online Dynamic Spatio-Semantic Memory for Open World Mobile Manipulation (Read more on arXiv or HuggingFace) | Chris Paxton, Soumith Chintala, Mohit Warke, Zhanqiu Guo, Peiqi Liu | DynaMem is a novel spatio-semantic memory architecture for open-vocabulary mobile manipulation in dynamic environments. The research aimed to address the limitation of current open-vocabulary mobile manipulation systems that assume static environments, hindering real-world applicability. The core methodology involves a dynamic 3D voxel map that adds and removes points based on observed changes, combined with either vision-language model features or multimodal LLM queries for object localization. In real-world robot experiments, DynaMem achieved a 70% pick-and-drop success rate on non-stationary objects, a 2x improvement over static baselines. This improvement demonstrates the value of dynamic memory for real-world robotic manipulation systems and offers AI practitioners a more robust approach for object interaction in changeable environments. |
| Needle Threading: Can LLMs Follow Threads through Near-Million-Scale Haystacks? (Read more on arXiv or HuggingFace) | Samuel Albanie, Kai Han, Jonathan Roberts | This paper evaluates the long-context retrieval capabilities of 17 Large Language Models (LLMs). The research investigates how effectively LLMs utilize their context windows, particularly in following “threads” of linked information. The study uses synthetically generated datasets of key-value pairs (UUIDs) with varying context lengths up to 900k tokens and tests performance on single/multiple needle retrieval, conditional retrieval, and threading/multi-threading tasks. Results show performance degradation with increasing context lengths and thread lengths in most models; for example, Gemini 1.5 Flash achieves 24% accuracy on multiple needle retrieval with 10 needles at a context length of 128k characters, but only 10% accuracy at 630k characters. This suggests the existence of a task-specific effective context limit shorter than the advertised model limit, which has implications for practical deployment scenarios. |
| GazeGen: Gaze-Driven User Interaction for Visual Content Generation (Read more on arXiv or HuggingFace) | Kao-Den Chang, Wei-Te Mark Ting, Sai Qian Zhang, Ziyun Li, He-Yen Hsieh | GazeGen is a novel system for generating and editing visual content using real-time gaze tracking. The research aimed to create a hands-free, intuitive system for visual content manipulation using eye gaze. The system combines a novel lightweight gaze estimation model (DFT Gaze) with object detection and generative AI techniques like Stable Diffusion. DFT Gaze, with only 281K parameters, achieved a mean angular gaze error of 2.14° on the AEA dataset and operates 2x faster on edge devices than a larger model. This efficient and accurate real-time gaze estimation allows AI practitioners to develop novel human-computer interaction methods for visual content creation and editing accessible on resource-constrained devices. |
| RetrieveGPT: Merging Prompts and Mathematical Models for Enhanced Code-Mixed Information Retrieval (Read more on arXiv or HuggingFace) | Subhankar Maity, Aniket Deroy | This paper presents a novel approach for retrieving information from code-mixed text. The research aimed to improve information retrieval from Roman transliterated Bengali mixed with English, particularly in online conversations. The methodology involved using GPT-3.5 Turbo with carefully crafted prompts and integrating the output into a mathematical model considering sequential document dependencies. Results showed a marginal improvement in Mean Average Precision (MAP) from 0.701773 to 0.703734 in the best-performing submission. This suggests that prompting LLMs combined with mathematical modeling can offer minor improvements for information retrieval in code-mixed text, but further research is needed for substantial gains. |
| SG-I2V: Self-Guided Trajectory Control in Image-to-Video Generation (Read more on arXiv or HuggingFace) | Igor Gilitschenski, Yash Kant, Ziyi Wu, Sherwin Bahmani, Koichi Namekata | SG-I2V offers zero-shot control over object and camera trajectories in image-to-video generation. The research aimed to develop a method for controllable image-to-video generation without the computational expense of fine-tuning or reliance on external datasets. The key methodology involved modifying the spatial self-attention mechanism within a pre-trained video diffusion model (SVD) to align feature maps across frames and then optimizing the latent representations to enforce feature similarity along specified trajectories. On the VIPSeg dataset, SG-I2V achieved a mean object motion control (ObjMC) score of 14.43, demonstrating competitive motion fidelity compared to supervised methods. This offers AI practitioners a computationally efficient method for controlling video generation dynamics without requiring training data with motion annotations, streamlining the creation of videos with user-specified motion patterns. |
| VideoGLaMM: A Large Multimodal Model for Pixel-Level Visual Grounding in Videos (Read more on arXiv or HuggingFace) | Eric Xing, Jiale Cao, Wenqi Zhu, Hanan Gani, Shehan Munasinghe | VideoGLaMM is a large multimodal model designed for pixel-level visual grounding in videos, connecting language instructions with spatio-temporal visual content. The research aimed to develop a model capable of generating text responses intertwined with spatio-temporal object masks, demonstrating a fine-grained understanding of video content. The key methodology involved a dual vision encoder (spatial and temporal), a large language model (LLM), a spatio-temporal pixel decoder, and tunable Vision-Language (V→L and L→V) adapters, trained on a newly curated dataset of grounded video-QA triplets. VideoGLaMM achieved a mean Intersection over Union (mIOU) of 62.34% and a Recall of 0.103 on a grounded conversation generation task. This impactful mIOU result indicates that AI practitioners can leverage VideoGLaMM’s architecture and training methods to develop models for tasks requiring precise alignment of textual descriptions and visual elements in videos, like video captioning and content retrieval. |
| SVDQunat: Absorbing Outliers by Low-Rank Components for 4-Bit Diffusion Models (Read more on arXiv or HuggingFace) | Xiuyu Li, Tianle Cai, Zhekai Zhang, Yujun Lin, Muyang Li | SVDQuant is a post-training quantization technique for 4-bit weights and activations in diffusion models. The research aims to accelerate diffusion models while preserving image quality by quantizing both weights and activations to 4 bits. The key methodology involves migrating outliers from activations to weights via smoothing, then absorbing these magnified weight outliers using a 16-bit low-rank branch derived from Singular Value Decomposition (SVD), and finally fusing computations with a specialized inference engine called Nunchaku. On the 12B FLUX.1 model, SVDQuant achieved a 3.5x reduction in DiT inference memory and a 3.0x speedup compared to the 4-bit weight-only quantized (NF4 W4A16) baseline on an NVIDIA RTX 4090 GPU. This allows practitioners to deploy large diffusion models on resource-constrained hardware like laptops and accelerate interactive applications. |
Papers for 2024-11-07
| Title | Authors | Summary |
|---|---|---|
| Large Language Models Orchestrating Structured Reasoning Achieve Kaggle Grandmaster Level (Read more on arXiv or HuggingFace) | Albert Thomas, Giuseppe Paolo, James Doran, Alexandre Maraval, Antoine Grosnit | Agent K v1.0, an autonomous data science agent, automates and optimizes the data science lifecycle using structured reasoning and experiential learning. The research aimed to develop an end-to-end autonomous agent capable of achieving high performance on diverse data science tasks. The agent employs a structured reasoning framework with a memory module, interacting with various tools like Bayesian optimization and pre-trained models from Torchvision and HuggingFace. Agent K v1.0 achieved a 92.5% success rate in automating Kaggle competition tasks across multiple modalities and ranked in the top 38% of 5,856 human competitors based on Elo-MMR scores. AI practitioners can leverage Agent K v1.0’s approach to automate and improve performance across diverse data science tasks, potentially reducing manual effort and enhancing efficiency. |
| Both Text and Images Leaked! A Systematic Analysis of Multimodal LLM Data Contamination (Read more on arXiv or HuggingFace) | Benyou Wang, Lichao Sun, Shunian Chen, Sicheng Lai, Dingjie Song | MM-Detect, a framework for detecting multimodal data contamination in Large Language Models (LLMs), is introduced. The research aims to analyze and detect data contamination in Multimodal Large Language Models (MLLMs). The framework employs two methods: Option Order Sensitivity Test for multiple-choice VQA and Slot Guessing for Perturbation Captions for caption-based VQA, alongside metrics evaluating performance changes after applying these perturbations. Experiments on eleven MLLMs across five VQA datasets revealed that incorporating contaminated ScienceQA training data during LLaVA-1.5-7B training increased average CR by 8.2% and PCR by 3.7%. This indicates that data contamination is prevalent in both open-source and proprietary MLLMs, impacting performance evaluation and potentially creating unfair comparisons, and thus should be considered by practitioners when developing and benchmarking MLLMs. |
Papers for 2024-11-06
| Title | Authors | Summary |
|---|---|---|
| HtmlRAG: HTML is Better Than Plain Text for Modeling Retrieved Knowledge in RAG Systems (Read more on arXiv or HuggingFace) | Weipeng Chen, Mang Wang, Wen Wang, Zhicheng Dou, Jiejun Tan | HtmlRAG uses HTML instead of plain text to represent retrieved knowledge in Retrieval-Augmented Generation (RAG) systems. The research investigates whether HTML is superior to plain text for modeling retrieved knowledge and mitigating LLM hallucinations in RAG systems utilizing web data. The methodology involves HTML cleaning, compression, and a two-step pruning method (embedding-based and generative) to reduce HTML size and noise while preserving relevant information. On the ASQA dataset, HtmlRAG achieved a 33.31% Exact Match score with Llama-3.1-8B-Instruct-4k, outperforming all plain-text baselines. AI practitioners developing RAG systems can leverage HTML structure and semantics to improve the accuracy and factuality of LLM-generated responses, especially when utilizing web-based knowledge sources. |
| LLaMo: Large Language Model-based Molecular Graph Assistant (Read more on arXiv or HuggingFace) | Hyunwoo J. Kim, Dohwan Ko, Minseong Bae, Jinyoung Park | LLaMo is a large molecular graph-language model for instruction-following response generation in the molecular domain. The research aimed to develop an end-to-end trained large molecular graph-language model capable of general-purpose molecule and language understanding. The key methodology involves a multi-level graph projector that transforms graph representations into tokens, bridging the gap between graph and language modalities, coupled with instruction tuning using machine-generated molecular graph instruction data. LLaMo achieved a BLEU-4 score of 38.9 for molecular description generation, outperforming GPT-4 with in-context learning (27.0). This implies that AI practitioners can leverage LLaMo for improved performance in molecular tasks involving text and graph modalities, including description generation, property prediction, and IUPAC name prediction. |
| DeeR-VLA: Dynamic Inference of Multimodal Large Language Models for Efficient Robot Execution (Read more on arXiv or HuggingFace) | Shenzhi Wang, Yizeng Han, Bingyi Kang, Yulin Wang, Yang Yue | DeeR-VLA dynamically adjusts the size of activated Multimodal Large Language Models (MLLMs) for efficient robot execution. The research aims to reduce the computational demands of MLLMs for robotics, given limited hardware resources on robotic platforms. The key methodology is a dynamic early-exit framework that leverages a multi-exit MLLM architecture and algorithms to determine termination criteria based on resource constraints and action consistency. Experiments on the CALVIN benchmark showed a 5.2-6.5x reduction in LLM computational cost and a 2-6x reduction in LLM GPU memory without performance loss. This allows AI practitioners to deploy more complex MLLMs on robots with limited computational resources while maintaining performance. |
| Sample-Efficient Alignment for LLMs (Read more on arXiv or HuggingFace) | Min Lin, Wee Sun Lee, Chao Du, Changyu Chen, Zichen Liu | This paper introduces SEA, a sample-efficient algorithm for aligning Large Language Models (LLMs) with human preferences. The research aims to address the challenge of aligning LLMs effectively with limited human feedback. The key methodology involves a Thompson sampling-based algorithm incorporating an epistemic reward model, policy-guided search, and mixed preference learning. Experiments demonstrate SEA achieves higher win rates and 2-5x better sample efficiency compared to baseline approaches across multiple model scales and direct preference optimization methods. This implies AI practitioners can achieve more effective LLM alignment with significantly less human feedback using SEA. |
| DreamPolish: Domain Score Distillation With Progressive Geometry Generation (Read more on arXiv or HuggingFace) | Shiyu Huang, Wendi Zheng, Ming Ding, Yean Cheng, GhostCai | DreamPolish is a text-to-3D generation model that produces refined geometry and photorealistic textures. The objective is to generate high-quality 3D assets from text prompts, addressing limitations in existing methods regarding geometric detail and texture realism. The method uses progressive geometry construction with multiple neural representations, surface polishing with a normal estimator, and a novel domain score distillation (DSD) objective for texture enhancement. DreamPolish achieves a CLIP Score of 0.759, outperforming baseline models. This provides AI practitioners with a new method for generating high-fidelity 3D assets from text, potentially improving applications in areas like virtual reality, gaming, and 3D printing. |
| Zebra-Llama: A Context-Aware Large Language Model for Democratizing Rare Disease Knowledge (Read more on arXiv or HuggingFace) | Lashaw Salta, Chinmay Agrawal, Catalina Villouta, Andrew Langdon, ksoman | Zebra-Llama is a context-aware large language model specialized for Ehlers-Danlos Syndrome (EDS) information retrieval. The objective was to develop a model capable of providing accurate and comprehensive responses to EDS-related queries, including proper citations. The researchers fine-tuned a Llama 3.1-8B-Instruct model using a dataset of question-context-answer triplets derived from medical literature, patient forums, and social media discussions, with a focus on context-aware training using a specialized RAG implementation. Zebra-Llama achieved 77.5% thoroughness compared to 70.1% for the base model on a test set of real-world questions from EDS patients and clinicians. This improved performance suggests that context-aware, domain-specific fine-tuning can significantly enhance LLMs for specialized information retrieval tasks, offering a promising avenue for developing AI solutions for rare diseases and other specialized domains. |
| Controlling Language and Diffusion Models by Transporting Activations (Read more on arXiv or HuggingFace) | Nicholas Apostoloff, Luca Zappella, Michal Klein, Arno Blaas, Pau Rodriguez | Activation Transport (ACT) offers fine-grained control over Large Language Models (LLMs) and text-to-image diffusion models (T2Is) by steering activations. The research aimed to develop a modality-agnostic framework for steering activations to control the generation of LLMs and T2Is. The key methodology involves using optimal transport theory to learn a transport map between source and target activation distributions and applying this map at inference time. Linear-ACT achieved up to a 7.5x reduction in toxicity on the Gemma2-2B LLM benchmark with minimal impact on perplexity and MMLU accuracy. AI practitioners can leverage ACT to enhance the controllability and safety of generative models by mitigating unwanted behaviors (like toxicity) and inducing desired concepts or styles during generation, without retraining. |
| GarVerseLOD: High-Fidelity 3D Garment Reconstruction from a Single In-the-Wild Image using a Dataset with Levels of Details (Read more on arXiv or HuggingFace) | Zirong Jin, Wanghao Du, Chenghong Li, Haolin Liu, Zhongjin Luo | GarVerseLOD introduces a new dataset and framework for reconstructing high-fidelity 3D garment meshes from single in-the-wild images. The research aimed to address the challenges of generalizing to diverse poses, deformations, and details in single-view 3D garment reconstruction. The key methodology involves a hierarchical dataset (GarVerseLOD) with levels of detail (LOD) and a coarse-to-fine reconstruction approach that leverages linear blend skinning and implicit garment representations with geometry-aware boundary prediction. The method achieved a Chamfer Distance of 7.825, outperforming compared methods. This provides AI practitioners with a new dataset and model for robust 3D garment reconstruction applicable to various fields like virtual try-on and fashion design, enabling the generation of detailed garment models from limited visual input. |
| Correlation of Object Detection Performance with Visual Saliency and Depth Estimation (Read more on arXiv or HuggingFace) | Dylan Seychell, mbar0075 | This paper investigates the correlation of object detection accuracy with visual saliency and depth prediction. The research aimed to determine whether visual saliency or depth prediction correlates more strongly with object detection accuracy. The study used four pre-trained models (DeepGaze IIE, Depth Anything, DPT-Large, and Itti’s model) to generate predictions on the COCO and Pascal VOC datasets, comparing them to ground truth annotations using mean Average Pearson Correlation (mAp). Visual saliency exhibited a stronger correlation (mAp up to 0.459 on Pascal VOC) with object detection accuracy than depth prediction (mAp up to 0.283 on Pascal VOC). This suggests that incorporating visual saliency features into object detection models may improve performance, particularly in complex scenes. |
Papers for 2024-11-05
| Title | Authors | Summary |
|---|---|---|
| AndroidLab: Training and Systematic Benchmarking of Android Autonomous Agents (Read more on arXiv or HuggingFace) | Hao Yu, Siyi Cheng, Xueqiao Sun, Xiao Liu, Yifan Xu | ANDROIDLAB is a framework for training and evaluating autonomous agents interacting with Android devices. The research aimed to create a standardized environment and benchmark for Android agents using both large language models (LLMs) and large multimodal models (LMMs). They developed a benchmark with 138 tasks across 9 apps, and created the Android Instruct Dataset for fine-tuning models. Fine-tuning with their dataset improved the success rate of open-source LLMs from 4.59% to 21.50%, and LMMs from 1.93% to 13.28%. This resource allows AI practitioners to train and systematically evaluate open-source Android agent models using a standardized benchmark and dataset, facilitating development and comparison of new agent models. |
| WebRL: Training LLM Web Agents via Self-Evolving Online Curriculum Reinforcement Learning (Read more on arXiv or HuggingFace) | Hanyu Lai, Iat Long Iong, Xiao Liu, Zehan Qi, tianjiezhang | WEBRL is a novel reinforcement learning framework for training large language model (LLM) web agents in online environments. The research aimed to improve the performance of open-source LLMs on web-based tasks, addressing challenges like task scarcity, sparse feedback, and policy distribution drift. The study uses a self-evolving online curriculum, an outcome-supervised reward model, and adaptive reinforcement learning strategies in online web environments. Llama-3.1-8B, trained with WEBRL, achieved a 42.4% success rate on WebArena-Lite, surpassing previous state-of-the-art open LLM-based web agents and even proprietary LLMs like GPT-4-Turbo (17.6%). This implies that WEBRL can significantly enhance the performance of open-source LLMs in web-based tasks, making autonomous web agents more accessible and powerful for AI practitioners. |
| Training-free Regional Prompting for Diffusion Transformers (Read more on arXiv or HuggingFace) | Wenzhao Zheng, Jianjin Xu, wanghaofan, wangyida, antonio-c | This paper introduces a training-free regional prompting method for diffusion transformers. The objective is to enhance compositional text-to-image generation in diffusion transformer models, specifically FLUX.1, by enabling them to handle complex, multi-regional prompts with precise layout control. The key methodology involves manipulating the attention maps within the diffusion transformer architecture based on user-provided or LLM-generated regional prompt-mask pairs. Results show the method generates images that adhere to multiple regional prompts simultaneously and achieves up to 9x faster inference speed compared to an RPG-based regional control method for 16 masks. This provides AI practitioners with a more efficient and flexible approach to achieving fine-grained control over image generation using diffusion transformers without requiring model retraining or additional training data. |
| DynaMath: A Dynamic Visual Benchmark for Evaluating Mathematical Reasoning Robustness of Vision Language Models (Read more on arXiv or HuggingFace) | Bin Hu, Junyu Zhang, Xingang Guo, Chengke Zou, Ray2333 | DYNAMATH, a dynamic visual benchmark, evaluates the robustness of Vision Language Models (VLMs) in mathematical reasoning. The research investigated whether VLMs’ reasoning procedures are robust to problem variations that pose no challenge to humans. The key methodology involved creating 501 seed questions as Python programs, enabling generation of 5,010 concrete questions with variations in visual and textual content. Evaluation showed the worst-case accuracy (percentage of correctly answered seed questions across all variants) of the best performing VLM, Claude-3.5, was 35.3%, significantly lower than its average-case accuracy. This substantial difference between average-case and worst-case accuracy highlights the unreliability of current VLMs when handling variations in mathematical reasoning tasks, signaling a critical area for improvement in model robustness. |
| Hunyuan-Large: An Open-Source MoE Model with 52 Billion Activated Parameters by Tencent (Read more on arXiv or HuggingFace) | Jiaqi Zhu, Xingwu Sun, Ruobing-Xie, Mimosa77, YanfengChen | Tencent introduces Hunyuan-Large, a 389 billion parameter Mixture-of-Experts (MoE) model with 52 billion activated parameters. The objective was to develop a large, open-source MoE model with superior performance across diverse NLP tasks compared to similar-sized models. They leveraged large-scale synthetic data (7 trillion tokens), a novel recycle routing strategy within the MoE architecture, and explored scaling laws for MoE models. Hunyuan-Large achieved 88.4% on MMLU, outperforming the LLama3.1-70B model and exhibiting comparable performance to the significantly larger LLama3.1-405B. The release of Hunyuan-Large offers AI practitioners a powerful, open-source MoE model for a wide range of applications, as well as insights into effective MoE model training for future development. |
| How Far is Video Generation from World Model: A Physical Law Perspective (Read more on arXiv or HuggingFace) | Yang Zhao, Zhijie Lin, Rui Lu, Bingyi Kang, Yang130 | Here’s a summary of the AI research paper following the specified guidelines: i) 1-line summary: A study evaluates the ability of scaled video generation models to learn and generalize fundamental physical laws from visual data alone. ii) Main research question/objective: Can video generation models, scaled in data and parameters, discover and generalize fundamental physical laws solely from visual observations without human priors? iii) Key methodology: A 2D physics simulation testbed generated videos of objects governed by deterministic physical laws (uniform linear motion, elastic collisions, parabolic motion). Diffusion-based video generation models were trained and evaluated on in-distribution, out-of-distribution, and combinatorial generalization tasks. Quantitative metrics assessed adherence to physical laws. iv) Primary results: While scaling improved in-distribution generalization, out-of-distribution generalization remained poor, with velocity errors an order of magnitude higher than in-distribution errors even with maximum model size and data. Combinatorial generalization showed improvement with scaling but was still imperfect (67% to 10% reduction in “abnormal” cases). Analysis revealed a “case-based” generalization mechanism, prioritizing color over shape, size, and velocity. v) Principal implication for AI practitioners: Scaling alone is insufficient for video generation models to uncover fundamental physical laws; models prioritize superficial visual features over underlying physical principles, necessitating further research on generalization mechanisms beyond simple scaling. The significant gap between in-distribution and out-of-distribution generalization suggests that current approaches have significant limitations in truly understanding and modeling the physical world. |
| Survey of Cultural Awareness in Language Models: Text and Beyond (Read more on arXiv or HuggingFace) | Junho Myung, Arnav Arora, Junyeong Park, jinjh0123, sidicity | This paper surveys research on incorporating cultural awareness into text-based and multimodal language models (LLMs). The survey aims to consolidate research on making LLMs culturally inclusive, encompassing benchmarks, training data creation, and alignment methodologies. The authors review over 300 papers, categorizing cultural awareness efforts across various modalities, including image, video, and audio, in addition to text. Multilingual descriptions in image captioning benchmarks yield 29.9% more objects, 24.5% more relations, and 46.0% more attributes compared to monolingual captions. AI practitioners should consider incorporating culture-specific data and benchmarks in the development and evaluation of LLMs to mitigate biases and improve cross-cultural understanding, but should carefully evaluate sources for bias, inconsistencies in culture definitions, and the ethical implications of cultural alignment. |
| LIBMoE: A Library for comprehensive benchmarking Mixture of Experts in Large Language Models (Read more on arXiv or HuggingFace) | Quang Pham, Van Nguyen, Luong Tran, doantienthongbku, DavidNguyen | LibMoE is a modular toolkit for streamlining the research, training, and evaluation of Mixture of Experts (MoE) algorithms in Large Language Models (LLMs). The research aimed to develop a comprehensive framework making MoE algorithm research more accessible and standardized. The key methodology involved implementing various state-of-the-art MoE algorithms within a modular framework incorporating distributed training and zero-shot evaluation across 11 benchmarks, utilizing sparse upcycling from pre-trained LLM checkpoints. Results showed no single MoE algorithm consistently outperformed others across all benchmarks, with performance averaging 55-56% accuracy across the tasks. A key implication for AI practitioners is that the standard Sparse Mixture of Experts (SMoE) strategy remains a highly competitive choice due to its simplicity and scalability, despite the existence of more complex MoE algorithms. |
| Sparsing Law: Towards Large Language Models with Greater Activation Sparsity (Read more on arXiv or HuggingFace) | Chaojun Xiao, Yingfa Chen, Chenyang Song, Yuqi Luo, SillyXu | This paper investigates scaling properties and influential factors of intrinsic activation sparsity in decoder-only Transformer LLMs. The research aims to understand how to achieve greater activation sparsity in LLMs without compromising performance. Researchers used a proposed metric, PPL-p% sparsity, to measure activation sparsity while controlling for performance degradation (perplexity). They found ReLU-activated LLMs achieve greater sparsity than SiLU-activated LLMs at the same parameter scale, while maintaining comparable performance. Specifically, ReLU activation ratio on a 0.1B parameter model converges to approximately 6.14% with sufficient training data, whereas SiLU converges to approximately 40.9%. These findings suggest AI practitioners should consider ReLU as the activation function when aiming to maximize activation sparsity for efficiency and interpretability gains in LLMs. |
| GenXD: Generating Any 3D and 4D Scenes (Read more on arXiv or HuggingFace) | Linjie Li, Zhiwen Yan, Kevin Lin, Chung-Ching Lin, Yuyang Zhao | GenXD is a unified model for generating 3D and 4D scenes from single or multiple conditioned images. The research aimed to develop a unified framework for generating consistent and high-quality 3D (static viewpoint changes) and 4D (spatial and temporal changes) content. The authors curated a large-scale 4D dataset (CamVid-30K) from videos, estimating camera poses and object motion, and designed GenXD with multiview-temporal modules within a masked latent conditioned diffusion model. On the Cam-DAVIS benchmark, GenXD achieved an FID score of 101.78 for single view 4D generation, surpassing existing camera-conditioned video generation methods. This allows AI practitioners to generate videos aligned with camera trajectories and containing realistic object motion, advancing the capabilities of 3D and 4D content creation. |
| DynaSaur: Large Language Agents Beyond Predefined Actions (Read more on arXiv or HuggingFace) | Ryan A. Rossi, Seunghyun Yoon, Viet Dac Lai, Dang Nguyen, Franck-Dernoncourt | DynaSaur is an LLM agent framework that dynamically creates and composes actions as Python functions, accumulating them for reuse in subsequent tasks. The research aims to address limitations of existing LLM agents restricted to predefined action sets by enabling dynamic action creation and composition. The key methodology involves representing actions as Python functions, executing them through an interpreter, and accumulating generated actions. DynaSaur outperformed baseline models on the GAIA benchmark, achieving an average exact match percentage of 51.61% with GPT-40 on Level 1 tasks. This framework allows AI agents greater flexibility in problem-solving and adaptability to diverse tasks by generating and executing arbitrary actions, which is highly relevant for building more general and versatile agents. |
| Adaptive Caching for Faster Video Generation with Diffusion Transformers (Read more on arXiv or HuggingFace) | Menglin Jia, Ding Liu, Sen He, Haozhe Liu, kumarak | AdaCache accelerates video diffusion transformer inference by adaptively caching and reusing computations. The research aims to reduce the computational cost of generating high-fidelity videos with Diffusion Transformers (DiTs), especially over longer durations. The core method involves a content-dependent caching schedule within transformer blocks, guided by a distance metric measuring the change in residual connections between diffusion steps, and further regularized by a motion estimation component (MoReg). AdaCache achieves up to a 4.7× speedup on Open-Sora 720p - 2s video generation compared to the baseline, with comparable or slightly reduced quality based on quantitative metrics. This training-free, plug-and-play method allows AI practitioners to significantly improve the inference latency of video DiTs without requiring model retraining or sacrificing substantial generation quality. |
| Decoding Dark Matter: Specialized Sparse Autoencoders for Interpreting Rare Concepts in Foundation Models (Read more on arXiv or HuggingFace) | Virginia Smith, Mona Diab, Aashiq Muhamed | Specialized Sparse Autoencoders (SSAEs) are introduced to capture rare concepts in foundation models. The research aims to address the challenge of current Sparse Autoencoders (SAEs) failing to capture rare, yet crucial, concepts within subdomains of data. The key methodology involves finetuning general-purpose SAEs on subdomain data selected via dense retrieval and trained with Tilted Empirical Risk Minimization (TERM). SSAEs achieved a 12.5% increase in worst-group classification accuracy compared to general-purpose SAEs on the Bias in Bios dataset when used to remove spurious gender information. This result indicates that SSAEs offer a more powerful lens for inspecting subdomain-specific features in foundation models, potentially leading to improvements in fairness and bias mitigation by enhancing the representation of underrepresented groups or tail concepts. |
| Swan and ArabicMTEB: Dialect-Aware, Arabic-Centric, Cross-Lingual, and Cross-Cultural Embedding Models and Benchmarks (Read more on arXiv or HuggingFace) | Muhammad Abdul-Mageed, Fakhraddin Alwajih, Abdellah El Mekki, El Moatez Billah Nagoudi, Gagan Bhatia | This paper introduces Swan, a family of Arabic-centric embedding models, and ArabicMTEB, a benchmark for evaluating them. The research aimed to develop improved Arabic text embedding models addressing dialectal and cultural nuances not captured by existing multilingual models. The researchers trained Swan-Small and Swan-Large models using a diverse corpus of Arabic text, including MSA, dialectal variations, and cross-lingual data, and evaluated them on ArabicMTEB, covering retrieval, classification, and bitext mining tasks. Swan-Large achieved a state-of-the-art average score of 62.45 on ArabicMTEB, outperforming Multilingual-E5-large (61.65). This provides AI practitioners with new state-of-the-art, cost-effective Arabic embedding models and a benchmark for developing and evaluating future Arabic-centric NLP systems. |
Papers for 2024-11-04
| Title | Authors | Summary |
|---|---|---|
| OS-ATLAS: A Foundation Action Model for Generalist GUI Agents (Read more on arXiv or HuggingFace) | Fangzhi Xu, Zhenyu Wu, Zhiyong Wu, heroding77, QiushiSun | OS-Atlas is a large action model designed to improve GUI agent performance in grounding and out-of-distribution (OOD) scenarios. The research aimed to develop a foundation model for GUI agents that excels in grounding and generalizes to unseen interfaces, addressing the limitations of existing open-source models. The authors created a multi-platform GUI grounding data synthesis toolkit and curated the largest open-source, multi-platform GUI grounding dataset to date, containing over 13 million GUI elements across web, desktop, and mobile platforms. OS-Atlas-Base achieved state-of-the-art grounding accuracy of 82.47% on ScreenSpot benchmark. This work provides AI practitioners with a high-performing, open-source foundation model and dataset, facilitating the development of more robust and generalizable GUI agents. |
| Constant Acceleration Flow (Read more on arXiv or HuggingFace) | Youngjoon Hong, Taehoon Lee, Sihyeon Kim, Sojin Lee, Dogyun Park | Constant Acceleration Flow (CAF) is a novel ODE-based generative model for faster, high-quality image generation. The research aimed to improve the speed and accuracy of diffusion-based image generation by addressing limitations of constant velocity models like Rectified Flow. CAF introduces a constant acceleration term into the ODE trajectory and employs initial velocity conditioning and a reflow process to improve trajectory estimation. On CIFAR-10 with conditional settings, CAF achieved a Fréchet Inception Distance (FID) of 1.39 in one-step generation, surpassing state-of-the-art baselines. AI practitioners can leverage CAF for faster, higher-quality image generation in applications requiring few-step inference. |
| Randomized Autoregressive Visual Generation (Read more on arXiv or HuggingFace) | Liang-Chieh Chen, Xiaohui Shen, Xueqing Deng, turkeyju, yucornetto | This paper introduces Randomized AutoRegressive modeling (RAR) for enhanced visual generation using autoregressive transformers. The objective is to improve autoregressive image generation quality while maintaining compatibility with language modeling frameworks. RAR uses a randomness annealing training strategy where input image tokens are randomly permuted during training with a probability that linearly decays from 1 to 0, encouraging bidirectional context learning. On ImageNet-256, RAR achieves a FID score of 1.48, surpassing previous autoregressive and even some leading diffusion and masked transformer models. This implies that AI practitioners can leverage RAR to develop higher-quality autoregressive image generation models that are also compatible with existing language modeling architectures and optimization techniques. |
| Adapting While Learning: Grounding LLMs for Scientific Problems with Intelligent Tool Usage Adaptation (Read more on arXiv or HuggingFace) | Leon Bergen, Duncan Watson-Parris, Yadi Cao, yuqirose, Bohan22 | The paper introduces a two-stage training method to improve LLM performance on scientific problems, balancing inherent reasoning and external tool use. The research aims to address the issue of LLMs over-relying on tools or hallucinating answers for complex scientific problems. The methodology involves World Knowledge Distillation (WKD) to internalize domain knowledge and Tool Usage Adaptation (TUA) to train adaptive tool usage based on problem complexity. Results show an average 28.18% improvement in answer accuracy and a 13.89% improvement in tool usage precision across six scientific datasets. This implies that AI practitioners can enhance LLM accuracy and efficiency on scientific tasks by training models to adaptively leverage external tools based on problem difficulty. |
| Personalization of Large Language Models: A Survey (Read more on arXiv or HuggingFace) | Yijia Shao, Branislav Kveton, Ryan A. Rossi, Zhehao Zhang, Franck-Dernoncourt | This paper surveys techniques for personalizing Large Language Models (LLMs). The authors aim to unify the disparate research on personalized text generation and downstream task personalization using LLMs. They propose taxonomies for personalization granularity (user-level, persona-level, global preference), techniques (RAG, prompting, representation learning, RLHF), evaluation metrics (intrinsic, extrinsic), and datasets. One study found that larger LLMs (100B+ parameters) performed comparably or better than traditional recommender systems in user rating prediction after fine-tuning with minimal user interaction data. AI practitioners can leverage these taxonomies and techniques, along with insights into evaluation and datasets, to build more user-centric and effective personalized LLM applications. |
| SambaMixer: State of Health Prediction of Li-ion Batteries using Mamba State Space Models (Read more on arXiv or HuggingFace) | Sergio Martin, Clara Pérez-Molina, sascha-kirch, jolalde5 | SambaMixer is a novel structured state space model (SSM) for predicting the state of health (SOH) of Li-ion batteries. The objective is to develop a deep learning model capable of accurately predicting Li-ion battery SOH using multivariate time series data from discharge cycles. The proposed SambaMixer model uses a MambaMixer architecture incorporating anchor-based resampling of time series data, positional encodings based on sample time and time between discharge cycles, and a regression head. On the NASA battery dataset, SambaMixer achieved a Mean Absolute Error (MAE) of 1.072% for SOH prediction. This result suggests that SambaMixer, using Mamba SSMs, offers a performant and efficient alternative to transformer-based models for multivariate time series prediction tasks relevant to battery health management. |
| In-Context LoRA for Diffusion Transformers (Read more on arXiv or HuggingFace) | Huanzhang Dou, Yupeng Shi, Zhi-Fan Wu, Wei Wang, lhhuang | This paper introduces In-Context LoRA (IC-LORA), a method for adapting text-to-image diffusion transformers to diverse generative tasks. The research investigates whether existing text-to-image DiTs possess inherent in-context generation capabilities and, if so, how to effectively leverage them. The key methodology involves concatenating images and their corresponding captions, then fine-tuning a LoRA with small task-specific datasets (20-100 samples). Qualitative results demonstrate high-fidelity image set generation across various tasks, including portrait photography, font design, and home decoration. The paper does not present quantitative benchmarks, so specific performance metrics like FID or CLIP scores are unavailable. This pipeline offers AI practitioners a simplified and computationally efficient approach to adapt pre-trained text-to-image models for various downstream tasks without extensive training or architectural modifications, emphasizing the potential of inherent in-context learning capabilities within these models. |
| M2rc-Eval: Massively Multilingual Repository-level Code Completion Evaluation (Read more on arXiv or HuggingFace) | Shukai Liu, Jian Yang, Congnan Liu, Ken Deng, Jiaheng Liu | This paper introduces M²RC-EVAL, a benchmark for evaluating repository-level code completion in multiple programming languages. The objective is to address the limitations of existing benchmarks that focus on few languages and lack fine-grained analysis, hindering comprehensive evaluation of multilingual code LLMs. The researchers created M²RC-EVAL by collecting data from The Stack v2, selecting completion positions based on abstract syntax tree (AST) nodes, and adding bucket-level and semantic-level annotations. After fine-tuning StarCoder-7B on the accompanying M²RC-INSTRUCT dataset, the model achieved 44.4% exact match and 71.4% edit similarity on M²RC-EVAL, significantly outperforming the non-finetuned model. The demonstrated effectiveness of cross-file context and fine-tuning on M²RC-INSTRUCT indicates that AI practitioners should incorporate these elements when developing or improving code LLMs for real-world repository-level completion tasks, particularly in multilingual settings. |
| HelloMeme: Integrating Spatial Knitting Attentions to Embed High-Level and Fidelity-Rich Conditions in Diffusion Models (Read more on arXiv or HuggingFace) | Chenhui Xue, Chaojie Yang, Tian Li, Nianhong Jiao, Shengkai Zhang | HelloMeme introduces Spatial Knitting Attentions (SK Attentions) to enhance text-to-image diffusion models for complex downstream tasks like meme video generation. The research aimed to develop a method for adapting pre-trained text-to-image models to specialized tasks without sacrificing generalization performance. The core methodology involves integrating adapters employing SK Attentions into the diffusion model’s UNet architecture, facilitating the fusion of high-level (head pose, facial expression) and fidelity-rich (reference image) features. In self-reenactment experiments, the method achieved an average PSNR of 31.08 dB, outperforming other open-source state-of-the-art methods. This method provides AI practitioners with a plugin-based approach for post-training text-to-image models, enabling adaptation to tasks requiring high fidelity and complex control while preserving the base model’s capabilities. |
| Zipfian Whitening (Read more on arXiv or HuggingFace) | Hidetoshi Shimodaira, Hiroto Kurita, Han Bao, Sho Yokoi | This paper proposes Zipfian whitening, a post-processing method for word embeddings that incorporates word frequency. The research investigates whether accounting for the non-uniform distribution of word frequencies (Zipf’s law) when symmetrizing word embedding spaces improves downstream task performance. The key methodology involves performing PCA whitening weighted by empirical word frequencies, emphasizing low-frequency words. Zipfian whitening consistently outperformed standard centering/whitening and other baselines, achieving a 66.92% score on the STS-B benchmark using GloVe embeddings. AI practitioners should consider using Zipfian whitening as a post-processing step for word embeddings, as it demonstrably improves performance on downstream tasks by better capturing the information content of rare words. |
| WikiNER-fr-gold: A Gold-Standard NER Corpus (Read more on arXiv or HuggingFace) | Pierre-François Marteau, Nicolas Béchet, Danrun Cao | This paper presents WikiNER-fr-gold, a manually corrected version of a subset of the French portion of the WikiNER corpus for Named Entity Recognition (NER). The objective was to create a gold-standard NER dataset by correcting inconsistencies and errors in the silver-standard WikiNER-fr. The authors manually reviewed and corrected 20% (26,818 sentences, ~700,000 tokens) of the French portion of the WikiNER corpus, using a labeling tool and referring to Wikipedia pages for disambiguation and consistency checks. The corrected sub-corpus, WikiNER-fr-gold, exhibits improved annotation consistency compared to the original WikiNER-fr. This provides AI practitioners with a higher-quality gold-standard French NER dataset for training and evaluating NER models, potentially improving their performance. |
| Survey of User Interface Design and Interaction Techniques in Generative AI Applications (Read more on arXiv or HuggingFace) | Reuben Luera, puneetm, zhangry868, subright, Franck-Dernoncourt | This paper surveys user interface (UI) design and interaction techniques in user-guided generative AI applications. The objective is to create a design compendium of current UI/UX trends and techniques for generative AI, focusing on user-guided interactions. The methodology involved surveying over 100 research articles on generative AI, categorizing UI interaction techniques, layouts, and human-AI engagement levels. The survey identified common interaction patterns like prompting, selection, system manipulation, and object manipulation, as well as prevalent UI layouts like conversational and canvas-based interfaces. One key finding is that users utilizing hybrid interactions in DirectGPT completed tasks 50% faster compared to single-dimensional interactions like those in ChatGPT. This implies that AI practitioners should consider incorporating multimodal and hybrid interaction designs to optimize user workflow and efficiency in generative AI applications. |
| GRS-QA – Graph Reasoning-Structured Question Answering Dataset (Read more on arXiv or HuggingFace) | Jincen Shuai, Devasha Trivedi, Anish Pahilajani, Franck-Dernoncourt, namyongp | GRS-QA, a new dataset, is introduced for evaluating multi-hop question answering models with explicit reasoning structures. The research aimed to investigate the impact of reasoning structures on Large Language Model (LLM) performance in multi-hop question answering. The authors constructed reasoning graphs from existing multi-hop QA datasets, categorizing them by structure and generating negative samples by perturbing graph structures. When using retrieved evidence, GPT-3.5 achieved an F1 score of 0.70 on bridge_2_1 questions and 0.78 on comparison_2_1 questions. AI practitioners should consider reasoning structures alongside semantic content when developing and evaluating multi-hop QA models, as model performance varies significantly with differing reasoning graph complexities. |
Papers for 2024-11-01
| Title | Authors | Summary |
|---|---|---|
| Unpacking SDXL Turbo: Interpreting Text-to-Image Models with Sparse Autoencoders (Read more on arXiv or HuggingFace) | Robert West, Justin Deschenaux, Mikhail Terekhov, Chris Wendler, surokpro2 | This paper investigates the interpretability of SDXL Turbo, a few-step text-to-image diffusion model. The research objective is to understand the computational roles of transformer blocks within SDXL Turbo’s U-net during image generation. The methodology involves training sparse autoencoders (SAEs) on the updates performed by four key transformer blocks, followed by qualitative and quantitative analysis of the learned features. The results reveal that different transformer blocks specialize in distinct aspects of image generation, such as composition (down.2.1), local details (up.0.0), and style/color (up.0.1), with average pairwise CLIP similarity between images activating the same feature being significantly higher than the random baseline. This specialization suggests that AI practitioners can potentially manipulate specific image attributes by targeting interventions at corresponding transformer blocks within SDXL Turbo or similar architectures. |
| What Happened in LLMs Layers when Trained for Fast vs. Slow Thinking: A Gradient Perspective (Read more on arXiv or HuggingFace) | Tianyi Zhou, Yanhong Li, MingLiiii | This paper investigates the layer-wise gradient patterns in LLMs during instruction-tuning with varying reasoning paths and response types. The research aims to understand how “fast” (without Chain-of-Thought) and “slow” (with detailed Chain-of-Thought) thinking affects the training dynamics of LLMs. The study analyzes gradient norms, particularly in projection layers (Query, Key, Value, Output), using Singular Value Decomposition and metrics like Mean Absolute Difference and Relative Difference, across different layers and models (pre-trained and instruction-finetuned). Results on datasets like AQUA and ECQA show that slow thinking leads to more stable gradients across layers, with smaller Mean Absolute Differences compared to fast thinking (e.g., on AQUA, fast thinking had a MAD of 4.42, while slow thinking had a MAD of 0.28 for all projection layers). This suggests slow thinking, via CoT, improves the stability of LLM training and potentially informs more efficient and stable instruction-tuning strategies for AI practitioners. |
| A Pointer Network-based Approach for Joint Extraction and Detection of Multi-Label Multi-Class Intents (Read more on arXiv or HuggingFace) | Pawan Goyal, Gajula Sai Chaitanya, Abhilash Nandy, Sombit Bose, Ankan Mullick | This paper introduces a novel approach for extracting multiple intent spans and detecting multiple intents within a sentence. The research aimed to address the limitations of existing intent detection models, which primarily handle single-intent queries, by developing a model capable of extracting multiple intent spans and classifying coarse and fine-grained intent labels. The researchers propose a pointer network-based architecture (MLMCID) using RoBERTa and XLM-R embeddings with a novel multi-label, multi-class intent dataset (MLMCID-dataset). RoBERTa with Pointer Network in MLMCID achieved 92.3% accuracy and 88.3% Macro F1-score for primary intent detection with coarse labels on the CLINC dataset. This research provides AI practitioners with a specialized architecture for building more robust and context-aware dialogue systems capable of handling complex, multi-intent user queries, even in few-shot settings. |
| Constraint Back-translation Improves Complex Instruction Following of Large Language Models (Read more on arXiv or HuggingFace) | Lei Hou, Bin Xu, Xiaozhi Wang, Hao Peng, Yunjia Qi | Constraint back-translation improves complex instruction following in LLMs. The research aimed to enhance LLMs’ ability to follow instructions with multiple constraints. The key methodology involved generating constraints from existing instruction-response pairs using Llama3-70B-Instruct and creating a dataset called CRAB. Post-training on CRAB improved performance across benchmarks, with Llama3CRAB+DPO achieving 49.7% average score on IFEval. This implies that AI practitioners can leverage constraint back-translation to improve the complex instruction-following capabilities of LLMs. |
| Language Models can Self-Lengthen to Generate Long Texts (Read more on arXiv or HuggingFace) | Dayiheng Liu, An Yang, Bowen Yu, Tianyi Tang, Shanghaoran Quan | Self-Lengthen, an iterative training framework, enhances LLMs’ ability to generate long, aligned text. The research aimed to address the limitation of current LLMs in generating lengthy, aligned outputs due to a training gap in pre-training and post-training data. The methodology involves a Generator that produces initial responses and an Extender that lengthens them iteratively, with both models being retrained on the longer outputs. Experiments showed Self-Lengthen increased output length from approximately 1,000 words to 8,000 words while preserving quality. This provides AI practitioners a method to improve long text generation capabilities of LLMs without needing external long-form data or proprietary models. |
| BenchX: A Unified Benchmark Framework for Medical Vision-Language Pretraining on Chest X-Rays (Read more on arXiv or HuggingFace) | Xinxing Xu, Sicong Leng, Yanyu Xu, Tan Li Hui Faith, youngzhou12 | BenchX provides a standardized benchmark for evaluating Medical Vision-Language Pretraining (MedVLP) models on chest X-ray tasks. The research aimed to create a unified framework for comparing and analyzing MedVLP methods, addressing inconsistencies in existing evaluation protocols. The framework uses the MIMIC-CXR dataset for pretraining and nine public chest X-ray datasets across classification, segmentation, report generation, and retrieval tasks, with standardized preprocessing and finetuning protocols. ConVIRT, an early MedVLP method, achieved 77.0% AUROC on NIH ChestX-ray dataset with 1% of training data when finetuned with layer normalization, truncated normal initialization, and discriminative learning rates. This suggests that proper training configurations are crucial for evaluating MedVLP methods and that the efficacy of some older models may be underestimated due to variations in prior evaluation methodologies. |
| BitStack: Fine-Grained Size Control for Compressed Large Language Models in Variable Memory Environments (Read more on arXiv or HuggingFace) | Yunhua Zhou, Dong Zhang, Bo Wang, Pengyu Wang, Xinghao Wang | BitStack is a training-free weight compression method for LLMs that allows dynamic adjustment of model size based on available memory. The research aimed to address the challenge of deploying compressed LLMs in environments with variable memory availability. The core methodology involves iterative absolute value decomposition of weight matrices and sorting of resulting residual blocks based on their impact on perplexity, allowing dynamic loading and unloading of these blocks. On the Llama 3.1 70B model, BitStack achieved 89% of the original FP16 model’s zero-shot performance at a high compression ratio. This allows AI practitioners to deploy LLMs on resource-constrained devices and dynamically adjust the model size based on real-time memory availability, improving usability and performance within memory constraints. |
| Navigating the Unknown: A Chat-Based Collaborative Interface for Personalized Exploratory Tasks (Read more on arXiv or HuggingFace) | Qingwei Lin, Jue Zhang, Zhiyang Zhang, Xiaoting Qin, Yingzhe Peng | CARE, a chat-based collaborative interface, enhances personalized exploratory tasks using a multi-agent LLM framework. The research aimed to improve personalization and reduce cognitive load in LLM-based chatbots for exploratory tasks, particularly when users begin with vague queries. A within-subject user study with 22 participants compared CARE to a baseline LLM chatbot. 16 out of 22 participants preferred CARE, and CARE was rated significantly higher in reducing cognitive load (x²(4) = 19.04, p = 0.001). This structured, multi-agent approach can guide AI practitioners in designing more effective and personalized conversational AI systems for complex tasks. |
| DELTA: Dense Efficient Long-range 3D Tracking for any video (Read more on arXiv or HuggingFace) | Sergey Tulyakov, Evangelos Kalogerakis, Chuang Gan, Peiye Zhuang, Tuan Duc Ngo | DELTA performs dense 3D tracking of every pixel in a video using a coarse-to-fine strategy. The research aims to develop an efficient method for dense, long-range 3D motion tracking from monocular video. The method leverages a joint global-local attention mechanism at reduced resolution for initial tracking, followed by an attention-based upsampler for high-resolution predictions. On the Kubric 3D dataset, DELTA achieves 81.4% Average Jaccard (AJ) for 3D tracking, outperforming prior methods while being significantly faster. This provides AI practitioners with a computationally efficient and accurate method for dense 3D motion estimation, applicable to tasks requiring fine-grained motion analysis in videos. |
| Learning Video Representations without Natural Videos (Read more on arXiv or HuggingFace) | Yossi Gandelsman, Xinlei Chen, Xueyang Yu | This paper explores learning video representations using solely synthetic data and natural still images. The research investigates whether natural videos are essential for training effective video representations. The authors train VideoMAE models on a progression of synthetic video datasets with increasing complexity, alongside datasets of natural image crops. A VideoMAE model pre-trained on synthetic videos with natural image crops achieves 91.3% accuracy on UCF101 action classification, matching the performance of a model pre-trained on UCF101 itself. This suggests that AI practitioners may be able to train effective video models without large, curated natural video datasets, potentially simplifying data acquisition and addressing privacy or bias concerns. |
Papers for 2024-10-31
| Title | Authors | Summary |
|---|---|---|
| CORAL: Benchmarking Multi-turn Conversational Retrieval-Augmentation Generation (Read more on arXiv or HuggingFace) | Hongjin Qian, Ziliang Zhao, Kelong Mao, dongguanting, ariya2357 | CORAL is a new benchmark for evaluating multi-turn conversational Retrieval-Augmented Generation (RAG) systems. The research aimed to create a benchmark dataset for evaluating the performance of RAG systems in multi-turn conversational settings. The key methodology involved automatically converting English Wikipedia pages into 8,000 multi-turn, information-seeking conversations using four different conversation flow sampling strategies and large language models. Qwen2.5-1.5B-SFT achieved the highest retrieval score, outperforming commercial closed-source LLMs with 23.1 MRR. This benchmark enables AI practitioners to rigorously evaluate and improve multi-turn conversational RAG systems, facilitating the development of more robust and knowledge-grounded conversational AI agents. |
| A Large Recurrent Action Model: xLSTM enables Fast Inference for Robotics Tasks (Read more on arXiv or HuggingFace) | Korbinian Pöppel, Maximilian Beck, Vihang Patil, Thomas Adler, Thomas Schmied | Here’s a concise summary of the AI research paper following your strict guidelines: i) This paper investigates the suitability of modern recurrent architectures, particularly xLSTM, for building large action models (LAMs) to achieve fast inference in robotics. ii) The main objective was to test the hypothesis that modern recurrent models are better suited for LAMs than Transformers regarding training and inference speed. iii) The researchers developed a Large Recurrent Action Model (LRAM) using xLSTM and trained it on a large-scale multi-domain dataset (894M transitions from 432 tasks) using a supervised learning setting similar to Decision Transformer. iv) Experiments showed that xLSTM-based LRAMs outperformed Transformers in terms of both performance and speed across 432 tasks; specifically, on the 206M parameter models, xLSTM achieved better performance than Transformers and the inference time was significantly lower, with significantly reduced latency compared to Transformer-based models across different context lengths. v) The most impactful finding, the superior inference speed of xLSTM-based LRAMs, suggests that modern recurrent architectures offer a compelling alternative to Transformers for real-time robotic applications requiring fast inference. The paper lacks information regarding the specific hardware used for the comparison of speed/latency. |
| Stealing User Prompts from Mixture of Experts (Read more on arXiv or HuggingFace) | Nicholas Carlini, Jamie Hayes, Ilia Shumailov, Itay Yona | This paper demonstrates a novel attack exploiting architectural flaws in Mixture-of-Experts (MoE) LLMs to extract user prompts. The research aimed to determine if an adversary could exploit Expert-Choice-Routing (ECR) in MoE models to disclose a victim’s prompt when batched together. The attack manipulated expert routing within a two-layer Mixtral model using crafted adversarial batches, triggering the ECR tie-breaker to leak information. In their evaluation, 99.9% (4833/4838) of the secret tokens across a test set of 1000 common English words were successfully recovered. This vulnerability highlights the critical need for AI practitioners to consider prompt security and batch independence during the design and deployment of MoE-based LLMs. |
| AutoMIR: Effective Zero-Shot Medical Information Retrieval without Relevance Labels (Read more on arXiv or HuggingFace) | Xiao Zhou, Xiangxu Zhang, Lei Li, zl101 | This paper introduces SL-HyDE, a self-learning framework for zero-shot medical information retrieval. The research aims to develop an effective dense retrieval system for medical information without requiring relevance-labeled training data. The key methodology involves a self-learning framework that iteratively refines a large language model (LLM) for generating hypothetical documents and a dense retrieval model for document ranking. SL-HyDE improved NDCG@10 by an average of 4.9% across ten datasets compared to HyDE (Qwen2 as generator + BGE as retriever). This improvement suggests that AI practitioners can leverage SL-HyDE to develop more accurate medical information retrieval systems without the need for expensive and time-consuming manual annotation of relevance data. |
| TokenFormer: Rethinking Transformer Scaling with Tokenized Model Parameters (Read more on arXiv or HuggingFace) | Jan Eric Lenssen, Yongqin Xian, Muhammad Ferjad Naeem, Yue Fan, Haiyang Wang | TokenFormer introduces a fully attention-based architecture for scaling transformer models. The research aims to address the high computational cost of scaling transformers, which traditionally requires retraining from scratch when architectural changes are made. The core methodology replaces linear projections in transformers with a token-parameter attention layer, treating model parameters as tokens that interact with input tokens via attention. Scaling TokenFormer from 124M to 1.4B parameters incrementally achieves a perplexity of 11.77, comparable to a transformer trained from scratch at 1.4B parameters but at significantly reduced training cost. This allows AI practitioners to scale transformer models more efficiently by reusing pre-trained models and avoiding computationally expensive retraining from scratch. |
Papers for 2024-10-30
| Title | Authors | Summary |
|---|---|---|
| CLEAR: Character Unlearning in Textual and Visual Modalities (Read more on arXiv or HuggingFace) | Denis Bobkov, Boris Mikheev, Alexey Zhavoronkin, Dmitrii Korzh, therem | This research aims to evaluate machine unlearning (MU) techniques in multimodal large language models (MLLMs). The authors introduce CLEAR, a synthetic dataset of fictitious individuals with associated images and text, and evaluate 10 adapted MU methods across textual, visual, and multimodal setups using metrics like ROUGE-L, probability score, truth ratio, and forget quality. In multimodal unlearning on the CLEAR dataset using the LLaVa model, the SCRUB method maintained a retain metric of approximately 0.48 while achieving a forget metric of 0.36. This suggests that current state-of-the-art unlearning algorithms struggle with multimodal setups, demonstrating the need for new approaches specifically designed for MLLMs. The paper also indicates that L1 regularization on LoRA adapter weights can mitigate catastrophic forgetting. Follow-up questions: 1. How does the performance of the evaluated MU methods on the synthetic CLEAR dataset compare to performance on real-world multimodal datasets, and what modifications might be necessary for practical application? 2. What is the computational cost of applying L1 regularization on LoRA weights during unlearning, and how does this impact the feasibility of applying this technique to larger MLLMs? 3. Given the observed challenges in multimodal unlearning, what specific research directions might be most promising for developing more effective MMU algorithms, such as exploring alternative regularization techniques or novel architectural modifications? |
| AutoKaggle: A Multi-Agent Framework for Autonomous Data Science Competitions (Read more on arXiv or HuggingFace) | Qianbo Zang, Ziming Li, zhangysk, Liam-Liu, aaabiao | This paper aims to develop AutoKaggle, a framework for autonomously completing Kaggle data science competitions using tabular data. The framework utilizes a phase-based workflow with five specialized agents (Reader, Planner, Developer, Reviewer, and Summarizer) combined with iterative debugging, unit testing, and a machine learning tools library. In evaluations across eight Kaggle competitions, AutoKaggle achieved a valid submission rate of 0.83 using the GPT-40 model. This indicates the potential for multi-agent systems to automate complex data science workflows, achieving near-human-level performance. The paper does not explicitly state the performance metrics of the individual agents, which makes it difficult to assess their respective contributions. Follow-up questions: 1. Could the authors elaborate on the specific roles and interactions of each agent within the multi-agent system, and provide quantitative measures of their individual performance or contribution to the overall system performance? 2. How does the performance of AutoKaggle vary across different types of Kaggle competitions (e.g., classification vs. regression, different dataset sizes)? Are there certain competition characteristics where it performs particularly well or poorly, and why? 3. What are the limitations of the current machine learning tools library, and what future extensions or improvements are planned to enhance its capabilities and address the observed debugging challenges related to feature engineering tools? |
| SocialGPT: Prompting LLMs for Social Relation Reasoning via Greedy Segment Optimization (Read more on arXiv or HuggingFace) | Chuang Gan, Donglai Wei, Jiawei Zhou, zmeng0116, EthanTaylor | a) Objective: To develop a zero-shot social relation recognition framework that addresses the limitations of existing end-to-end models in terms of generalizability and interpretability. b) Methodology: SocialGPT, a modular framework, utilizes Vision Foundation Models (VFMs) to convert images into textual social stories and Large Language Models (LLMs) with a structured prompt (SocialPrompt) for text-based reasoning. Greedy Segment Prompt Optimization (GSPO) automatically tunes the SocialPrompt using gradient information at the segment level. c) Results: SocialGPT with Vicuna-13B and GSPO achieved 69.23% accuracy on the PIPA dataset, exceeding the prior state-of-the-art TRGAT by 1.4%. d) Implication: AI practitioners can leverage SocialGPT as a strong zero-shot baseline for social relation recognition, utilizing the power of pre-trained VFMs and LLMs while benefiting from GSPO for automatic prompt optimization and enhanced performance. The paper mentions additional benefits of interpretability of results and generalization to novel image styles but does not provide supporting quantitative details. Follow-up Questions: 1. How does the performance of GSPO compare to other prompt optimization methods on social relation recognition tasks, particularly those not relying on segment-level optimization? 2. What are the computational costs and time complexities of GSPO, particularly concerning the number of segments and candidate prompts? 3. The paper claims generalization to novel image styles. What is the quantifiable performance on these styles (e.g. sketch, cartoon) compared to existing models and in what domains or use cases are these improvements most significant? |
| OpenWebVoyager: Building Multimodal Web Agents via Iterative Real-World Exploration, Feedback and Optimization (Read more on arXiv or HuggingFace) | Hongming Zhang, Wenhao Yu, Kaixin Ma, Wenlin Yao, Hongliang He | This research aims to develop an open-source, multimodal web agent capable of improving its performance through iterative real-world exploration and feedback. The methodology involves imitation learning from a GPT-40-based agent, followed by cycles of self-exploration, GPT-40 feedback, and optimization using the Idefics2-8b-instruct LMM. On the WebVoyager test set, the agent’s task success rate increased from 19.9% after imitation learning to 25.8% after three optimization cycles. This suggests that iterative optimization with real-world feedback can improve open-source, multimodal web agent performance. The paper does not detail the computation resources or time required for training or optimization. Follow-up Questions: 1. What were the specific hyperparameter settings used for fine-tuning Idefics2-8b-instruct during both the imitation learning and iterative optimization phases? 2. How does the performance of OpenWebVoyager compare to closed-source multimodal models like GPT-4V on more complex web navigation tasks not included in the evaluated datasets? 3. What is the breakdown of successes and failures attributed to visual understanding versus textual understanding limitations within the agent? |
| Flow-DPO: Improving LLM Mathematical Reasoning through Online Multi-Agent Learning (Read more on arXiv or HuggingFace) | Paul Mineiro, ydeng9 | a) This research aims to improve the quality of reasoning traces generated by Large Language Models (LLMs) for mathematical problem-solving. b) The proposed method uses an online learning Flow comprising multiple LLMs that collaboratively construct solutions, trained via Direct Preference Optimization (DPO) with rollouts. c) Using flow-generated reasoning traces for Supervised Fine-Tuning (SFT) led to an accuracy of 71.3% on GSM8K and 27.8% on MATH for Llama-3-8B-Instruct, outperforming SFT with self-generated and ground-truth traces. d) AI practitioners can use online-learned multi-agent Flows to generate superior reasoning traces for LLM fine-tuning, leading to improved performance in complex reasoning tasks. The paper highlights the impact of flow-generated reasoning traces for improving single-model SFT performance in math problem-solving, offering a new approach to enhance LLM reasoning capabilities. Follow-up questions: 1. What are the computational resource requirements (e.g., GPU hours, memory) for training the flow and performing SFT with the proposed method compared to baseline methods? 2. How does the chunk size parameter affect the performance and training efficiency of the Flow, and what strategies can be used for optimizing this parameter? 3. Could this approach be generalized to other reasoning tasks beyond mathematics, such as commonsense reasoning or logical deduction? |
| ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inference (Read more on arXiv or HuggingFace) | Ningxin Zheng, Size Zheng, Wenlei Bao, Li-Wen Chang, preminstrel | a) The research aimed to improve the throughput of long-context large language model (LLM) inference, which is hampered by the growing memory footprint and access needs of the key-value (KV) cache. b) SHADOWKV, a proposed system, stores a low-rank representation of the pre-Rotary Position Embedding (pre-RoPE) key cache on the GPU, offloads the value cache to the CPU, and employs a chunk-based approximation method with outlier detection for sparse attention during decoding. c) On an A100 GPU, SHADOWKV achieved up to a 3.04× throughput increase for Llama-3.1-8B with a batch size of 122K context length samples, surpassing the theoretical throughput of an infinite batch size under the assumption of infinite GPU memory. d) AI practitioners can leverage SHADOWKV to significantly improve the serving efficiency of long-context LLMs without substantial accuracy degradation by reducing the KV cache’s memory footprint and optimizing sparse attention mechanisms. Follow-up questions: 1. What are the practical considerations and potential trade-offs involved in implementing the low-rank approximation and value offloading strategy for different hardware configurations (e.g., systems with limited CPU memory or varying PCIe bandwidth)? 2. How does SHADOWKV’s chunk-based KV selection method compare to other sparse attention techniques in terms of computational complexity and robustness to different LLM architectures and tasks? 3. Is the code publicly available, and what level of technical expertise is required to integrate SHADOWKV into existing LLM serving pipelines? |
| Robots Pre-train Robots: Manipulation-Centric Robotic Representation from Large-Scale Robot Dataset (Read more on arXiv or HuggingFace) | Yongyuan Liang, Huanyu Li, Tao Huang, Yifei Sun, Guangqi Jiang | This research investigates whether manipulation-centric visual representations improve robot learning. The authors propose Manipulation Centric Representation (MCR), which pre-trains a visual encoder on the DROID robotic dataset and incorporates dynamics information (robot actions and proprioceptive states) via a novel contrastive loss, an action prediction loss, and a time contrastive loss. Across four simulated robotic manipulation domains, MCR outperforms the strongest baseline by 14.8% in terms of average success rate. The most impactful finding is the strong correlation between “manipulation centricity,” the representation’s ability to focus on manipulation-relevant regions, and downstream task performance. This implies that AI practitioners can improve robot learning efficiency by designing representations that prioritize manipulation-relevant information. Follow-up questions: 1. How does the choice of pre-trained backbone architecture (ResNet vs. ViT) influence the effectiveness of MCR and its manipulation centricity? 2. Could MCR be adapted for other robotic tasks beyond manipulation, such as navigation or grasping, and if so, how might the pre-training objectives need to be modified? 3. What are the limitations of using Grad-CAM to measure manipulation centricity, and are there alternative, potentially more robust methods for evaluating this characteristic? |
| Precise and Dexterous Robotic Manipulation via Human-in-the-Loop Reinforcement Learning (Read more on arXiv or HuggingFace) | Sergey Levine, Jeffrey Wu, charlesxu0124, jianlanluo | a) This research aims to develop a reinforcement learning (RL) system for vision-based robotic manipulation capable of acquiring diverse dexterous skills in real-world settings. b) The system, HIL-SERL, uses a sample-efficient off-policy RL algorithm (RLPD) with a pretrained visual backbone, incorporates human demonstrations and corrections, and employs a sparse reward function based on a trained binary classifier. c) HIL-SERL achieves a 100% success rate on nearly all evaluated tasks within 1 to 2.5 hours of real-world training, representing an average 101% improvement in success rate and 1.8x faster cycle time compared to imitation learning baselines trained with an equivalent amount of human data. d) The results indicate that carefully designed RL systems can enable real-world acquisition of complex vision-based manipulation policies within practical training times, exceeding imitation learning and potentially unlocking wider application of robots in complex manipulation tasks. The most impactful finding is the high success rate achieved in short training times, highlighting the potential of RL for real-world robotics applications previously considered infeasible. Follow-up questions: 1. How does the system’s performance vary with different pretrained visual backbones, and are there ways to optimize backbone selection for specific manipulation tasks? 2. What are the limitations of the current human correction interface (SpaceMouse), and how could more intuitive and efficient interfaces enhance performance and broaden the range of correctible errors? 3. While the paper mentions the lack of extensive randomization and tests in unstructured environments, how could these be incorporated into future research to validate the generalizability and deployability of HIL-SERL in real-world scenarios? |
Papers for 2024-10-29
| Title | Authors | Summary |
|---|---|---|
| Bielik 7B v0.1: A Polish Language Model – Development, Insights, and Evaluation (Read more on arXiv or HuggingFace) | Remek, adgw, djstrong, lflis, chrisociepa | This research aimed to develop a high-performing Polish language model. The authors adapted the Mistral 7B v0.1 model and further pre-trained it on a curated dataset of Polish and English texts, incorporating techniques like Weighted Instruction Cross-Entropy Loss and Adaptive Learning Rate. Evaluation on the Open PL LLM Leaderboard showed a 9 percentage point improvement over Mistral-7B-v0.1 on the RAG Reader task. This implies that adapting and further training existing multilingual models can significantly improve performance for specific languages. The paper does not detail the exact composition of the training dataset (sizes of Polish vs. English portions, etc.) and the rationale behind the chosen weights for the Weighted Instruction Cross-Entropy Loss. Follow-up questions: 1. What were the specific data cleaning and quality assessment procedures used for the Polish portion of the training dataset, and how did they contribute to the observed performance gains? 2. Could the authors provide further details on the distribution of weights assigned to the instruction-response pairs in the Weighted Instruction Cross-Entropy Loss and explain how these specific values were determined? 3. What is the detailed split between instruction data from OpenHermes-2.5, orca-math, and the manually generated instruction data in the post-training dataset? |
| AgentStore: Scalable Integration of Heterogeneous Agents As Specialized Generalist Computer Assistant (Read more on arXiv or HuggingFace) | Fangzhi Xu, Qiushi Sun, Zhuohang Dang, Minnan Luo, Chengyou Jia | This research aimed to develop a scalable platform for integrating heterogeneous agents to automate computer operating system tasks. The key methodology involved creating AgentStore, a platform with an AgentPool of specialized agents, an AgentEnroll protocol for adding new agents, and a MetaAgent using an AgentToken strategy to manage and select agents for task execution. On the OSWorld benchmark, AgentStore achieved a 23.85% success rate, more than doubling the previous best system’s performance (11.21%). This implies that for AI practitioners, integrating specialized agents significantly enhances agent systems in both generalization and specialization for complex, open-ended computer tasks. The paper does not provide details about the training data or the agent integration protocol, stating they will be available when the project is open-sourced. Follow-up questions: 1. What is the specific architecture of the MetaAgent, including details about its multimodal processing capabilities and how it integrates the system state information? 2. Can you elaborate on the agent integration protocol, specifically the format and content of the document developers need to provide during AgentEnroll? 3. How does the automated process with self-instruct generate diverse and consistent training data for AgentToken, and what mechanisms prevent hallucination or irrelevant data generation during this process? |
| GPT-4o System Card (Read more on arXiv or HuggingFace) | Adam Perelman, Adam P. Goucher, Adam Lerer, Aaron Hurst, OpenAI | a) This system card analyzes GPT-40, an omni-modal AI model, assessing its capabilities, limitations, and safety implications, with a focus on speech-to-speech interactions. b) Evaluations include external red teaming across diverse languages and demographics, converting existing text-based evaluations to audio using text-to-speech, and Preparedness Framework assessments for cybersecurity, bio-threats, persuasion, and model autonomy. c) GPT-40’s voice output classifier achieved 96% precision and 100% recall in English for detecting deviations from authorized voices. d) AI practitioners should be aware of the potential for misuse of voice generation capabilities, the residual risk of unintentional voice generation despite mitigations, and the potential for disparate performance across accents and languages, necessitating further research and mitigation development. Follow-up questions: 1. What specific techniques were used in post-training to align the voice model to ideal completions and prevent unauthorized voice generation? 2. How does GPT-40’s performance on non-English languages compare to its performance on English across other modalities besides text, such as image and video understanding? 3. What are the limitations of the current evaluations, especially concerning the use of TTS for converting text-based evaluations to audio, and how can future evaluations be improved to address these limitations? |
| Document Parsing Unveiled: Techniques, Challenges, and Prospects for Structured Information Extraction (Read more on arXiv or HuggingFace) | Zhengren Wang, Junyuan Zhang, Bin Wang, Victor Shea-Jay Huang, Qintong Zhang | This paper surveys document parsing techniques for extracting structured information from various document formats. The authors review both modular pipeline systems, comprised of layout analysis, content extraction, and relation integration stages, and end-to-end approaches using vision-language models (VLMs). The survey consolidates commonly used datasets, like PubLayNet for layout analysis and ICDAR for OCR, and associated evaluation metrics, including IoU for layout analysis and character error rate for text recognition. While lacking quantitative comparisons between the modular and VLM approaches, the authors highlight the emerging trend of unified frameworks and universal OCR paradigms exemplified by models like GOT, which achieved performance improvements on complex charts and non-traditional content. This suggests that VLMs offer a promising path towards more general and efficient document parsing solutions. Follow-up Questions: 1. Given the limitations discussed for both modular systems and VLMs, what specific strategies (e.g., architectural changes, training techniques) could be most effective for improving the performance of VLMs on high-density text and complex table structures found in document images? 2. What are the comparative computational resource requirements (training time, memory, inference speed) of modular systems and end-to-end VLM approaches for document parsing, and how do these impact practical deployment considerations? 3. While GOT demonstrates a promising universal OCR approach, how effectively does it generalize to diverse document types and languages beyond the datasets mentioned in the paper, and what further research is needed to assess its real-world applicability across different domains? |
| LongReward: Improving Long-context Large Language Models with AI Feedback (Read more on arXiv or HuggingFace) | Zhenyu Hou, Shulin Cao, Xin Lv, Zhongni Hou, Jiajie Zhang | a) The research aims to improve the performance of long-context large language models (LLMs), addressing the issue of compromised quality in LLM-synthesized training data. b) The proposed method, LongReward, uses an off-the-shelf LLM to provide rewards for model responses based on helpfulness, logicality, faithfulness, and completeness, combined with the Direct Preference Optimization (DPO) reinforcement learning algorithm. c) Experiments showed that DPO models using LongReward outperformed supervised fine-tuning (SFT) models on long-context tasks by 4.9% and 5.5% for Llama-3.1-8B and GLM-4-9B, respectively. d) LongReward provides a practical method for aligning long-context LLMs with human preferences, enabling AI practitioners to train models with improved long-context capabilities and reduced hallucinations. Follow-up questions: 1. What is the computational cost of using LongReward, particularly with respect to the number of API calls to the judge LLM, and how can this be optimized for practical deployment? 2. How does the choice of the “off-the-shelf” LLM used as the judge in LongReward affect the performance and biases of the final trained long-context LLM? 3. Could LongReward be adapted for other RL algorithms beyond DPO, and what might be the potential benefits or drawbacks of such adaptations? |
| DreamClear: High-Capacity Real-World Image Restoration with Privacy-Safe Dataset Curation (Read more on arXiv or HuggingFace) | Xiaotian Han, Huaibo Huang, Xiaoqiang Zhou, Yuang Ai, Ye27 | This research aims to improve real-world image restoration (IR) by addressing dataset limitations and developing a high-capacity model. The authors introduce GenIR, a privacy-preserving data pipeline using text-to-image diffusion models and multimodal large language models to generate a synthetic dataset of one million high-quality images. They then present DreamClear, a Diffusion Transformer-based IR model incorporating degradation priors via a Mixture of Adaptive Modulator (MoAM). On the LSDIR-Val benchmark, DreamClear achieves a 0.3836 LPIPS score. This work offers practitioners a method for creating large-scale, privacy-safe IR datasets and a high-performing model leveraging diffusion and degradation priors. Follow-up questions: 1. What are the specific architectural details and hyperparameters of the routing network (R) within the MoAM module, and how were these determined? 2. While the paper mentions model distillation and quantization as potential solutions for improving inference speed, are there any specific experiments or preliminary results demonstrating the effectiveness of these methods on DreamClear? 3. Could the GenIR pipeline be adapted for other vision tasks beyond image restoration, and what modifications might be necessary for such adaptations? |
| MarDini: Masked Autoregressive Diffusion for Video Generation at Scale (Read more on arXiv or HuggingFace) | Yanping Xie, Mengmeng Xu, Zijian Zhou, Shikun Liu, Haozhe Liu | a) The research aimed to develop a scalable and efficient video generation model that combines the flexibility of masked autoregressive (MAR) modeling with the stability of diffusion models (DMs). b) MarDini uses an asymmetric architecture with a MAR planning model operating on low-resolution inputs to generate planning signals, and a lightweight DM generating high-resolution frames conditioned on these signals and unmasked frames. A progressive training strategy with increasing task difficulty (from video interpolation to image-to-video generation) and resolution was employed. c) MarDini-L/T achieved an FVD score of 117.13 on the DAVIS-7 video interpolation benchmark, surpassing previous methods. The paper does not explicitly report results for image-to-video generation on VBench without motion score guidance. d) AI practitioners can leverage MarDini’s architecture and training strategy to develop efficient and scalable video generation models trained from scratch without relying on generative image pre-training, enabling the creation of long-term video interpolations, video expansions, and image-to-video animations using a single model. The paper does not provide sufficient detail to assess general image-to-video generation performance compared to state-of-the-art, only reporting a subset of the evaluated VBench metrics. Follow-up Questions: 1. Could you elaborate on the specific implementation details of the “Identity Attention” mechanism and quantify its impact on training stability across different model sizes and resolutions? 2. How does MarDini’s performance on standard image-to-video generation tasks (with full motion score guidance) compare to state-of-the-art models on VBench? The paper references improved “physical principles” but doesn’t quantify this, and it only compares MarDini to other methods on a subset of VBench’s metrics. 3. What are the limitations of the current progressive training scheme, and how can it be further optimized for even greater scalability and efficiency in terms of both training time and resource utilization? |
| A Survey of Small Language Models (Read more on arXiv or HuggingFace) | Samyadeep Basu, Yu Xia, Ryan Aponte, Xuan Shen, Chien Van Nguyen | a) This survey aims to provide a comprehensive overview of Small Language Models (SLMs), focusing on their architectures, training techniques, and model compression methods. b) The authors propose a novel taxonomy categorizing SLM optimization methods based on the techniques used (pre-processing, training, post-processing) and the constraints addressed (inference compute, training time, etc.). c) MobileBERT achieved a 4.3x size reduction and a 5.5x speedup compared to the base version of BERT. d) AI practitioners can utilize this taxonomy and the survey’s summary of existing techniques to select appropriate methods for developing and deploying SLMs under specific resource constraints. Follow-up questions: 1. While the survey mentions trade-offs between optimization goals, are there any quantitative analyses or specific examples that illustrate these trade-offs (e.g., memory-efficient training vs. inference speed)? 2. The paper mentions neural architecture search (NAS) for SLMs. Are there recommended NAS methods or tools specifically suited for the scale and characteristics of SLMs, and how do they compare in terms of computational cost and effectiveness? 3. How does data privacy for small language models compare to data privacy for large language models with the same underlying architecture, i.e. is privacy “easier” with small language models because less data is available to analyze for extraction of personal or protected data? |
| GrounDiT: Grounding Diffusion Transformers via Noisy Patch Transplantation (Read more on arXiv or HuggingFace) | Minhyuk Sung, Taehoon Yoon, Phillip Y. Lee | a) This research aims to develop a training-free spatial grounding technique for text-to-image generation using Diffusion Transformers (DiT) that allows for precise control over object placement within user-specified bounding boxes. b) The proposed method, GrounDiT, employs a two-stage denoising process: a global update based on cross-attention map alignment with bounding boxes and a local update involving the cultivation and transplantation of noisy image patches, leveraging DiT’s “semantic sharing” property. c) On the HRS benchmark, GrounDiT achieves 45.01% spatial accuracy, a +14.87% improvement over the previous state-of-the-art training-free method (R&B). d) AI practitioners can use GrounDiT to enhance user controllability in text-to-image generation with DiT models by achieving fine-grained spatial grounding without model retraining. This enables more precise object placement and layout control for various applications like image editing and compositional image generation. Follow-up questions: 1. The paper mentions increased computational cost due to separate object branches. How does this cost scale with the number of bounding boxes, and what are the practical implications for real-time applications? 2. Could the semantic sharing property be exploited for other tasks beyond spatial grounding, such as style transfer or controlled image manipulation within specific regions? 3. While the paper focuses on PixArt-α, how adaptable is GrounDiT to other DiT architectures, and what modifications might be necessary for optimal performance? |
| COAT: Compressing Optimizer states and Activation for Memory-Efficient FP8 Training (Read more on arXiv or HuggingFace) | Kurt Keutzer, Yao Lu, Ligeng Zhu, Han Cai, Haocheng Xi | a) The paper investigates reducing the memory footprint of FP8 training for large language and vision-language models, specifically targeting optimizer states and activations which are often kept in higher precision in existing FP8 training frameworks. b) COAT (Compressing Optimizer States and Activations for FP8 Training) introduces Dynamic Range Expansion for optimizer states and Mixed-Granularity Activation Quantization, combining per-tensor and per-group quantization. c) COAT achieved a 1.54x reduction in end-to-end training memory compared to BF16 and a 1.43x speedup on Llama-7B, 13B, and 30B models, while maintaining nearly lossless performance across various tasks. d) AI practitioners can utilize COAT to enable full-parameter training of larger models on fewer GPUs or double batch sizes in distributed settings, facilitating more efficient large-scale model training. This improved memory efficiency translates directly into larger batch sizes and potentially longer context lengths, both beneficial for training larger models. Follow-Up Questions: 1. How does COAT’s Dynamic Range Expansion handle potential overflow or underflow issues, particularly with second-order momentum which the paper mentions is sensitive to quantization? 2. The paper mentions per-group quantization for activations of non-linear layers - what specific group sizes were found to be optimal for different model architectures and how sensitive is the performance to these group size choices? 3. What is the impact of COAT on inference latency, and how easily can models trained with COAT be deployed for inference with existing FP8 inference solutions? |
| Vision Search Assistant: Empower Vision-Language Models as Multimodal Search Engines (Read more on arXiv or HuggingFace) | Xiangyu Yue, Xiaohan Ding, Yiyuan Zhang, Zhixin Zhang | a) The paper aims to improve the generalization ability of Vision-Language Models (VLMs) to handle unseen images and novel concepts by integrating them with web search agents. b) The proposed Vision Search Assistant framework uses a three-step process: 1) Visual Content Formulation to extract object-level descriptions and correlations from images using a VLM. 2) Web Knowledge Search, an iterative algorithm using an LLM as a planning agent to generate sub-questions and a searching agent to retrieve and summarize web information. 3) Collaborative Generation, combining visual content, user prompt, and web knowledge to generate the final answer using the VLM. c) In closed-set evaluations on the LLaVA-W benchmark, Vision Search Assistant achieved an overall score of 84.9%, a +6.4% improvement over the baseline LLaVA 1.6-7B model. d) AI practitioners can leverage this framework to build more robust and adaptable VLMs capable of handling real-world, open-domain scenarios requiring up-to-date information and complex reasoning about visual content. The ability to integrate real-time information access through a web search significantly enhances VLM performance, particularly in reasoning tasks. Follow-up questions: 1. What are the computational costs and latency implications of the iterative Web Knowledge Search process, particularly for complex images requiring multiple iterations? 2. How robust is the system to noisy or irrelevant web search results, and what mechanisms are in place to ensure the quality and reliability of the retrieved information? 3. Could the Visual Content Formulation stage benefit from more advanced scene graph generation techniques to better capture relationships between objects beyond simple co-occurrence in captions? |
| LARP: Tokenizing Videos with a Learned Autoregressive Generative Prior (Read more on arXiv or HuggingFace) | Abhinav Shrivastava, Hao Chen, Yixuan Ren, Saksham Suri, Hanyu Wang | a) The paper aims to develop a video tokenizer optimized for autoregressive (AR) generative models, addressing limitations of existing patchwise tokenizers in capturing holistic representations and efficiently aligning with AR generation. b) LARP employs holistic tokenization using learned queries, a stochastic vector quantizer (SVQ), and a lightweight AR transformer as a training-time prior model to structure the latent space for AR generation. c) On the UCF101 class-conditional video generation benchmark, LARP achieved a state-of-the-art Fréchet Video Distance (FVD) score of 57. d) AI practitioners can utilize LARP to improve the quality and efficiency of AR video generation, potentially enabling the development of more sophisticated and scalable video generation models. The paper’s emphasis on aligning the latent space with the generative process is impactful, suggesting a potential pathway for enhancing AR model performance in various visual domains. Follow-up questions: 1. How does the computational cost of LARP, including the training-time prior model, compare to existing video tokenizers, particularly during inference? 2. Could the holistic tokenization approach of LARP be adapted for other AR tasks beyond video generation, such as video captioning or action recognition? 3. The paper mentions using a Llama-like transformer as the AR generative model. What specific architecture and hyperparameters were used, and how were they chosen? |
| Fast Best-of-N Decoding via Speculative Rejection (Read more on arXiv or HuggingFace) | Jiahao Qiu, Huitao Yang, Ruiqi Zhang, Momin Haider, Hanshi Sun | a) The research aims to develop a more computationally efficient inference-time alignment algorithm for Large Language Models (LLMs) that achieves comparable performance to Best-of-N decoding with large N. b) The proposed Speculative Rejection algorithm begins with a large initial batch size and iteratively prunes lower-scoring partial utterances based on a reward model, dynamically reducing computational cost. c) Using Llama-3-8B with the RM-Mistral-7B reward model on the AlpacaFarm dataset, Speculative Rejection achieved a reward score comparable to Best-of-N with N between 1920 and 3840, requiring 16-32x fewer GPUs. d) AI practitioners can utilize Speculative Rejection to significantly reduce the computational resources needed for inference-time alignment of LLMs, enabling the use of higher effective N values on single accelerators, potentially improving alignment effectiveness. e) The paper notes that different combinations of LLMs and reward models vary in reward score improvement, and the relation between this variance and LLM or reward model properties is not fully explored. Follow-up questions: 1. How does the choice of rejection rate (α) affect the trade-off between computational cost and final reward score across different LLM architectures and reward model complexities? 2. Could the performance of Speculative Rejection be further improved by incorporating prompt-dependent adaptive rejection rates or by using reward models trained as value functions? 3. Are there other metrics beyond reward score, such as diversity or fairness, that could be incorporated into the rejection criteria for Speculative Rejection? |
| Neural Fields in Robotics: A Survey (Read more on arXiv or HuggingFace) | Abhinav Valada, Nick Heppert, Yen-Chen Lin, Mauro Comi, Muhammad Zubair Irshad | a) This survey paper reviews the applications of Neural Fields (NFs) across various robotics domains, analyzing their benefits and limitations. b) The authors categorize and analyze over 200 research papers on NFs in robotics, focusing on core frameworks like Occupancy Networks, Signed Distance Fields, Neural Radiance Fields, and Gaussian Splatting, and their use in pose estimation, manipulation, navigation, physics simulation, and autonomous driving. c) The paper shows a rapid growth in NF robotics publications, increasing from 6 publications comprising 10% of total NF publications in 2021 to 73 publications making up 22% in 2023. d) The survey provides AI practitioners with a comprehensive overview of existing NF techniques in robotics, highlighting their strengths and weaknesses in different applications, aiding in informed selection and development of future NF-based robotic systems. Follow-up questions: 1. Given the computational intensity of NFs, what specific optimization strategies are most promising for deploying them in real-time robotic applications on resource-constrained hardware? 2. What are the most effective methods for integrating semantic information, like that from foundation models, into NF representations to improve generalization and enable higher-level reasoning capabilities in robots? 3. How can NFs be effectively combined with physics simulators to create physically realistic training environments for robots, and what are the main challenges in ensuring successful sim-to-real transfer of learned policies? |
| Language Models And A Second Opinion Use Case: The Pocket Professional (Read more on arXiv or HuggingFace) | David Noever | This research investigated the effectiveness of Large Language Models (LLMs) as second opinion tools in complex medical and legal scenarios. The study analyzed LLM performance on 183 challenging medical cases from Medscape and 21 Supreme Court cases, comparing responses to crowd-sourced physician and published judicial decisions, respectively. Foundational LLMs achieved >81% accuracy on straightforward medical cases but only 43% accuracy on complex medical cases, compared to consensus human expert answers. This disparity suggests that while LLMs excel in information retrieval and structured scenarios, they currently struggle with the nuanced reasoning required for complex, real-world problem-solving. The paper doesn’t specify details of the LLM prompting or fine-tuning strategies used. Follow-up questions: 1. What specific prompting strategies were employed to elicit detailed reasoning and alternative diagnoses from the LLMs, and how did prompt engineering influence performance, particularly in ambiguous cases? 2. How did the inclusion of visual data (for the subset of cases with imaging) affect LLM performance across different models, and were there specific image processing or multimodal fusion techniques employed to integrate this information? 3. What specific metrics beyond accuracy, such as F1-score, precision, and recall, were used to evaluate LLM performance, especially in cases with multiple viable diagnoses? |
| Leveraging Locality to Boost Sample Efficiency in Robotic Manipulation (Read more on arXiv or HuggingFace) | Yang Gao, Jiacheng You, Yingdong Hu, Tong Zhang | a) This research aims to improve sample efficiency in robotic manipulation by leveraging the inductive bias of action locality, which posits that robot actions are primarily influenced by the target object and its local environment. b) The authors introduce SGRv2, an imitation learning framework built upon the Semantic-Geometric Representation (SGR) that incorporates action locality through an encoder-decoder architecture, relative target position prediction, point-wise weighting, and dense supervision. c) SGRv2 achieves a 53.2% average success rate on 26 RLBench tasks using only 5 demonstrations, outperforming the RVT baseline on 23 of these tasks and demonstrating improved sample efficiency. d) AI practitioners can utilize the principles of action locality and the SGRv2 framework to develop more sample-efficient robotic manipulation models, reducing the reliance on large demonstration datasets which are costly to acquire. The most impactful finding is the significant improvement in sample efficiency, directly addressing the practical challenge of limited real-world robotic data. Follow-up questions: 1. How does the computational cost of SGRv2 compare to other methods like RVT and PerAct, especially considering the use of point-wise predictions and weighted averaging? 2. Could the concept of action locality and the techniques employed in SGRv2 be generalized to other robotic tasks beyond manipulation, such as navigation or multi-agent scenarios? 3. While the paper demonstrates robustness to visual distractors, how robust is SGRv2 to variations in the physical properties of the environment, such as changes in friction or object weight? |
Papers for 2024-10-28
| Title | Authors | Summary |
|---|---|---|
| ROCKET-1: Master Open-World Interaction with Visual-Temporal Context Prompting (Read more on arXiv or HuggingFace) | Xiaojian Ma, Zhancun Mu, Zihao Wang, kevinLian, phython96 | This research aims to improve embodied decision-making of vision-language models (VLMs) in open-world environments. The authors introduce “visual-temporal context prompting,” a communication protocol where VLMs provide object segmentations and interaction types to a low-level policy (ROCKET-1), which then predicts actions. In Minecraft experiments, ROCKET-1 combined with a Molmo 72B reasoner achieved a 91% success rate on the “place oak door on the diamond block” task, outperforming language- and image-based prompting baselines. This suggests that visual-temporal context prompting is an effective way to leverage the spatial reasoning capabilities of VLMs for embodied AI tasks. The paper lacks specific details about the training dataset size and composition beyond mentioning using OpenAI’s Contractor dataset. Follow-up questions: 1. What are the specific architectural details and hyperparameters of the causal transformer used in ROCKET-1, and how were these parameters tuned? 2. How robust is the system to noisy or incomplete segmentation masks, and what strategies could be employed to mitigate the impact of such imperfections during real-world deployment? 3. Beyond Minecraft, how generalizable is the visual-temporal prompting approach to other embodied AI tasks and environments, particularly those with continuous action spaces? |
| Continuous Speech Synthesis using per-token Latent Diffusion (Read more on arXiv or HuggingFace) | Hagai Aronowitz, Slava Shechtman, Arnon Turetzky, Avihu, NimrodShabtay1986 | a) This research investigates whether continuous representations, modeled with per-token latent diffusion, can be effectively used for zero-shot text-to-speech (TTS) synthesis, as opposed to the prevalent discrete, quantization-based approaches. b) The authors introduce SALAD, a per-token latent diffusion model incorporating a transformer architecture and semantic tokens. They evaluate three SALAD variants (Text2Acoustic, Semantic2Acoustic Autoregressive, Semantic2Acoustic Non-Autoregressive), along with corresponding discrete baseline models using RVQ. c) SALAD’s Text2Acoustic (T2A) continuous model achieved the lowest character error rate (CER) of 0.739% on the LibriSpeech test-clean dataset, suggesting superior intelligibility. Subjective listening tests showed comparable quality and speaker similarity to ground truth for several models. d) AI practitioners working on TTS systems may consider exploring continuous latent diffusion models like SALAD, particularly for applications prioritizing intelligibility. The findings suggest competitive performance with existing discrete methods and the potential for improved performance in certain aspects. Follow-up questions: 1. What is the computational cost difference between the continuous diffusion approach and the discrete RVQ-based methods, both during training and inference? This would be crucial for practical deployment considerations. 2. How sensitive is SALAD’s performance to the choice of VAE architecture and bottleneck dimension? Exploring the trade-off between reconstruction quality and generation performance would be beneficial. 3. Could the authors elaborate on the limitations of using likelihood or confidence measures with the diffusion approach, and potential alternative solutions for decoding strategies beyond random token unmasking in the NAR model? This could open avenues for further optimization. |
| Infinity-MM: Scaling Multimodal Performance with Large-Scale and High-Quality Instruction Data (Read more on arXiv or HuggingFace) | Jialing Zhang, Shuhao Gu, ZacLiu, bowen92, ldwang | a) The research aimed to improve the performance of open-source vision-language models (VLMs) by addressing the limitations of existing instruction datasets in terms of scale and quality. b) The researchers constructed a 40-million-sample multimodal instruction dataset, Infinity-MM, from existing open-source datasets and synthetic data generated using open-source VLMs, along with rigorous quality filtering and deduplication. They then trained a 2-billion parameter VLM, Aquila-VL-2B, using a curriculum learning approach. c) Aquila-VL-2B achieved state-of-the-art performance among similar-sized models, scoring 54.9 on MMStar, a benchmark for multimodal understanding. An ablation study confirmed the positive impact of the synthetic data on model performance. d) AI practitioners can leverage large-scale, high-quality instruction datasets like Infinity-MM and synthetic data generation methods to improve the performance of open-source VLMs, potentially reducing reliance on closed-source models or proprietary data. Follow-up questions: 1. The paper mentions a “mapping rules” technique used in question generation based on image tags and instruction tags. What are the specific details of these mapping rules, and how were they established and validated? 2. The data scaling experiment shows performance improvement with increasing dataset size, but plateaus toward the end. What are the computational and data resource requirements for training with datasets larger than those tested, and what further performance gains might be expected? 3. How does the performance of Aquila-VL-2B compare to closed-source SOTA models on the same benchmarks, and what specific areas of improvement would be needed to close any remaining performance gap? |
| Teach Multimodal LLMs to Comprehend Electrocardiographic Images (Read more on arXiv or HuggingFace) | Ping Zhang, Xiang Yue, Yuelin Bai, Ruoqi Liu | a) This research investigates the capability of Multimodal Large Language Models (MLLMs) to interpret electrocardiographic (ECG) images for automated cardiac assessment. b) The authors developed PULSE, an MLLM fine-tuned on ECGInstruct, a novel dataset of over one million ECG image-text pairs, and evaluated it on ECGBench, a new benchmark encompassing four ECG interpretation tasks across nine datasets. c) PULSE achieved state-of-the-art performance, outperforming proprietary MLLMs like GPT-40 by 15% to 30% average accuracy improvement on out-of-domain datasets. d) AI practitioners can leverage PULSE and ECGInstruct for developing more robust and generalizable ECG image interpretation models, potentially enhancing clinical practice. The paper’s most impactful finding is the significant performance improvement of the specialized PULSE MLLM over existing general-purpose MLLMs, demonstrating the potential of fine-tuning for domain-specific medical image analysis. Follow-up questions: 1. What specific vision encoder architecture and pre-training dataset were used for the PULSE model, and how did these choices impact performance compared to other open-source vision encoders? 2. Could the authors elaborate on the distribution of ECG abnormalities within the ECGInstruct dataset, and how this distribution compares to real-world clinical prevalence? Specifically, was the dataset assessed for class imbalance, and if so, what techniques were used to address it? 3. The paper mentions challenges with report generation and multi-turn conversations. What specific strategies, beyond increased data, might be explored to further improve PULSE’s performance on these more complex tasks, such as incorporating reinforcement learning from human feedback? |
| FasterCache: Training-Free Video Diffusion Model Acceleration with High Quality (Read more on arXiv or HuggingFace) | Yu Qiao, Zhenyu Yang, Junhao Song, Chenyang Si, Zhengyao Lv | a) The paper investigates accelerating video diffusion model inference while maintaining high-quality generation without requiring retraining. b) FasterCache, a training-free strategy, dynamically reuses features from attention modules and introduces CFG-Cache to leverage redundancy between conditional and unconditional outputs of classifier-free guidance (CFG). c) On Vchitect-2.0, FasterCache achieves a 1.67× speedup with a comparable VBench score (80.84%) to the baseline (80.80%). d) AI practitioners can use FasterCache to significantly reduce the computational cost of video diffusion models, making them more practical for real-time or resource-constrained applications. The dynamic feature reuse and CFG-Cache components offer readily implementable optimizations for existing and future video diffusion models. Follow-up questions: 1. What are the memory implications of FasterCache, especially regarding the feature cache for dynamic feature reuse and CFG-Cache? 2. How does the performance of FasterCache scale with higher-resolution videos beyond those tested in the paper, and what adjustments to the hyperparameters might be necessary? 3. Does FasterCache impact the diversity of generated videos? |
| MMAU: A Massive Multi-Task Audio Understanding and Reasoning Benchmark (Read more on arXiv or HuggingFace) | Ramaneswaran Selvakumar, Ashish Seth, Sonal Kumar, Utkarsh Tyagi, S Sakshi | MMAU aims to evaluate advanced audio perception and reasoning in AI models. The benchmark uses 10,000 audio clips paired with multiple-choice questions spanning speech, sound, and music, requiring models to demonstrate 27 distinct skills. Evaluation of 18 large audio-language models (LALMs) revealed that even the best-performing model achieved only 53% accuracy, significantly below human performance (82%). Analysis showed that models struggled most with perceptual understanding of audio. The key implication for AI practitioners is the need for significant improvements in audio perception and reasoning capabilities of LALMs to achieve human-level performance in complex audio tasks. Follow-up questions: 1. What specific architectural changes or training strategies could be explored to address the identified perceptual limitations in LALMs? 2. How can the MMAU benchmark be expanded to include more open-ended tasks that better reflect real-world audio understanding scenarios? 3. What are the potential downstream applications of improved LALM performance on the MMAU benchmark, specifically in areas like human-computer interaction and audio content analysis? |
| Counting Ability of Large Language Models and Impact of Tokenization (Read more on arXiv or HuggingFace) | Chenyu You, Juntai Cao, Wyattz23 | a) This research investigates how tokenization choices impact the counting ability of large language models (LLMs). b) The study uses a model-agnostic approach, manipulating input string formats to control tokenization in both open and closed-source LLMs (GPT-40-mini, Claude-3.5-sonnet) and evaluates their performance on letter-counting tasks with and without Chain-of-Thought (CoT) prompting. c) With CoT, using clearly separated target letter tokenization (via delimiters) increased GPT-40-mini’s counting accuracy by up to 80% compared to standard Byte Pair Encoding (BPE) tokenization of consecutive characters. d) LLM developers should carefully consider tokenization strategies, particularly moving beyond BPE tokenization of consecutive characters when precise reasoning or counting tasks are required. The demonstrated impact of tokenization highlights its often-overlooked role in realizing the theoretical reasoning capabilities of LLMs. Follow-up questions: 1. How does the performance improvement from delimiter-based tokenization scale with increasingly large input strings and more complex counting scenarios beyond single letter counts? 2. Given the observed impact, what specific tokenization algorithms or modifications to existing methods could be explored to further enhance LLMs’ reasoning abilities in practical applications? 3. Does the impact of tokenization on counting ability generalize to other, non-English languages, and if so, are there language-specific tokenization strategies that could be particularly beneficial? |
| Fictitious Synthetic Data Can Improve LLM Factuality via Prerequisite Learning (Read more on arXiv or HuggingFace) | Yang Zhang, Tommi Jaakkola, code-terminator, yujianll | PREREQ-TUNE, a novel fine-tuning strategy, aims to reduce LLM hallucinations by disentangling knowledge and skill acquisition. The method introduces a prerequisite learning stage to teach an LLM task-relevant knowledge via a knowledge LoRA, followed by supervised fine-tuning (SFT) to train a skill LoRA focused solely on task performance. Experiments on biography generation, medical question answering, and short question answering demonstrated that PREREQ-TUNE, trained with fictitious synthetic data, outperformed baselines, improving factuality (achieving 74.35% accuracy on medical QA). Results also confirmed PREREQ-TUNE’s disentanglement capabilities, preventing knowledge pollution. Follow-up questions: 1. How does the performance of PREREQ-TUNE compare to other methods when scaling the size of real training data, rather than synthetic data? 2. Could the knowledge LoRA approach be adapted for real-time knowledge retrieval within a RAG framework, and what are the potential latency implications? 3. What are the practical considerations for implementing the “unfamiliar knowledge” and “verbalized uncertainty” extensions in production systems? |
| Hybrid Preferences: Learning to Route Instances for Human vs. AI Feedback (Read more on arXiv or HuggingFace) | Valentina Pyatkin, Sachin Kumar, Yanai Elazar, Yizhong Wang, ljvmiranda921 | a) The research investigates how to combine human and large language model (LLM) generated preference annotations to maximize the performance of reward models in reinforcement learning from human feedback (RLHF), aiming for more efficient and accurate preference data collection. b) The proposed routing framework involves a performance prediction model (PPM) trained on MULTIPREF, a new dataset with human and LLM preference labels, to predict a reward model’s performance based on the proportion of human-annotated instances. A routing strategy then selects a combination of human and LLM annotations that maximizes the PPM’s predicted performance. c) Reward models trained on the hybrid datasets generated by the routing framework achieved a 7-13% absolute improvement on RewardBench compared to using either 100% human or 100% synthetic preferences. d) The study suggests that AI practitioners can optimize preference data collection by strategically routing instances to human annotators or LLMs, reducing annotation costs while improving the quality of trained reward models. The most impactful finding is that a hybrid approach, rather than relying solely on humans or LLMs, can substantially improve reward model performance. Follow-up questions: 1. How does the performance of the routing framework and the resulting hybrid preferences vary with different LLMs used for both synthetic preference generation and as the base reward model? 2. Could the features used in the PPM be expanded to incorporate characteristics beyond text similarity and prompt metadata, such as user demographics or task difficulty, to further personalize the routing strategy? 3. What are the practical implications for integrating this routing framework into existing RLHF pipelines, specifically addressing the challenges of real-time routing and the potential for feedback loops between the PPM, reward model, and policy model? |
| Leveraging Skills from Unlabeled Prior Data for Efficient Online Exploration (Read more on arXiv or HuggingFace) | Sergey Levine, Kevin Frans, Qiyang Li, Max Wilcoxson | a) This research investigates how unlabeled prior trajectory data can be used to learn efficient exploration strategies in reinforcement learning (RL). b) The proposed method, SUPE (Skills from Unlabeled Prior data for Exploration), extracts low-level skills from unlabeled trajectories using a variational autoencoder (VAE) and then uses an optimistic reward model to pseudo-label the trajectories for training a high-level off-policy RL agent to compose these skills. c) SUPE outperforms baseline methods on a suite of long-horizon, sparse-reward tasks, achieving an average success rate of 25% after 300,000 environment steps on the antmaze-ultra task, compared to 17% for the next-best method. d) AI practitioners can leverage unlabeled prior trajectory data to improve sample efficiency in online reinforcement learning, particularly in challenging exploration settings. This allows quicker learning and potentially higher asymptotic performance compared to methods that do not leverage such prior data effectively. Follow-up questions: 1. The paper mentions potential instability of the KL penalty objective, particularly in the Kitchen domain. Could the authors elaborate on the specific nature of this instability and potential mitigation strategies beyond switching to the tanh policy parameterization? 2. While the paper demonstrates the benefits of SUPE on several benchmark tasks, what are the limitations of this approach regarding the types of environments or tasks where it might be less effective? For instance, how would SUPE perform in environments with highly stochastic transitions or where the prior data is significantly mismatched with the target task? 3. How sensitive is SUPE’s performance to the quality of the learned low-level skills? Are there specific metrics or analyses that could be used to assess the quality of these skills and their impact on the overall performance of the online learning phase? |
| Dynamic 3D Gaussian Tracking for Graph-Based Neural Dynamics Modeling (Read more on arXiv or HuggingFace) | Yunzhu Li, Kaifeng Zhang, MingtongZ | This research aims to learn object dynamics directly from multi-view RGB videos for action-conditioned video prediction and model-based planning. The methodology involves using a modified Dynamic 3D Gaussian Splatting (Dyn3DGS) method for dense object tracking, followed by training a graph neural network (GNN) on sparse control particles to predict object motions under robot actions. The proposed method achieves a Median Trajectory Error (MTE) of 6.90mm for ropes, 13.14mm for cloth, and 12.83mm for toy animals in 3D tracking, outperforming 2D and depth-based baselines. This implies AI practitioners can leverage this framework to develop more accurate and robust 3D dynamics models directly from video data, enabling applications like robotic manipulation and video prediction in 3D. The paper does not detail the architecture of the GNN used, which leaves a key methodological aspect unclear. Follow-up questions: 1. What specific GNN architecture was used for the dynamics model, and how were its hyperparameters tuned? Details on the GNN’s design and training process would be valuable for replication and comparison to other architectures. 2. How does the computational cost of the proposed method scale with the number of Gaussians and the complexity of the object? This is critical for evaluating the feasibility of real-time applications. 3. How robust is the dense motion interpolation scheme to significant variations in Gaussian scale or distribution during object deformation, and how does this impact rendering quality? Further details regarding the robustness to changes in Gaussian representation would be beneficial. |
| Reflection-Bench: probing AI intelligence with reflection (Read more on arXiv or HuggingFace) | Yan Teng, Shuqi Kong, Haiquan Zhao, Yixu Wang, LingyuLi | a) This research aims to evaluate the reflection capabilities of Large Language Models (LLMs), defined as the ability to adapt beliefs or behaviors based on unexpected outcomes. b) The authors introduce Reflection-Bench, a benchmark comprising seven tasks adapted from cognitive science paradigms, including probabilistic reversal learning, Wisconsin card sorting test, and a meta-bandit task. c) Evaluation of 13 LLMs revealed varying performance levels, with o1-preview achieving the highest overall score, while all models scored zero on the meta-bandit task, indicating a lack of meta-reflection ability. d) AI practitioners should consider incorporating reflection-based benchmarks like Reflection-Bench to evaluate and enhance the adaptability and learning capabilities of LLMs, particularly for real-world applications requiring dynamic decision-making. Follow-up Questions: 1. Given the observed limitations of Chain-of-Thought (CoT) in the oddball paradigm and its high computational cost, what alternative strategies could be explored to improve LLMs’ automatic surprise detection without compromising performance in other reflection tasks? 2. How can the insights from the universal failure of LLMs on the meta-bandit task be leveraged to develop specific training methodologies or architectural modifications that foster meta-reflection capabilities? 3. Beyond accuracy, what other metrics could be introduced into Reflection-Bench to provide a more granular assessment of the internal processes underlying LLMs’ reflection abilities, such as information processing and belief updating strategies? |
Papers for 2024-10-25
| Title | Authors | Summary |
|---|---|---|
| Breaking the Memory Barrier: Near Infinite Batch Size Scaling for Contrastive Loss (Read more on arXiv or HuggingFace) | Kehan Li, Hang Zhang, LidongBing, Zhiqiang007, ClownRat | a) This research addresses the quadratic growth of GPU memory consumption when scaling batch sizes for contrastive loss, which limits performance gains. b) The paper proposes Inf-CL, a tile-based computation strategy that partitions the contrastive loss calculation, avoiding full materialization of the similarity matrix and leveraging a multi-level tiling approach across GPUs and CUDA cores. c) Inf-CL enabled training a ViT-L/14 CLIP model with a batch size of 12M on 32 A800 80GB GPUs using only 1.44GB of memory per GPU. d) AI practitioners can leverage Inf-CL to scale contrastive learning batch sizes to significantly larger values than previously possible, potentially improving model performance without incurring substantial memory overhead or significant speed reduction. Follow-up questions: 1. The paper mentions that excessively large batch sizes resulted in suboptimal performance in some cases. What specific hyperparameter tuning strategies are recommended when scaling to these very large batch sizes enabled by Inf-CL? 2. How does the performance of Inf-CL in other contrastive learning tasks (e.g., self-supervised learning, dense text retrieval) compare to its performance in image-text retrieval, and are there task-specific adaptations or optimizations needed? |
| LOGO – Long cOntext aliGnment via efficient preference Optimization (Read more on arXiv or HuggingFace) | Min Zhang, Qiaoming Zhu, Zechen Sun, douvleplus, ZetangForward | a) This research aims to improve the generation capability of long-context models (LCMs) to address misaligned outputs like hallucinations and instruction unfollowing. b) The study introduces LOGO, a training strategy using reference-free preference optimization with a tailored data construction pipeline involving positional indices synthesis and automatic evaluation of chunk importance. It modifies the SimPO objective to incorporate multiple dis-preference examples and an SFT regularization term. c) The Llama-3-8B-LOGO model, trained with LOGO, outperforms GPT-3.5-Turbo on real-world long-context tasks from LongBench and approaches the performance of GPT-4, showing a 5-point average improvement over the baseline Llama-3-8B-Instruct-80K. d) AI practitioners can use LOGO to fine-tune LCMs for improved generation performance in long-context tasks with reduced computational resources, potentially allowing for efficient context window scaling. Follow-up questions: 1. The paper mentions a lack of suitable evaluation models for detecting hallucinations. What specific evaluations beyond NIAH and LongBench would provide more robust insights into the reduction of hallucinations with LOGO? 2. The paper mentions adjusting the weighting of dis-preference samples as future work. What are the potential benefits and drawbacks of weighting these samples differently, and how might this weighting be implemented in the LOGO objective function? 3. How does LOGO’s performance compare to other long-context alignment methods in terms of inference speed and memory usage, especially when dealing with extremely long contexts? |
| Unleashing Reasoning Capability of LLMs via Scalable Question Synthesis from Scratch (Read more on arXiv or HuggingFace) | Qiaoming Zhu, Xiaobo Liang, douvleplus, XinyuShi, dyyyyyyyy | This research aims to improve the reasoning capabilities of Large Language Models (LLMs) by developing a scalable and cost-effective data synthesis method. The key methodology, ScaleQuest, uses smaller open-source LLMs to generate math questions from scratch, followed by filtering and response generation using larger models and reward filtering. Fine-tuning Qwen2-Math-7B with the synthetic dataset resulted in a 73.4% accuracy on the MATH benchmark, matching GPT-4-Turbo’s performance. This implies that AI practitioners can utilize ScaleQuest to create large-scale, high-quality training data for LLMs, potentially reducing reliance on expensive proprietary models and datasets. The paper does not clearly specify the size of the final dataset used in the instruction tuning phase after filtering, which impacts the interpretability of the 1M figure. Follow-up questions: 1. What are the specific details of the filtering process (e.g., thresholds, filtering model sizes) and how were these parameters determined? 2. Could the authors provide more detail about the dataset size used in instruction tuning after filtering, as the paper mentions both 1M and seems to imply a smaller number in the filtering process description. How does performance vary with different dataset sizes generated by ScaleQuest? 3. How does ScaleQuest perform on other reasoning tasks beyond mathematics? What modifications, if any, would be required to apply this method to other domains? |
| Can Knowledge Editing Really Correct Hallucinations? (Read more on arXiv or HuggingFace) | kaishu666, apayani, XiongxiaoXu, canyuchen, BaixHuang | a) The paper investigates whether knowledge editing techniques effectively correct factual hallucinations in Large Language Models (LLMs). b) Researchers constructed HalluEditBench, a dataset of LLM-generated hallucinations spanning 9 domains and 26 topics, and evaluated seven knowledge editing techniques across five facets: Efficacy, Generalization, Portability, Locality, and Robustness. c) While some methods like ICE and GRACE achieved high Efficacy scores (e.g., over 60% on Llama2-7b and Mistral-v0.3-7B), none consistently outperformed others across all five facets, and some even negatively impacted performance in areas like Generalization. It was also observed that FT-M achieved only around 60% Efficacy on Llama2-7B and Mistral-v0.3-7B, despite near-perfect scores on existing datasets. d) AI practitioners should exercise caution when relying on existing knowledge editing evaluation datasets, as their results may not reflect real-world hallucination correction effectiveness. The domain and LLM-specific nature of performance highlights the need for tailored editing strategies. Follow-up questions: 1. Given the domain-specific performance variations, what strategies can be employed to improve the generalization of knowledge editing techniques across different domains? 2. What specific metrics or evaluation frameworks could better capture the holistic impact of knowledge editing, beyond simple accuracy on benchmark datasets, considering the trade-offs observed across Efficacy, Generalization, Portability, Locality, and Robustness? 3. How can the limitations of parameter-preserving methods like ICE and GRACE regarding robustness be addressed while maintaining their high efficacy in correcting hallucinations? |
| Unbounded: A Generative Infinite Game of Character Life Simulation (Read more on arXiv or HuggingFace) | flavoredquark, mohitbansal, davejacobs, NealWadhwa, yzli | This research introduces the concept of a generative infinite game, aiming to create a video game with open-ended mechanics and narrative generated by AI. The methodology combines a specialized distilled large language model (LLM) for real-time game logic and narrative generation with a novel dynamic regional image prompt Adapter (IP-Adapter) for consistent visual generation of characters and environments. Results show improved character and environment consistency compared to existing approaches, with the distilled LLM achieving a 0.264 improvement in CLIP-IC for character consistency over Story Diffusion. This implies that AI practitioners can leverage distilled LLMs and regional IP-Adapters to create more dynamic and consistent generative games, moving beyond the limitations of traditional hard-coded systems. The paper does not quantify latency or frame rate for the “real-time” claim. Follow-up questions: 1. What specific architectural details of the distilled LLM (beyond being based on Gemma-2B) contribute to its interactive speed, and how does its performance compare to larger LLMs in terms of both latency and resource consumption? 2. How does the dynamic mask in the regional IP-Adapter contribute to the balance between preserving character details and incorporating environment style, and are there any observed trade-offs or limitations? 3. Can the regional IP-Adapter be generalized to other generative tasks beyond character life simulation, such as generating objects in diverse scenes for synthetic data generation? |
| Framer: Interactive Frame Interpolation (Read more on arXiv or HuggingFace) | Wen Wang, BiaoGong, Azily, zkcys001, qiuyuu | a) The research aims to develop an interactive frame interpolation framework that allows users to customize transitions between two images using point trajectory control, while also offering an automated “autopilot” mode. b) Framer fine-tunes a pre-trained image-to-video diffusion model with additional last-frame conditioning and incorporates a point trajectory controlling branch. An “autopilot” mode uses bi-directional point-tracking to estimate and refine trajectories automatically. c) Framer outperforms existing video interpolation methods in user studies, achieving a 90.5% preference rate compared to other state-of-the-art methods, demonstrating enhanced user control and visual quality. d) AI practitioners can leverage Framer to create customized and high-quality video frame interpolations for applications like image morphing, slow-motion generation, and novel view synthesis, improving the controllability and creative potential of video editing and generation tasks. The paper does not clearly define the specifics of how “Framer with Co-Tracker” differs from Framer in training or testing, although it reports superior performance for “Framer with Co-Tracker”. Follow-up questions: 1. Could the bi-directional point tracking method used in “autopilot” mode be integrated into the interactive mode to provide users with suggested or refined trajectories, further enhancing the interactive experience? 2. How does the computational cost of Framer, particularly during inference with the diffusion model, compare to traditional frame interpolation techniques, and what are the implications for real-time applications? 3. What are the specific architectural details and training procedures of “Framer with Co-Tracker”, and how do these differences contribute to the reported performance gains? |
| Distill Visual Chart Reasoning Ability from LLMs to MLLMs (Read more on arXiv or HuggingFace) | zifeishan, cnxup, zh2001, WooooDyy, hewei2001 | a) This research aims to improve visual chart reasoning abilities in Multimodal Large Language Models (MLLMs). b) The authors propose Code-as-Intermediary Translation (CIT), synthesizing chart-plotting code and using LLMs to generate reasoning-intensive questions and answers, creating the REACHQA dataset. c) Fine-tuning LLaVA-Next-Llama3-8B on REACHQA resulted in a 34.8% average performance improvement across multiple benchmarks. d) AI practitioners can leverage CIT and synthetic datasets like REACHQA for cost-effective improvement of MLLMs’ reasoning capabilities, generalizing beyond chart-specific tasks to broader multimodal reasoning. Follow-up questions: 1. Could the CIT method be adapted to other visual domains beyond charts, and if so, what adaptations would be necessary? 2. How robust is the performance improvement from REACHQA across different MLLM architectures and sizes? 3. What are the limitations of using synthetic data for training, and how can these limitations be addressed in future research? |
| Why Does the Effective Context Length of LLMs Fall Short? (Read more on arXiv or HuggingFace) | Shansan Gong, Lei Li, Ming Zhong, Jun Zhang, Chenxin An | This research investigates why the effective context lengths of large language models (LLMs) often fall short of their trained lengths. The authors introduce ShifTed Rotray position embeddING (STRING), a training-free method that shifts well-trained position indices to overwrite less-frequently encountered ones during inference. On the Needle-in-a-Haystack (4-needle) benchmark, STRING improved the average score across seven LLMs by 18 points. This suggests under-trained long-range position indices hinder LLM performance, and leveraging frequently-encountered indices can improve long-context processing without further training. This provides AI practitioners with a readily implementable technique for enhancing the effective context utilization of existing LLMs. Here are some follow-up questions an AI practitioner might have: 1. How does the choice of the shift offset (S) and local window (W) in STRING affect performance across different LLM architectures and sizes? 2. Does STRING impact other aspects of LLM performance, such as inference speed or memory usage, and how does this trade-off with the observed gains in effective context length? 3. Could the insights about the left-skewed position frequency distribution inform improved training data generation strategies for LLMs to more effectively utilize the full context window during training itself? |
| Robust Watermarking Using Generative Priors Against Image Editing: From Benchmarking to Advances (Read more on arXiv or HuggingFace) | Adams Wai-Kin Kong, Zihan Zhou, Yuanzhi, devSulyvahn, LUSHILIN | a) The research aims to develop a robust, invisible watermarking method for images that can withstand various image editing techniques, including those powered by text-to-image models. b) The researchers introduce W-Bench, a benchmark for evaluating watermarking robustness against image editing, and propose VINE, a novel watermarking method that leverages blurring distortions as surrogate training attacks and adapts the SDXL-Turbo text-to-image model as a generative prior for the watermark encoder. c) VINE-Robust achieves a True Positive Rate of 99.66% at a 0.1% False Positive Rate against image regeneration and 86.86% against global editing with InstructPix2Pix, outperforming existing methods. d) AI practitioners developing image watermarking methods can utilize W-Bench to comprehensively evaluate robustness against a wider range of image editing techniques and consider incorporating generative priors and surrogate training attacks, as demonstrated by VINE, to enhance resilience. e) The paper does not fully clarify the performance limitations of VINE with Image-to-Video generation, observing low overall detection rates but not providing extensive analysis or solutions. Follow-up questions: 1. Given the computational cost of VINE, what optimization strategies could be explored to reduce inference time and GPU memory usage for real-time applications? 2. How does the choice of blurring distortions as surrogate attacks in VINE affect the robustness against specific image editing techniques not included in W-Bench, and how can this selection be tailored for different editing models? 3. Could the insights from the frequency analysis of image editing in W-Bench be applied to improve the robustness of other watermarking techniques beyond VINE, such as those based on different network architectures or embedding strategies? |
| Skywork-Reward: Bag of Tricks for Reward Modeling in LLMs (Read more on arXiv or HuggingFace) | Jujie He, Rui Yan, Jiacai Liu, zengliangcs, chrisliu298 | a) This research aims to enhance reward modeling in LLMs, focusing on data-centric techniques for curating high-quality preference datasets. b) The researchers curated the Skywork-Reward dataset (80K preference pairs) from existing public sources and trained discriminative reward models using the Bradley-Terry loss. c) The resulting Skywork-Reward-Gemma-2-27B model achieved state-of-the-art performance on RewardBench with an average score of 93.8 and a Chat Hard score of 91.4. d) This work demonstrates the importance of meticulous data selection and filtering for training effective reward models, suggesting that smaller, high-quality preference datasets can outperform larger, less curated ones. It shows that current best-in-class models can be improved significantly by focusing on dataset quality and selection and provides practical techniques for AI practitioners to improve LLM alignment through efficient reward modeling. Follow-up questions: 1. What specific filtering techniques were applied to the WildGuardMix dataset, and how did the two-stage filtering process contribute to the final performance? The paper mentions a two-stage process but doesn’t detail it. 2. While the paper mentions experimenting with maximizing the margin between chosen and rejected responses using alternative loss functions, it doesn’t provide details about the specific configurations used (e.g., margin values, hyperparameter settings for each loss). Providing this information would enable reproduction and further analysis. 3. The paper highlights potential contamination in several datasets, including their own. What steps were taken to verify the nature of these overlaps (true contamination vs. misaligned preferences), and what is the long-term plan for maintaining dataset integrity as new training data becomes available? |
| MotionCLR: Motion Generation and Training-free Editing via Understanding Attention Mechanisms (Read more on arXiv or HuggingFace) | Lei Zhang, Shunlin Lu, Xuan Ju, Wenxun Dai, Ling-Hao Chen | a) This research aims to develop a text-driven human motion generation model capable of interactive, fine-grained editing without retraining. b) The researchers introduce MotionCLR, a diffusion-based model with a novel CLR block incorporating convolution, self-attention, cross-attention, and feed-forward network layers. Cross-attention explicitly models word-level text-motion correspondence, while self-attention captures temporal coherence between motion frames. c) MotionCLR achieves comparable generation performance to state-of-the-art methods, with an R-Precision of 0.544 for text-motion matching (Top 1) on the HumanML3D dataset. It also supports novel editing capabilities like motion (de-)emphasizing, in-place replacement, and sequence shifting through attention map manipulation. d) AI practitioners can leverage MotionCLR’s attention mechanism analysis for more explainable and controllable motion generation, enabling interactive editing based on textual prompts or example motions without model retraining. The specific roles of cross- and self-attention elucidated by this work can inform the design and development of other multi-modal generative models. Follow-up questions: 1. What are the computational resource requirements (memory, processing power) for running MotionCLR inference, specifically for real-time editing applications? 2. How does the performance of the in-place motion replacement operation scale with the length and complexity of the motion sequences being edited? 3. What specific strategies were used to mitigate the potential instability of manipulating attention maps, particularly when applying large weights for motion (de-)emphasis, and are there any limitations to the range of editable weights? |
| Should We Really Edit Language Models? On the Evaluation of Edited Language Models (Read more on arXiv or HuggingFace) | Zeyu Li, Peijie Dong, Zhenheng Tang, Qi Li, Dominic789654 | a) The paper investigates how sequential model editing affects the general abilities of large language models (LLMs). b) Multiple LLMs were edited with various methods (ROME, MEMIT, PMET, MEND, KN, GRACE, SERAC) and evaluated on benchmarks assessing world knowledge, arithmetic, commonsense reasoning, reading comprehension, and safety. c) After 10 edits on Llama2-7B using the KN method, the model failed to generate coherent, human-like text, demonstrating a “muting effect”; other methods preserved functionality at this level, though many showed performance degradation at higher edit counts. d) Current LLM editing methods are only suitable for small-scale knowledge updates (generally fewer than a few dozen), as larger-scale edits can disrupt intrinsic knowledge structures and compromise safety, even in aligned models. Follow-up questions: 1. Given the observed “muting effect” and performance degradation with increasing edits, what specific modifications to existing editing algorithms could improve their scalability and minimize negative impact on general LLM capabilities? 2. Beyond the benchmarks used in this paper, how would sequential editing affect performance on specific downstream tasks like named entity recognition, question answering, and natural language inference? 3. What are the practical implications of the observed safety degradation in edited models for real-world deployments, and what mitigation strategies could be employed to address these safety concerns? |
| ADEM-VL: Adaptive and Embedded Fusion for Efficient Vision-Language Tuning (Read more on arXiv or HuggingFace) | Han Hu, Yong Luo, Li Shen, Jianyuan Guo, Zhiwei840 | a) Objective: To develop a more parameter- and computationally-efficient vision-language (VL) model fine-tuning framework for tasks like visual question answering and image captioning. b) Methodology: The ADEM-VL framework modifies cross-attention modules within pretrained LLMs by replacing parameterized similarity measurements with a parameter-free approach using SiLU activation. It also incorporates multiscale visual features using pooling and an adaptive fusion scheme that discards less relevant visual features based on attention scores. c) Results: On the ScienceQA dataset, ADEM-VL fine-tuned on LLaMA-13B achieved 94.55% average accuracy, outperforming existing methods by 0.77%. The paper also reports efficiency improvements in both training and inference times, but specific quantitative comparisons across all relevant baselines are not provided for these metrics. d) Implication for AI Practitioners: ADEM-VL offers a more efficient method for fine-tuning VL models, potentially reducing computational costs and resource requirements for training and deploying these models, specifically concerning memory and inference speed. Follow-Up Questions: 1. The paper mentions efficiency gains but lacks comprehensive speed comparison data across PEFT baselines. Could you elaborate on the inference speed improvement on ScienceQA compared to all mentioned baselines (LLaVA-LoRA, LaVIN, MemVP) using LLaMA-7B and 13B? 2. How does the adaptive fusion scheme’s performance vary across different datasets and tasks beyond ScienceQA and image captioning? Are there tasks where dynamically dropping features might be detrimental? 3. What are the memory footprint reduction during training compared to other parameter-efficient methods when using LLaMA-7B and LLaMA-13B? |
| CCI3.0-HQ: a large-scale Chinese dataset of high quality designed for pre-training large language models (Read more on arXiv or HuggingFace) | Xiaofeng Shi, Hanyu Zhao, Chengwei Wu, Bo-Wen Zhang, ldwang | This research aimed to create a high-quality Chinese dataset for pre-training large language models (LLMs). The researchers used a two-stage filtering pipeline, involving fundamental processing (e.g., safety filtering, deduplication) and high-quality processing using Qwen2-72B-instruct and a trained 0.5B classifier. A 0.5B LLM trained on CCI3.0-HQ achieved an average score of 0.395 on a mixed dataset evaluation (60% English, 10% code, 30% Chinese) and 0.350 on a purely Chinese dataset, outperforming models trained on comparable datasets like SkyPile and WanjuanV1. This provides AI practitioners with a new high-quality Chinese dataset, CCI3.0-HQ, for pre-training and benchmarking Chinese LLMs. Follow-up questions: 1. What is the specific data mixture used in the 100B token training set for the Chinese Dataset Experiment besides the named datasets (Wanjuan-v1, SkyPile, CCI3.0, and CCI3.0-HQ)? The paper mentions the inclusion of these datasets but does not specify the proportions or any additional data. 2. How does the performance of the CCI3.0-HQ classifier compare to other quality classifiers on specific categories of positive samples, such as news articles, scientific literature, or social media posts? This could inform selection based on downstream tasks. 3. What specific hardware resources (e.g., number of GPUs, type of GPUs, RAM) and how much time was required for training the 0.5B LLM model on 100B tokens with the different dataset compositions? This information would help other researchers estimate the computational resources required for similar experiments. |
| CAMEL-Bench: A Comprehensive Arabic LMM Benchmark (Read more on arXiv or HuggingFace) | Ines Riahi, Ali Alharthi, Omkar Thawakar, Sara Ghaboura, ahmedheakl | a) The research aimed to create a comprehensive benchmark for evaluating Arabic Large Multimodal Models (LMMs) across diverse domains. b) The researchers curated a dataset, CAMEL-Bench, with 29,036 questions across eight domains (e.g., multimodal understanding and reasoning, medical image understanding) and 38 sub-domains, using translated and manually verified data from various sources and GPT-40 generated questions. They then evaluated several closed and open-source LMMs using metrics including exact match accuracy, edit distance, and fuzzy evaluation. c) GPT-4o achieved the highest performance across most domains, with an accuracy of 73.57% on chart and diagram understanding tasks, highlighting the general superiority of closed-source models while also revealing that even the best-performing models struggle with Arabic multimodal data. d) AI practitioners developing or deploying LMMs for Arabic should consider CAMEL-Bench as a crucial evaluation tool, given the demonstrated need for substantial improvement in Arabic LMM performance across various tasks, even for leading closed-source models. The benchmark’s diverse domains highlight specific areas needing improvement. Follow-up questions: 1. What are the specific prompts used with GPT-40 to generate the multiple-choice questions for the dataset, and how could these prompts be refined to target specific aspects of Arabic linguistic understanding or cultural context? 2. Could the researchers provide more details on the “fuzzy evaluation” methodology employed with GPT-4o, specifically regarding the prompt design and parameters used for comparing predicted and ground-truth answers in context? How reproducible is this approach, and what are its limitations? |
| WAFFLE: Multi-Modal Model for Automated Front-End Development (Read more on arXiv or HuggingFace) | Lin Tan, Shangshu Qian, jiang719, shanchao | This research aims to improve automated front-end development by addressing challenges in translating UI design images to HTML code. The authors introduce WAFFLE, a fine-tuning pipeline utilizing structure-aware attention and contrastive learning on multi-modal large language models (MLLMs). On the WebSight-Test benchmark, WAFFLE achieved up to a 9.00 percentage point increase in HTML Match compared to standard fine-tuning methods. This suggests that WAFFLE improves the MLLM’s understanding of HTML structure and visual details in UI images, facilitating more accurate code generation. AI practitioners can leverage WAFFLE to improve the performance of UI-to-HTML generation models. Follow-up questions: 1. How does the performance of WAFFLE compare to existing UI-to-HTML generation methods on real-world, complex UI designs beyond the Design2Code dataset? 2. What are the computational resource requirements for training and deploying WAFFLE with different backbone MLLMs? 3. How does the choice of hyperparameters, such as the portion of attention heads using structure-aware attention and the contrastive learning weight (λ), impact performance and training stability across different datasets and MLLM architectures? |
| Language Models are Symbolic Learners in Arithmetic (Read more on arXiv or HuggingFace) | Hanjie Chen, Ruidi Chang, Roy Xie, Zhiqi Li, Chunyuan Deng | a) This research investigates whether large language models (LLMs) utilize partial products in arithmetic calculations or function as symbolic learners. b) The study employed fine-tuning experiments on open-source LLMs (Gemma-2-2B and Llama-3.1-8B) with diagnostic tasks related to four multiplication algorithms and various rule and format perturbations. c) LLMs showed improved identification of partial products after fine-tuning on multiplication (+17.45% for standard multiplication), but fine-tuning on partial products did not improve multiplication performance; instead, position-level accuracy followed a U-shaped curve, suggesting an easy-to-hard subgroup selection based on subgroup quality. d) The paper implies that AI practitioners should consider LLMs as symbolic pattern matchers rather than calculators, focusing on subgroup complexity and selection when designing or analyzing arithmetic tasks for LLMs. Follow-up Questions: 1. Could incorporating explicit subgroup identification and training during fine-tuning improve the performance of LLMs on arithmetic tasks, particularly for the more difficult middle digits? 2. How does the observed symbolic learning behavior in arithmetic tasks generalize to other symbolic reasoning domains, such as logical inference or program synthesis? 3. Given the U-shaped accuracy curve, what specific curriculum learning strategies or training data augmentations could be most effective for improving LLM performance on arithmetic tasks across all digit positions? |
| Stable Consistency Tuning: Understanding and Improving Consistency Models (Read more on arXiv or HuggingFace) | Hongsheng Li, Gsunshine, wangfuyun | a) The paper investigates the limitations of current consistency training/tuning methods for generative models, particularly training variance and discretization error, aiming to improve performance and convergence speed. b) The authors propose Stable Consistency Tuning (SCT), building on Easy Consistency Tuning (ECT), which incorporates a variance-reduced training target via the score identity, a smoother progressive training schedule, and edge-skipping multistep inference. c) SCT achieves improved FID scores, demonstrated by a 2-step FID of 1.55 on ImageNet-64, a new state-of-the-art result for consistency models. d) AI practitioners can utilize SCT to train consistency models more efficiently and achieve higher-quality image generation with fewer sampling steps compared to existing methods. The paper also demonstrates the effectiveness of classifier-free guidance for consistency models, which could be valuable for practitioners working on conditional generation tasks. Follow-up questions: 1. How does the computational cost of calculating the variance-reduced training target in SCT compare to the standard consistency training/tuning target, and how does this trade-off impact overall training time? 2. The paper mentions adapting the variance-reduced score estimation for text-to-image generation using CLIP similarity, but leaves this for future study. How feasible is this adaptation, and what are the potential challenges in estimating probabilities based on CLIP similarity for conditional text-to-image generation using SCT? 3. Could the edge-skipping multistep inference strategy be applied to other generative model architectures beyond consistency models, and if so, what modifications would be required? |
| Taipan: Efficient and Expressive State Space Language Models with Selective Attention (Read more on arXiv or HuggingFace) | Hanieh Deilamsalehy, Ruiyi Zhang, Thang M. Pham, Huy Huu Nguyen, chiennv | a) The research aimed to develop a language model that efficiently handles long sequences while maintaining strong performance in memory-intensive tasks like in-context retrieval. b) The authors introduced Taipan, a hybrid architecture combining Mamba-2 (a State Space Model) with Selective Attention Layers (SALs) that strategically apply attention to key tokens identified by a gating network, while other tokens bypass the attention mechanism. c) Taipan outperformed Transformer, Mamba-2, and Jamba baselines in zero-shot language modeling and in-context retrieval tasks across different scales (190M, 450M, and 1.3B parameters). The 1.3B parameter Taipan model achieved an average score of 53.3 across Winograd, PIQA, HellaSwag, ARC-easy, ARC-challenge, OpenbookQA, TruthfulQA, RACE, and BoolQ, exceeding other models at the same scale. d) Taipan offers AI practitioners a more efficient alternative to Transformers for long-context language modeling, particularly in applications requiring extensive in-context retrieval or handling complex long-range dependencies, while maintaining constant memory usage. The paper doesn’t explicitly detail how the gating network’s selection criteria impacts the overall computational efficiency, leaving some ambiguity on the balance achieved. Follow-Up Questions: 1. What are the specific criteria used by the gating network to select tokens for attention processing, and how can these criteria be tuned or adapted for different downstream tasks? 2. What is the computational complexity of the gating network itself, and how does it scale with increasing sequence length and model size? 3. Could the selective attention mechanism be adapted for other efficient architectures beyond Mamba-2, such as S4 or other SSM variants? |
| Value Residual Learning For Alleviating Attention Concentration In Transformers (Read more on arXiv or HuggingFace) | Zhenzhong Lan, Zhiyun Jiang, Tianyi Wu, Zcchill | This research addresses the problem of attention concentration in deep transformers, where attention increasingly focuses on fewer tokens with depth. The authors propose ResFormer, which adds a residual connection from the first layer’s value embeddings to subsequent layers before the attention operation. Results on a 20B SlimPajama dataset show ResFormer achieves lower training loss than vanilla Transformers, DenseFormer, and NeuTRENO, with a 3% average accuracy improvement on downstream zero-shot reasoning tasks for an 82M parameter model. A variant, SVFormer, shares the first layer’s value embeddings across all layers, reducing KV cache by nearly half and demonstrating competitive performance on longer sequence lengths. The primary implication for AI practitioners is that ResFormer and SVFormer offer ways to improve training and inference efficiency of deep transformers. Follow-up Questions: 1. How does the performance of ResFormer and SVFormer vary across different downstream tasks beyond commonsense reasoning, and in different modalities like vision? 2. What are the memory and speed trade-offs of using SVFormer compared to other KV-efficient methods like GQA and CLA in real-world deployment scenarios? 3. Could the “anchor” approach of updating shared values in SVFormer using intermediate layers be further optimized, and how would this impact performance and stability on extremely long sequences? |
| Multi-Draft Speculative Sampling: Canonical Architectures and Theoretical Limits (Read more on arXiv or HuggingFace) | Roland Memisevic, Arash Behboodi, Hassan Dbouk, Ashish Khisti, mamaj92 | a) This research investigates multi-draft speculative sampling for accelerating large language model (LLM) inference, aiming to maximize the probability of accepting proposed tokens from multiple draft models. b) The authors analyze the optimal token-level draft selection problem, proposing a two-step canonical architecture involving importance sampling followed by single-draft speculative sampling, and derive an analytical expression for the optimal acceptance probability with two identical drafts. c) Experiments using the OPT model on Dolly, XSum, and WMT datasets demonstrate that their importance sampling scheme consistently outperforms baseline multi-draft speculative sampling methods, achieving, for example, over 2.1 block efficiency in the Dolly task with two drafts at a temperature of 1.2. d) The paper suggests that using importance sampling followed by speculative sampling offers improved block efficiency and token rates for LLM inference compared to existing multi-draft methods. It remains unclear how the proposed successive selection scheme scales with the number of drafts (K > 2) beyond the brief description in Remark 4. Follow-up questions: 1. How does the computational overhead of the importance sampling step compare to the gains in block efficiency, especially for different draft model sizes and numbers of drafts? 2. Could the theoretical analysis for two drafts be extended or approximated for a greater number of drafts (K>2) to guide the design of more efficient selection schemes? 3. How robust is the proposed method to variations in draft model quality, and what strategies could be employed to mitigate performance degradation with less accurate draft models? |
Papers for 2024-10-24
| Title | Authors | Summary |
|---|---|---|
| MIA-DPO: Multi-Image Augmented Direct Preference Optimization For Large Vision-Language Models (Read more on arXiv or HuggingFace) | conghui, KennyUTC, yhcao, yuhangzang, ziyuliu | a) The research aims to improve the ability of Large Vision-Language Models (LVLMs) to understand and reason with multi-image inputs, addressing the issue of hallucinations in these scenarios. b) The authors introduce Multi-Image Augmented Direct Preference Optimization (MIA-DPO), which extends single-image datasets to multi-image contexts by incorporating unrelated images and uses attention values to select rejected responses for Direct Preference Optimization (DPO) training. c) MIA-DPO improved performance on five multi-image benchmarks, achieving an average boost of 3.0% on LLaVA-v1.5 and 4.3% on InternLM-XC2.5. d) MIA-DPO offers a cost-effective and scalable approach for aligning LVLMs with human preferences in multi-image contexts, without relying on manual annotations or expensive APIs. This allows AI practitioners to enhance the multi-image reasoning capabilities of LVLMs using existing single-image data. Follow-up Questions: 1. How does the performance of MIA-DPO vary across different LVLM architectures beyond LLaVA and InternLM, and what modifications might be needed for optimal application to other models? 2. What are the computational resource requirements of MIA-DPO compared to other preference optimization methods, particularly regarding the attention-based selection process? 3. Could the attention-aware selection mechanism be further refined by incorporating other metrics or heuristics to enhance its effectiveness in identifying and filtering hallucinatory responses? |
| WorldSimBench: Towards Video Generation Models as World Simulators (Read more on arXiv or HuggingFace) | XihuiLiu, JeremyYin, LIJUNLI, Zhoues, CoachXP | This research aims to evaluate video generation models as “World Simulators,” capable of generating actionable, embodied video. The authors propose WorldSimBench, a dual evaluation framework comprising Explicit Perceptual Evaluation (using a Human Preference Evaluator trained on a novel HF-Embodied dataset with human feedback) and Implicit Manipulative Evaluation (assessing video-action consistency in simulated environments). Results show the Human Preference Evaluator surpasses GPT-40 in alignment with human preferences, achieving 89.4% accuracy in Open-Ended Embodied Environments. This implies that using human feedback to train evaluators is more effective for assessing video quality in embodied scenarios than zero-shot GPT-40 evaluations. The key takeaway for AI practitioners is that while current video generation models show some promise in generating realistic and controllable video, they still struggle to consistently represent complex physical rules and embody actions, hindering their practical use as World Simulators. Follow-up questions: 1. How does the architecture of the Human Preference Evaluator compare to other video quality assessment models, and what are the trade-offs of using a fine-tuned VideoLLM approach? 2. Could the HF-Embodied dataset, with its fine-grained human feedback, be used to improve video generation models themselves, in addition to training evaluators? 3. What are the specific limitations of the chosen simulation environments (Minecraft, CARLA, CALVIN) and how might these limitations affect the generalizability of the benchmark results to real-world applications? |
| Scaling Diffusion Language Models via Adaptation from Autoregressive Models (Read more on arXiv or HuggingFace) | Jiacheng Ye, Yizhe Zhang, kiaia, shivamag99, Sansa | This research explores scaling diffusion language models (DLMs) by adapting pre-trained autoregressive language models (AR LMs). The authors introduce a continual pre-training approach involving attention mask annealing and a shift operation to bridge the gap between AR and diffusion modeling objectives. Their adapted DLMs, DiffuGPT and DiffuLLaMA (scaled up to 7B parameters), outperform prior DLMs on language modeling, reasoning, and infilling tasks, with DiffuGPT-S achieving 50.2% accuracy on GSM8K after fine-tuning. This implies that adapting existing AR LMs is a viable method for developing competitive DLMs. AI practitioners can utilize this adaptation method to build more efficient and effective DLMs for various tasks, particularly those requiring infilling and global reasoning, without extensive training from scratch. Follow-up questions: 1. What are the computational resource requirements and training times for adapting larger AR LMs (e.g., >10B parameters) into DLMs using this method? 2. How does the choice of pre-training corpus (e.g., FineWeb vs. SlimPajama) affect the performance of the adapted DLMs on specific downstream tasks? 3. Could incorporating other techniques from AR LMs, like reinforcement learning with human feedback, further enhance the performance of adapted DLMs, especially for tasks like instruction following and code generation? |
| Lightweight Neural App Control (Read more on arXiv or HuggingFace) | Jianye Hao, ShaoKun-HW, Fahren24, gpap, semitable | This research aims to develop a lightweight, efficient mobile phone control architecture for cross-app interaction. The proposed LiMAC architecture combines a small Action Transformer (AcT) with a fine-tuned vision-language model (VLM), processing screenshots, UI trees, and text instructions to generate actions. LiMAC achieved up to 19% higher action accuracy compared to fine-tuned VLMs and up to 42% higher accuracy than prompt engineering baselines on two mobile control datasets. This implies AI practitioners can develop more accurate and resource-efficient mobile app agents using a gated architecture approach rather than relying solely on large foundation models. The paper is unclear on the exact size (parameter count) of AcT. Follow-up questions: 1. What are the specific implementation details and computational requirements of deploying the AcT + VLM architecture on resource-constrained mobile devices? 2. How does the performance of LiMAC compare with other lightweight models or techniques specifically designed for on-device inference, beyond those mentioned in the paper? 3. Could the contrastive learning approach used for click target prediction be extended or generalized to other types of action specifications beyond UI element selection? |
| Scalable Ranked Preference Optimization for Text-to-Image Generation (Read more on arXiv or HuggingFace) | Sergey Tulyakov, Zeynep Akata, anilkagak2, hcoskun, shyamgopal | This research aims to develop a scalable and cost-effective method for aligning text-to-image (T2I) models with human preferences. The authors introduce a synthetically labeled preference dataset (Syn-Pic) created by ranking images generated from multiple T2I models using pre-trained reward models and a ranking-based preference optimization method (RankDPO) leveraging this dataset. Results on DPG-Bench show RankDPO improves the DSG score for SDXL from 74.65 to 79.26. This implies AI practitioners can efficiently fine-tune T2I models for improved prompt following and visual quality without expensive human annotation. The paper doesn’t explicitly compare the computational cost of RankDPO with other DPO methods, only with reward optimization methods. Follow-up questions: 1. How does the diversity of the T2I models used to generate Syn-Pic impact the performance of RankDPO on downstream tasks, and what is the optimal number or combination of models? 2. How robust is RankDPO to the choice of pre-trained reward models used for creating Syn-Pic, and does using a larger ensemble of reward models always lead to better performance? 3. How does the performance of RankDPO, in terms of both effectiveness and computational cost, compare to other DPO variants applied to text-to-image generation, when using the same evaluation metrics and datasets? |
| DynamicCity: Large-Scale LiDAR Generation from Dynamic Scenes (Read more on arXiv or HuggingFace) | Yu Qiao, Liang Pan, Haozhe Xie, Lingdong Kong, Hengwei Bian | a) The research aims to develop a framework for generating large-scale, dynamic 4D LiDAR scenes capturing the temporal evolution of environments. b) DynamicCity uses a Variational Autoencoder (VAE) to learn a compact 4D representation called HexPlane, and a Diffusion Transformer (DiT) to generate novel HexPlanes, which are then decoded into 4D LiDAR scenes. A novel Projection Module and Expansion & Squeeze Strategy are introduced for enhanced VAE performance, and a Padded Rollout Operation prepares HexPlane features for DiT training. c) DynamicCity outperforms existing methods on CarlaSC and Waymo datasets in 4D scene reconstruction and generation tasks. For example, on CarlaSC, DynamicCity achieved a 38.6% improvement in mean Intersection over Union (mIoU) for 4D scene reconstruction compared to OccSora when using 16 frames as input. d) AI practitioners, specifically those working in autonomous driving and robotics, can leverage DynamicCity to generate synthetic 4D LiDAR data for training and testing perception systems, supplementing or replacing expensive and time-consuming real-world data collection. The ability to generate diverse and dynamic scenes, including rare edge cases, can lead to the development of more robust and safe autonomous systems. Follow-up questions: 1. What are the computational requirements for training and deploying DynamicCity, and how scalable is it to even larger datasets and longer sequence lengths? 2. The paper mentions known limitations related to highly congested scenes. Could you elaborate on the specific challenges encountered and potential strategies for mitigating these issues in future work? 3. What is the impact of different choices for the diffusion scheduler on the quality and diversity of the generated 4D LiDAR scenes? |
| ARKit LabelMaker: A New Scale for Indoor 3D Scene Understanding (Read more on arXiv or HuggingFace) | Hermann Blum, Marc Pollefeys, Francis Engelmann, Silvan Weder, Guangda Ji | This research investigates whether large-scale pre-training with automatically generated labels benefits 3D semantic segmentation similar to language and image generation tasks. The authors generated ARKit LabelMaker, a large-scale, real-world 3D dataset with dense semantic annotations by supplementing the ARKitScenes dataset with automatically generated labels using an enhanced LabelMaker pipeline. Pre-training PointTransformerV3 on this dataset achieved 81.2% mean Intersection-over-Union (mIoU) on the ScanNet validation set, exceeding vanilla training (77.5% mIoU) and comparable to multi-dataset joint training. This indicates the value of large-scale, real-world data for 3D semantic segmentation, even with imperfect labels. AI practitioners can leverage this dataset and the improved LabelMakerV2 pipeline for pre-training and potentially improve performance on downstream 3D scene understanding tasks. Follow-up questions: 1. How does the performance of models pre-trained on ARKit LabelMaker compare to those pre-trained on synthetic datasets of similar or larger scale, specifically regarding generalization to diverse real-world scenarios? 2. The paper mentions limitations due to computational cost for certain parts of LabelMaker and missing pose data in some ARKitScenes. How significantly do these limitations impact the overall quality and usability of the generated dataset for pre-training? 3. What are the specific details of the enhancements made to the LabelMaker pipeline in LabelMakerV2, and how do these improvements contribute to the scalability and robustness of the automatic labeling process? |
| MedINST: Meta Dataset of Biomedical Instructions (Read more on arXiv or HuggingFace) | Zirui Song, Yu Yin, Zihan Zhang, Meng Fang, Wenhan Han | a) This research aimed to address the challenge of limited biomedical instruction datasets for training large language models (LLMs) by creating a comprehensive resource and benchmark. b) The researchers created MEDINST, a meta-dataset of 133 biomedical natural language processing (NLP) tasks and over 7 million training samples, and MEDINST32, a benchmark subset of 32 tasks with varying difficulty levels, to evaluate LLM generalization. Several LLMs, including LLaMA-3 variants, were fine-tuned on MEDINST and evaluated on MEDINST32. c) LLaMA-3 fine-tuned on MEDINST (LLaMA3-MI) outperformed GPT-40 on 25 out of 32 tasks in MEDINST32. d) This suggests that using a comprehensive instruction dataset like MEDINST for fine-tuning significantly improves the performance of LLMs on biomedical tasks, even surpassing specialized models like BioMistral, offering practitioners a powerful resource for developing robust biomedical LLMs. Follow-up questions: 1. What specific prompting strategies were used during the few-shot evaluation of baseline models and zero-shot evaluation of fine-tuned models, and how did these choices affect performance? 2. Given the observed performance degradation in summarization and event extraction with increased training data size, attributed to data imbalance, what data augmentation or balancing techniques could be explored to mitigate this issue and improve performance on these tasks? 3. Could the authors provide further details on the annotation process for the human-annotated instructions, including inter-annotator agreement and quality control measures, to ensure the consistency and reliability of the MEDINST dataset? |
| M-RewardBench: Evaluating Reward Models in Multilingual Settings (Read more on arXiv or HuggingFace) | Drishti Sharma, Rishabh Maheshwary, Lester James V. Miranda, shayekh, srishti-hf1110 | This research investigates the performance of reward models (RMs) in multilingual settings. The authors created M-REWARDBENCH, a multilingual dataset with 2.87k preference instances across 23 languages and tasks including chat, safety, reasoning, and translation. Evaluation of 25 RMs on M-REWARDBENCH revealed a performance gap between English and non-English languages, with an average drop of over 8% for Classifier and Implicit RMs compared to their performance on the English-centric RewardBench. Generative RMs exhibited the smallest average performance drop at 3%. This implies that AI practitioners should prioritize evaluating and potentially adapting RMs for diverse languages to ensure consistent performance across global user bases. Follow-up questions: 1. How does the performance gap observed in M-REWARDBENCH translate to downstream performance of policy models fine-tuned with these RMs in different languages? 2. The paper mentions filtering English-centric prompts. What specific criteria were used for this filtering, and how might these criteria be adapted for other languages beyond those in M-REWARDBENCH? 3. Beyond the linguistic dimensions explored, what other cultural factors might influence RM preferences, and how can these be incorporated into future multilingual benchmark development? |
| TP-Eval: Tap Multimodal LLMs’ Potential in Evaluation by Customizing Prompts (Read more on arXiv or HuggingFace) | Tianhua Li, Yuxuan Xie, kpzhang, wqshao126 | a) This paper investigates the problem of prompt sensitivity in Multimodal Large Language Model (MLLM) evaluation, where minor prompt variations can lead to significant performance fluctuations, and proposes a new evaluation framework to mitigate this. b) The proposed framework, TP-Eval, uses an automatic prompt customization method employing an optimizer-scorer architecture with GPT-40 mini as an optimizer and the evaluated MLLM as a scorer, iteratively generating and evaluating prompts based on accuracy and semantic similarity to the original prompt. Error introspection from incorrect responses is also incorporated into the optimization process. c) On the MMT-S benchmark (a subset of MMT-Bench), LLaVA-1.5-7B achieved a 25.1% average performance improvement across 32 tasks after prompt customization using TP-Eval. d) AI practitioners evaluating MLLMs should consider prompt customization techniques like TP-Eval to mitigate underestimation caused by prompt sensitivity and obtain a more accurate assessment of model capabilities. The impactful finding is the significant performance improvement achieved by tailoring prompts to individual MLLMs, suggesting current evaluation methods may not fully reveal models’ potential. Follow-up questions: 1. How does TP-Eval’s performance compare to other prompt engineering techniques, specifically those designed for few-shot scenarios in multimodal settings? 2. How does the computational cost of running TP-Eval’s prompt optimization process scale with the size of the evaluation dataset and the complexity of the MLLM? 3. What are the limitations of relying on GPT-40 mini as the optimizer, and how could these limitations affect the optimization results for different MLLMs? |
Papers for 2024-10-23
| Title | Authors | Summary |
|---|---|---|
| PyramidDrop: Accelerating Your Large Vision-Language Models via Pyramid Visual Redundancy Reduction (Read more on arXiv or HuggingFace) | lindahua, jiaqiwang-rex, conghui, yhcao, yuhangzang | a) This research investigates whether all image tokens are necessary for all layers in Large Vision-Language Models (LVLMs) and, if not, how to reduce redundancy for improved efficiency. b) The researchers conduct empirical studies on token dropping at different LVLM layers and propose PyramidDrop, a method that partitions the LLM into stages and drops a pre-defined ratio of image tokens at the end of each stage based on a lightweight similarity calculation. c) PyramidDrop achieves a 40% training time reduction and 55% inference FLOPs reduction for LLaVA-NeXT-7B across 15 Vision-Language tasks without significant performance loss. It also allows training with doubled input resolution at 70% of the original training cost. d) AI practitioners can use PyramidDrop to accelerate both training and inference of LVLMs, particularly for high-resolution image understanding, without substantial performance degradation. The plug-and-play nature of PyramidDrop for inference acceleration is particularly advantageous for deployment on resource-constrained devices. Follow-up questions: 1. How does the performance of PyramidDrop compare to other token reduction methods, such as those focusing on text token reduction, when applied in conjunction? 2. What is the sensitivity of PyramidDrop’s performance to the choice of the stage count (S) and drop ratio (λ), and are there automated methods for determining optimal values for different LVLMs and tasks? 3. What are the memory implications of using PyramidDrop during training, specifically in relation to the maximum batch size that can be accommodated? |
| SpectroMotion: Dynamic 3D Reconstruction of Specular Scenes (Read more on arXiv or HuggingFace) | Jie-Ying Lee, Yi-Ruei Liu, Cheng-De Fan, yulunliu, stevenchang | a) The research aims to improve dynamic 3D scene reconstruction, particularly for scenes with specular (reflective) surfaces, using 3D Gaussian Splatting (3DGS). b) SpectroMotion combines 3DGS with physically-based rendering (PBR), deformation fields, a residual correction technique for normal computation, a deformable environment map, and a coarse-to-fine training strategy. c) On the NeRF-DS dataset, SpectroMotion achieved an average PSNR of 25.22, outperforming other methods like Deformable 3DGS (PSNR: 20.84) and 4DGS (PSNR: 18.77) for novel view synthesis. d) AI practitioners working on 3D scene reconstruction, particularly in areas like robotics or augmented reality, can leverage SpectroMotion’s techniques to improve rendering quality and handle challenging specular reflections in dynamic scenes. The improved handling of dynamic specular reflections enables more realistic and accurate 3D models, which can enhance various AI applications. Follow-up questions: 1. How does the computational cost of SpectroMotion compare to other dynamic 3DGS methods, particularly during the training and rendering phases? 2. What are the limitations of the deformable environment map, and how might it be further improved to handle more complex lighting variations in dynamic scenes? 3. How robust is SpectroMotion to different types of motion, and are there specific types of motion or deformations where it performs poorly, such as fast-moving objects or drastic changes in shape? |
| Aligning Large Language Models via Self-Steering Optimization (Read more on arXiv or HuggingFace) | Jingren, xphan, luyaojie, keminglu, sanmusunrise | a) This research aims to develop an automated alignment method for Large Language Models (LLMs) that eliminates the need for manual preference annotation. b) The proposed method, Self-Steering Optimization (SSO), autonomously generates preference signals during iterative training based on predefined principles, maintaining signal accuracy by ensuring a consistent quality gap between chosen and rejected responses while keeping them near on-policy. c) SSO improved the AlpacaEval 2.0 length control win rate by approximately 8% on average for the Llama3.1-8B-SFT model compared to the base model over three training iterations. d) SSO offers a scalable approach for LLM alignment, reducing the reliance on expensive and potentially limiting human annotation, which could enable more efficient and effective development of aligned LLMs. e) The paper mentions using a weight function and self-steering loss but does not fully explain their specific mathematical formulations or how the principles are predefined. Follow-up questions: 1. What is the specific mathematical formulation of the weight function (W) and self-steering loss (G) used in SSO? How are these components integrated into the overall training objective? 2. How are the “predefined principles” selected or generated, and what is the complete set of principles used in the experiments? How can these principles be adapted or extended for different alignment tasks or domains? 3. Could the authors elaborate on the computational overhead introduced by SSO compared to standard alignment techniques like RLHF or DPO? |
| JMMMU: A Japanese Massive Multi-discipline Multimodal Understanding Benchmark for Culture-aware Evaluation (Read more on arXiv or HuggingFace) | Yuki Imajuku, gneubig, ku21fan, AtsuMiyai, shtapm | This research aims to evaluate Large Multimodal Models (LMMs) on expert-level tasks in Japanese, focusing on both culture-agnostic and culture-specific understanding. The authors developed JMMMU, a benchmark dataset comprising 1,320 questions and 1,118 images across 28 subjects, including translated culture-agnostic components from MMMU and newly created culture-specific content. Evaluation of 18 LMMs revealed a performance ceiling of 58.6% accuracy achieved by GPT-4, indicating substantial room for improvement. GPT-4 outperformed Claude 3.5 Sonnet by 15.7% on culture-specific tasks, despite similar performance on English benchmarks and translated Japanese questions, highlighting the importance of culturally contextualized evaluation. This discrepancy has significant implications for practitioners developing multilingual LMMs, indicating that relying solely on translated benchmarks could overestimate true multilingual capability and lead to biased development. Follow-up questions: 1. Could the authors provide further details on the specific types of questions and images within the culture-specific subset of JMMMU to guide targeted model improvements? 2. What are the specific metrics used to determine “expert-level” difficulty, and how were these levels calibrated within the JMMMU dataset? 3. The paper mentions Japanese LMMs exhibit robustness to translation effects; could the authors elaborate on the specific training datasets and techniques that contribute to this robustness? |
| EvoPress: Towards Optimal Dynamic Model Compression via Evolutionary Search (Read more on arXiv or HuggingFace) | dalistarh, ekurtic, SpiridonSunRotator, OliverSieberling | This paper investigates optimal dynamic compression of Large Language Models (LLMs) to minimize accuracy loss under a global compression constraint. The researchers developed EvoPress, an evolutionary search algorithm with level-switch mutation and multi-step selection, which has provable convergence and low sample complexity. EvoPress achieved state-of-the-art results across structural pruning, unstructured sparsity, and quantization with dynamic bitwidths; for example, it improved zero-shot average accuracy by 4.1 points on Llama-3-8B at 70% unstructured sparsity. This implies that AI practitioners can use EvoPress to significantly improve the accuracy-compression trade-off in compressed LLMs. The paper does not provide detailed information on the computational resources (e.g., GPU memory) required to run EvoPress on the tested models. Follow-up questions: 1. Could EvoPress be effectively applied to dynamic compression during the training of LLMs, and if so, how would the search process be integrated with the training loop? 2. What is the memory footprint of EvoPress when running on larger LLMs (e.g., 70B parameter models) for different compression tasks, and how could this be optimized? 3. How does the choice of calibration dataset affect the final compressed model quality obtained by EvoPress, and are there guidelines for selecting a suitable calibration dataset for a given task or domain? |
| MiniPLM: Knowledge Distillation for Pre-Training Language Models (Read more on arXiv or HuggingFace) | Minlie Huang, Jie Zhou, Hao Zhou, fandong, t1101675 | a) The research aimed to develop an efficient and flexible knowledge distillation (KD) framework for pre-training language models (LMs) that addresses the limitations of existing online and offline KD methods. b) MINIPLM utilizes Difference Sampling, an offline method that refines the pre-training corpus based on the probability discrepancies between a large teacher LM and a small reference LM. The student LM is then pre-trained from scratch on this refined corpus. c) MINIPLM improved the zero-shot performance of a 500M parameter student LM by 2.2x compared to vanilla KD while using the same training compute budget, as measured by average zero-shot accuracy across nine downstream tasks. d) AI practitioners can use MINIPLM to train smaller, more efficient student LMs that achieve competitive performance with larger models while reducing computational costs and potentially data requirements. The framework’s flexibility also facilitates KD across different model families. Follow-up questions: 1. How does the performance of MINIPLM vary with different sizes of reference LMs, and how can we optimally choose the reference LM size for a given teacher-student pair? 2. The paper mentions reducing data requirements in a data-limited setting. Can this be quantified more precisely with different dataset sizes, and what are the tradeoffs between dataset size and performance when using MINIPLM? 3. How does MINIPLM compare to other recent KD methods for pre-training, especially those focusing on data selection or curriculum learning, in terms of both performance and efficiency? |
| Mitigating Object Hallucination via Concentric Causal Attention (Read more on arXiv or HuggingFace) | Shijian Lu, Ivan Laptev, Yiheng Li, xing0047 | a) The paper investigates the correlation between Rotary Position Encoding (ROPE) and object hallucination in Large Vision Language Models (LVLMs), aiming to mitigate this hallucination. b) The authors propose Concentric Causal Attention (CCA), a positional alignment strategy involving visual token reorganization and a modified causal attention mask, to address ROPE’s long-term decay issue. c) On the POPE benchmark, CCA achieves an accuracy improvement of 5.48% on the COCO dataset with random negative sampling, compared to the baseline LLaVA model. d) AI practitioners working with LVLMs can use CCA during training to reduce object hallucination by improving visual-instructional token interaction and mitigating the negative effects of ROPE’s long-term decay. This translates to more factually accurate responses from LVLMs. Follow-up questions: 1. How does CCA’s computational cost during training and inference compare to the baseline LLaVA and other hallucination mitigation strategies like VCD? 2. The paper mentions CCA’s potential for broader improvements to LVLM perception. Can the authors elaborate on the types and magnitudes of improvements observed on other perception tasks beyond object hallucination? 3. Could the authors provide more detail on the specific implementation of the concentric position alignment and causal masking within a standard transformer architecture? |
| Math Neurosurgery: Isolating Language Models’ Math Reasoning Abilities Using Only Forward Passes (Read more on arXiv or HuggingFace) | Thomas Hartvigsen, Jonathan Kropko, Zack Gottesman, Bryan R. Christ | a) This research investigates how mathematical reasoning abilities are encoded within Large Language Models (LLMs) and whether math-specific parameters can be isolated. b) The researchers developed MathNeuro, a method utilizing forward passes and weight-activation products to identify parameters important for math reasoning, while excluding those important for general language tasks (tested using RACE and MMLU datasets). c) Pruning MathNeuro-identified parameters eliminates math performance (measured on GSM8K), while scaling these parameters by a small factor improves GSM8K performance by 4-17% across various model sizes (1B-8B parameters) without significantly affecting non-math performance. d) AI practitioners can use MathNeuro to target and modify specific LLM parameters to improve mathematical reasoning abilities without negatively impacting performance on other tasks. The demonstrated ability to boost math reasoning by 4-17% through a simple scaling intervention is impactful, offering a concrete method for enhancing LLM capabilities for math-intensive applications. Follow-up questions: 1. How does the computational cost of MathNeuro scale with increasing LLM size, and what are the practical implications for applying this method to very large models? 2. Can MathNeuro be adapted to isolate and enhance other specific reasoning abilities beyond mathematics, such as logical reasoning or causal inference? 3. How robust is the parameter identification in MathNeuro to the choice of non-math datasets used for comparison, and are there alternative datasets or tasks that might provide more effective isolation? |
Papers for 2024-10-22
| Title | Authors | Summary |
|---|---|---|
| CompassJudger-1: All-in-one Judge Model Helps Model Evaluation and Evolution (Read more on arXiv or HuggingFace) | Hongwei Liu, Maosong Cao, zsytony, KennyUTC, acylam | a) This research aims to develop an open-source, all-in-one judge LLM, CompassJudger-1, for robust and versatile subjective evaluation of LLMs, along with a dedicated benchmark, JudgerBench. b) CompassJudger-1 was trained using a mixture of publicly available judge data, self-collected subjective evaluation data, reward data, and general SFT data, employing balanced sampling and data categorization strategies. c) CompassJudger-1 achieved 95.9% correlation with GPT-4 on JudgerBench-B (Benchmark component focused on critique generation and format adherence). d) AI practitioners can leverage CompassJudger-1 as a cost-effective alternative to closed-source models like GPT-4 for evaluating subjective LLM performance across various benchmarks and tasks, facilitating more efficient and reproducible model evaluation and iterative refinement. e) The paper does not provide specific implementation details of the training process, such as the specific model architecture or hyperparameters used beyond a learning rate of 2e-5 and 2 epochs, making reproducibility challenging. Follow-up Questions: 1. What specific model architecture and hyperparameters were used to train CompassJudger-1, and what were the computational resources required? 2. How does CompassJudger-1’s performance compare to GPT-4 and other judge models on specific subjective evaluation tasks beyond overall correlation, considering metrics like helpfulness, honesty, and harmlessness? 3. How can CompassJudger-1 be fine-tuned or adapted for specific evaluation tasks or domains, and what resources or guidelines are available for practitioners to do so? |
| SAM2Long: Enhancing SAM 2 for Long Video Segmentation with a Training-Free Memory Tree (Read more on arXiv or HuggingFace) | lindahua, guoyww, yhcao, yuhangzang, Mar2Ding | a) The research aimed to improve the long-term video object segmentation performance of the Segment Anything Model 2 (SAM 2), particularly in scenarios with occlusions and object reappearances. b) The authors introduced SAM2Long, a training-free method utilizing a constrained tree memory structure to maintain multiple segmentation pathways and an object-aware memory bank selection strategy within each pathway. The method also incorporates uncertainty handling to promote hypothesis diversity. c) SAM2Long consistently outperformed SAM 2 across six video object segmentation benchmarks. On the SA-V test set, SAM2Long-L improved the J&F score by 5.3 points compared to SAM 2-L. d) AI practitioners can leverage SAM2Long to improve the robustness and accuracy of video object segmentation applications, especially in challenging long-term scenarios, without needing additional training or parameter adjustments. The significant performance gain with minimal computational overhead makes it readily applicable to real-world video analysis tasks. Follow-up questions: 1. How does the computational cost of SAM2Long scale with the length of the video and the number of pathways P, and what are the practical implications for real-time applications? 2. The paper mentions exploring semantic interactions between multiple objects as future work. What specific approaches could be investigated to incorporate multi-object relationships into the SAM2Long framework? 3. Could the memory tree structure and uncertainty handling strategies of SAM2Long be generalized and applied to other video understanding tasks beyond segmentation, such as object tracking or action recognition? |
| PUMA: Empowering Unified MLLM with Multi-granular Visual Generation (Read more on arXiv or HuggingFace) | hsli-cuhk, daijifeng, zengxingyu, gogoduan, LucasFang | a) This research aims to address the limitations of existing Multimodal Large Language Models (MLLMs) in balancing diversity and controllability for various visual generation tasks by introducing a multi-granular approach. b) PUMA (emPowering Unified MLLM with Multi-grAnular visual generation) utilizes a multi-scale image encoder, a set of dedicated diffusion-based image decoders, and an autoregressive MLLM trained with a two-stage process of pretraining and instruction tuning. c) PUMA achieves 18.16 PSNR and 0.2215 LPIPS on ImageNet validation set reconstruction using its finest granularity level (f0), outperforming existing methods like Emu2, SEED-LLaMA, and SEED-X in reconstruction quality. d) PUMA offers AI practitioners a unified framework for diverse visual tasks, including image understanding, generation, editing, and conditional generation, by effectively handling multiple levels of feature granularity within a single MLLM. The significant improvement in fine-grained image reconstruction enables more precise image manipulation within the MLLM framework. Follow-up Questions: 1. The paper mentions using pre-trained SDXL models as decoders and fine-tuning them. What specific modifications were made to the SDXL architecture to accommodate multi-granular features, and how does this impact computational cost compared to single-scale approaches? 2. While Table 5 shows improved understanding performance with finer-grained features, it doesn’t clarify how the different feature scales are combined or weighted when multiple scales are used as input. What is the specific input format for the MLLM when using all features f4-f0? 3. The paper highlights diverse text-to-image generation. How does PUMA control or guide the style and content of the generated image beyond basic textual prompts, and what mechanisms are used to ensure the generated images align with user intent, particularly when using coarser granularity levels? |
| Baichuan Alignment Technical Report (Read more on arXiv or HuggingFace) | dongguosheng, YijieZhou, TJU-Tianpengli, zilchshen, lin5547 | a) This report details Baichuan Alignment, a suite of techniques for aligning large language models (LLMs) with human intentions and values. b) Baichuan Alignment utilizes three phases: a Prompt Augmentation System (PAS), Supervised Fine-Tuning (SFT), and Preference Alignment, incorporating optimizations like sample packing, multi-layer gradient checkpointing, and model merging. c) After applying Baichuan Alignment, the LLM Qwen2-Nova-72B shows a 26% absolute increase in performance on the ArenaHard benchmark compared to its base model Qwen2-72B, demonstrating substantial gains in instruction following. d) AI practitioners can use the insights from Baichuan Alignment, such as prompt engineering automation and task-aware embedding for prompt diversity, to improve alignment in their own LLM development, potentially leading to significant performance gains in various downstream tasks. The report emphasizes the critical role of high-quality data and iterative evaluation in alignment, providing practitioners with practical methodologies for building more aligned and capable LLMs. Follow-up questions: 1. The report mentions using a KL-divergence based PTX loss during Reinforcement Learning with merged models. Could the authors elaborate on the specifics of this implementation and its effectiveness compared to using cross-entropy loss, particularly in the context of preventing model collapse to a SFT model? 2. While the report demonstrates strong benchmark results, how robust is Baichuan Alignment across different model architectures and sizes? Are there specific adjustments needed when applying these techniques to significantly smaller or larger LLMs? |
| AutoTrain: No-code training for state-of-the-art models (Read more on arXiv or HuggingFace) | abhishek | a) The paper introduces AutoTrain (AutoTrain Advanced), a no-code tool to simplify training and fine-tuning state-of-the-art models across diverse modalities and tasks. b) AutoTrain leverages existing libraries like Transformers, Datasets, and Accelerate and provides a command-line interface, graphical user interface, and Python SDK for model training on custom datasets. c) AutoTrain currently supports 22 tasks, including 16 text-based, 4 image-based, and 2 tabular-based tasks. d) AutoTrain simplifies model training and deployment for AI practitioners by automating tasks like hyperparameter tuning, data preprocessing, and distributed training, allowing them to focus on data preparation and model selection. Follow-up questions: 1. How does AutoTrain handle class imbalance and other common data quality issues that can affect model performance? 2. What specific metrics are used for evaluating models trained with AutoTrain for each of the supported tasks? 3. What are the computational resource requirements (CPU, RAM, GPU) for running AutoTrain locally versus on a cloud platform? |
| FrugalNeRF: Fast Convergence for Few-shot Novel View Synthesis without Learned Priors (Read more on arXiv or HuggingFace) | Shih-Han Yen, Chang-Han Yeh, yulunliu, kkennethwu, chinyanglin | a) The paper addresses the challenge of slow convergence and overfitting in few-shot novel view synthesis using Neural Radiance Fields (NeRFs). b) FrugalNeRF employs weight-sharing voxels across multiple scales and a cross-scale geometric adaptation scheme that selects pseudo ground truth depth based on reprojection errors, guiding training without external priors. c) On the LLFF dataset with two input views, FrugalNeRF achieves an average PSNR of 18.07, outperforming several existing methods while significantly reducing training time to 10 minutes. d) AI practitioners can use FrugalNeRF for efficient and accurate 3D scene reconstruction from limited images, bypassing the need for pre-trained models and complex scheduling. The paper’s focus on rapid training and robust voxel training makes FrugalNeRF a practical approach for resource-constrained settings. Follow-up questions: 1. How does the performance of FrugalNeRF degrade with increasing sparsity of input views, particularly below two views? 2. What are the specific computational and memory requirements for deploying FrugalNeRF in real-world applications, such as augmented reality or robotics? 3. Could the cross-scale geometric adaptation scheme be generalized to other NeRF architectures beyond the voxel-based approach used in FrugalNeRF? |
| RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Style (Read more on arXiv or HuggingFace) | Rui Min, Yantao Liu, juanli, Nuomei, TranSirius | a) This research aims to create a benchmark, RM-BENCH, for evaluating reward models’ ability to discern subtle content differences and resist stylistic biases, addressing limitations in existing benchmarks. b) RM-BENCH evaluates reward models across four domains (Chat, Code, Math, Safety) using responses generated by the same LLM (gpt-40) with controlled stylistic variations, assessing accuracy in distinguishing preferred responses. c) Even state-of-the-art reward models achieved only 46.6% accuracy on Hard Accuracy, falling below random chance (50%) under style bias interference, indicating susceptibility to stylistic biases rather than content quality. d) AI practitioners should prioritize mitigating style bias in reward model training as it significantly impacts reward model effectiveness and may mislead policy model training in reinforcement learning from human feedback (RLHF) and inference scaling law techniques. e) The correlation between RM-BENCH performance and aligned language model performance is shown, but the specifics of how this correlation was measured (e.g., metric used for policy model performance) are not fully detailed. Follow-up questions: 1. How does RM-BENCH compare to other existing reward model benchmarks in terms of correlation with downstream task performance on specific datasets beyond those mentioned (e.g., HellaSwag, SQuAD)? 2. What specific methods or techniques are recommended for mitigating the style bias observed in reward models during training, given the findings of RM-BENCH? 3. Could the authors elaborate on the construction details for the rejected responses in the Code & Math section? How were the “incorrect” responses guaranteed to be incorrect while still being plausible enough to pose a genuine challenge to the reward model? |
| Pangea: A Fully Open Multilingual Multimodal LLM for 39 Languages (Read more on arXiv or HuggingFace) | Nyandwi, seungone, akariasai, yueqis, yuexiang96 | a) This research aimed to develop a multilingual, multimodal large language model (MLLM) that addresses the underrepresentation of many languages and cultural contexts in current MLLMs. b) The researchers created PANGEA, trained on PANGEAINS, a 6-million sample multilingual multimodal instruction dataset spanning 39 languages, and evaluated it using PANGEABENCH, a novel evaluation suite encompassing 14 datasets in 47 languages. PANGEAINS was constructed by translating English instructions, generating culturally aware instructions, and curating existing open-source datasets. c) PANGEA-7B outperformed the best existing open-source MLLMs by 7.3 points on English tasks and 10.8 points on multilingual tasks in PANGEABENCH. d) This work provides AI practitioners with open-source data, code, and model checkpoints for developing more inclusive and robust multilingual MLLMs, highlighting the importance of scaling multilingual multimodal instruction tuning. e) The paper does not provide specifics on the architecture used for PANGEA beyond mentioning it is based on the LLaVA-Next architecture with Qwen2-7B-Instruct as the language backbone. Follow-up Questions: 1. What are the specific architectural details and hyperparameters used for PANGEA, including details on the visual encoder and the fusion mechanism with the language model? 2. How does the performance of PANGEA on specific language pairs within PANGEABENCH reflect linguistic similarities and differences, and how can this inform future dataset curation strategies? 3. What are the ethical considerations and potential biases related to using machine translation for constructing multilingual instruction datasets for multimodal LLMs? |
| Meta-Chunking: Learning Efficient Text Segmentation via Logical Perception (Read more on arXiv or HuggingFace) | Zhiyuan Ji, jimi888, siminniu, MoCun, Robot2050 | This paper investigates how to improve the efficiency and effectiveness of text chunking in retrieval-augmented generation (RAG) pipelines. The authors propose “Meta-Chunking,” which leverages LLMs with two strategies: Margin Sampling Chunking (binary classification of segmentation points based on probability differences) and Perplexity Chunking (identifying chunk boundaries based on perplexity distribution minima). Results on eleven datasets, including 2WikiMultihopQA, demonstrate that Meta-Chunking with Qwen2-1.5B outperforms similarity chunking by 1.32 F1 points while using only 45.8% of the processing time. This suggests that Meta-Chunking, especially Perplexity Chunking, offers a more efficient and potentially more accurate method for text segmentation in RAG, allowing practitioners to optimize resource allocation and potentially improve the quality of downstream tasks like question answering. Follow-up questions: 1. How does the performance of Meta-Chunking compare to LumberChunker on additional datasets beyond those mentioned in the paper, especially focusing on resource consumption and processing time differences? 2. Could the dynamic merging strategy of Meta-Chunking be further refined by incorporating semantic similarity metrics or other logical relationship classifiers to optimize chunk coherence beyond length constraints? 3. What are the practical limitations or challenges of implementing Meta-Chunking in a real-world RAG system, specifically concerning the computational overhead of integrating LLMs for chunking and potential failure modes in diverse textual contexts? |
| Pre-training Distillation for Large Language Models: A Design Space Exploration (Read more on arXiv or HuggingFace) | Xin Lv, juanli, NeoZ123, bys0318, Wesleythu | a) This paper explores the design space of pre-training distillation (PD) for Large Language Models (LLMs), investigating whether distilling knowledge during the pre-training phase is feasible and how to optimize it. b) The researchers systematically explored four dimensions of PD: logits processing (truncation, normalization), loss selection (KL divergence, MSE, NLL), scaling laws (model and corpus size), and offline vs. online logits generation. They conducted controlled experiments using GLM-4-9B as the teacher model and various smaller student LLMs. c) Pre-training distillation with a WSD scheduler for both the combination factor of language modeling and distillation loss (α), and learning rate (WSD-α + WSD-LR) resulted in an average performance improvement of 8.0% across multiple datasets compared to a baseline LLM trained only with language modeling loss. d) AI practitioners can leverage pre-training distillation, particularly with a WSD scheduling strategy, to improve the performance of student LLMs trained from scratch, potentially reducing training time and resources. e) The paper lacks clear explanation regarding the hardware used in the SFT stage and the specific datasets used for fine-tuning. The selection rationale for the chosen dataset sizes in the preliminary and scaling law experiments is not explicitly provided. Follow-up questions: 1. What are the computational cost savings of using pre-training distillation compared to training a student LLM from scratch without distillation, considering the overhead of logits generation and storage? 2. Could the authors elaborate on the hardware and data used in the Supervised Fine-tuning (SFT) stage, and how these choices might affect the generalizability of the results? 3. How does the performance of pre-training distillation change with varying dataset sizes, particularly exceeding the explored range, and how could practitioners determine the optimal dataset size for a given LLM size and available resources? |
| Alchemy: Amplifying Theorem-Proving Capability through Symbolic Mutation (Read more on arXiv or HuggingFace) | Ping Wei, opotle, yegong, shuailu, EurekaWu123 | This research aims to improve Neural Theorem Proving (NTP) by addressing data scarcity. The authors propose “Alchemy,” a framework that synthesizes new theorems in the Lean formal system by symbolically mutating existing theorems in Mathlib4 using the rw and apply tactics. This method increased the number of theorems by an order of magnitude, from 110,657 to 6,326,679. After pretraining and finetuning LLMs on this augmented data, a 5% absolute performance improvement was observed on the Leandojo novel_premises benchmark. This implies that synthetic data generation can enhance the theorem-proving ability and generalization of LLMs, offering a valuable resource for developers of automated theorem provers. Follow-up questions: 1. How does the performance of the theorem prover vary with different filtering strategies applied to the set of invocable theorems Tᵢ? Could more sophisticated filtering based on theorem complexity or relevance further improve data quality and downstream performance? 2. The paper mentions the computational cost of the synthesis process. What specific optimizations to Leandojo or the synthesis algorithm itself could be implemented to make this approach more scalable and efficient for larger datasets or more complex tactic combinations? 3. Could the proposed symbolic mutation approach be generalized to other formal systems besides Lean, and what adaptations would be necessary to accommodate different syntax and proof structures? |
| SemiEvol: Semi-supervised Fine-tuning for LLM Adaptation (Read more on arXiv or HuggingFace) | Wei Ju, Xiao Luo, Shockzipper, XtremSup, luojunyu | This research investigates how to adapt LLMs to specific domains using both labeled and unlabeled data. The authors introduce SemiEvol, a framework that propagates knowledge from labeled to unlabeled data using in-weight and in-context methods, and then selects high-quality pseudo-labeled data through collaborative learning and adaptive selection for further fine-tuning. Experiments on seven datasets show SemiEvol improves Llama3.1-8B performance on MMLU from 67.9% (SFT baseline) to 70.3%. This implies that AI practitioners can significantly enhance LLM performance and adaptability in target scenarios by leveraging unlabeled data alongside limited labeled datasets. The paper doesn’t specify the hardware used for training or inference. Follow-up questions: 1. What is the computational cost of the collaborative learning stage, and how does it scale with the number of collaborating LLMs (n)? 2. How does the choice of embedding function ε(.) for in-context propagation affect overall performance on different downstream tasks? 3. Could the adaptive selection strategy be further improved by incorporating other metrics beyond entropy, such as model confidence scores or agreement among the collaborating LLMs? |
| Zero-shot Model-based Reinforcement Learning using Large Language Models (Read more on arXiv or HuggingFace) | GPaolo, albert9000, Xssama, ambroiseodt, abenechehab | This paper investigates how pre-trained Large Language Models (LLMs) can be used for zero-shot dynamics prediction in continuous-state Markov Decision Processes. The researchers developed Disentangled In-Context Learning (DICL), which uses Principal Component Analysis to address the challenges of incorporating action information and state dimension interdependence in LLM contexts. In the HalfCheetah environment, DICL reduced multi-step prediction error compared to a vanilla ICL approach and an MLP baseline. Specifically, using half the number of original features, DICL achieved lower multi-step prediction errors and significantly decreased computational time compared to vanilla ICL. This suggests LLMs, combined with DICL, can improve sample efficiency and accelerate learning in model-based reinforcement learning by accurately predicting dynamics from limited trajectories. Follow-up questions: 1. How does the choice of dimensionality reduction technique (PCA in this case) affect the performance and calibration of DICL in various environments, and are there alternative techniques that might be better suited for specific MDP characteristics? 2. What are the scaling properties of DICL with increasing state and action space dimensionality, and how can the computational cost of LLM inference be further optimized for real-time applications? 3. The paper mentions the potential for using autoencoders within DICL. Have experiments been conducted in this direction, and if so, how does the performance compare to the PCA-based approach, especially regarding the disentanglement capabilities? |
| Selecting Influential Samples for Long Context Alignment via Homologous Models’ Guidance and Contextual Awareness Measurement (Read more on arXiv or HuggingFace) | Yunshui Li, Gang Chen, Haozhe Zhao, Shuzheng Si, kaikai1 | a) This research addresses the challenge of selecting high-quality training samples from synthetic long instruction-following data for improved long context alignment in LLMs. b) The proposed GATEAU framework ranks samples based on combined scores from Homologous Models’ Guidance (HMG), which measures difficulty of response generation due to long-range dependencies, and Contextual Awareness Measurement (CAM), which evaluates the model’s focus on important segments in long input contexts. c) Using only 30% of the LongAlign dataset selected by GATEAU, the fine-tuned LLaMA model achieved a 9% improvement on the LongBench-Chat benchmark compared to training on the entire dataset. d) AI practitioners can use GATEAU to improve the data efficiency and performance of LLMs on long-context tasks by selecting influential training samples enriched with long-range dependencies. The impactful finding of a significant performance boost with a smaller, curated dataset has direct relevance for efficient LLM fine-tuning. Follow-up questions: 1. How does the computational cost of GATEAU’s sample selection process compare to the cost of training on the full dataset, and at what scale (dataset size, model size) does GATEAU become more cost-effective? 2. How robust is GATEAU to the choice of homologous models, particularly when applied to different LLM architectures or different pre-training datasets? 3. Could GATEAU be adapted for few-shot or zero-shot settings where fine-tuning isn’t possible, and if so, how would the selection criteria be modified? |
| CBT-Bench: Evaluating Large Language Models on Assisting Cognitive Behavior Therapy (Read more on arXiv or HuggingFace) | Travis Labrum, wangwilliamyang, xz97, Xianjun, billmianz | This research investigates the efficacy of Large Language Models (LLMs) in assisting Cognitive Behavioral Therapy (CBT). The authors developed CBT-BENCH, a three-level benchmark comprising multiple-choice questions, cognitive model understanding tasks (cognitive distortion, primary/fine-grained core belief classification), and therapeutic response generation tasks based on Deliberate Practice exercises. Experimental results showed that while larger LLMs performed better on basic CBT knowledge questions (e.g., Gemma-2-9B achieved 90% accuracy), their performance on fine-grained core belief classification remained poor (weighted F1 score of 54.6% for the best-performing model). This indicates a limitation in current LLMs’ ability to understand complex cognitive models, even with increasing size. AI practitioners should focus on improving LLMs’ capacity for deep cognitive model analysis beyond simple knowledge recall to enhance their potential for assisting in real-world CBT applications. Follow-up questions: 1. What specific architectural modifications or training strategies might be explored to improve LLMs’ performance on fine-grained belief classification and cognitive model understanding, given that simply increasing model size doesn’t seem sufficient? 2. How could the Deliberate Practice exercises for therapeutic response generation be adapted or expanded to better assess empathetic and autonomy-respecting responses, given that the current evaluation criteria might not fully capture these nuanced aspects of CBT? 3. What are the ethical implications of using LLMs to analyze patient speech and assist in therapy, and what safeguards should be implemented to ensure patient privacy and responsible use of this technology? |
| Cross-Lingual Auto Evaluation for Assessing Multilingual LLMs (Read more on arXiv or HuggingFace) | anoopk, prajdabre, dipsivenkatesh, safikhan, sumanthd | a) This research aimed to develop a framework for automated, cross-lingual evaluation of multilingual Large Language Models (LLMs). b) The researchers created a novel multilingual test set (RECON) and trained a series of evaluator LLMs (HERCULE) on an automatically translated training set (INTEL) derived from an English evaluation dataset. HERCULE uses reference answers in English to assess responses generated in other languages. c) On the RECON test set, the fine-tuned HERCULE model achieved a linear weighted Cohen’s Kappa (κ) score of 0.73, outperforming zero-shot evaluations with large, proprietary LLMs like GPT-4. d) This work provides AI practitioners with a scalable and more effective approach for evaluating multilingual LLMs, especially in low-resource scenarios, by leveraging readily available English references. The superior performance of the trained evaluator highlights the benefit of training specialized models for evaluation tasks. Follow-up questions: 1. How does the performance of HERCULE vary across different language families or typologically distinct languages? 2. Given the observation of HERCULE sometimes relying on parametric knowledge instead of the reference answer, what strategies could be employed to improve its reliance on the provided references? 3. What are the limitations of relying on automatically translated training data like INTEL, and how can these limitations be addressed in future research? |
| DM-Codec: Distilling Multimodal Representations for Speech Tokenization (Read more on arXiv or HuggingFace) | A K M Mahbubur Rahman, Md Fahim, amanchadha, tasnim, mubtasim | a) The research aims to improve speech tokenization by incorporating contextual information from language models (LMs) and semantic information from self-supervised speech models (SMs) alongside acoustic information. b) The proposed DM-Codec utilizes a neural codec architecture with Residual Vector Quantization (RVQ) and introduces novel LM-guided and combined LM and SM-guided distillation techniques to integrate multimodal representations into the learning process. c) DM-Codec achieved a Word Error Rate (WER) of 4.05 and a Word Information Lost (WIL) of 6.61 on the LibriSpeech benchmark, outperforming baseline models like SpeechTokenizer, FACodec, and EnCodec. d) AI practitioners can leverage DM-Codec’s distillation approach to build more contextually and semantically aware speech tokenizers, leading to improved performance in downstream speech-related tasks such as speech synthesis and speech-to-text. The significant reduction in WER and WIL directly translates to more accurate and information-rich speech transcription and generation. Follow-up Questions: 1. How does the computational cost of DM-Codec during inference compare to the baseline models, given the added complexity of multimodal distillation during training? 2. The paper mentions using a specific set of pre-trained LMs and SMs. What is the impact of using different pre-trained models (e.g., larger LMs or more recent SM architectures) on the performance of DM-Codec? 3. How does DM-Codec perform on noisy or accented speech data compared to the baseline models, and what modifications could be made to improve its robustness in such scenarios? |
Papers for 2024-10-21
| Title | Authors | Summary |
|---|---|---|
| Web Agents with World Models: Learning and Leveraging Environment Dynamics in Web Navigation (Read more on arXiv or HuggingFace) | jihoonkim25, Gwanwoo, ktio, kimnamssya, hyungjoochae | a) This research investigates the limitations of Large Language Models (LLMs) in web navigation, particularly their lack of “world models” (awareness of action outcomes), and proposes World-Model-Augmented (WMA) web agents to address this. b) WMA agents use a world model trained on a dataset with transition-focused observation abstraction (highlighting state differences between time steps) to predict action outcomes, and a value function to select the action leading to the highest estimated reward. c) WMA agents achieve a 43.6% improvement in success rate over vanilla Chain-of-Thought prompting in the Map domain of the WebArena benchmark using GPT-40-mini as the policy model. d) AI practitioners can leverage WMA agents to improve the decision-making of LLM-based web agents by incorporating the ability to simulate action consequences without training the policy model, leading to more efficient and goal-directed web navigation. This suggests world models are a promising direction for improving agent performance in complex, long-horizon web navigation tasks. Follow-up questions: 1. How does the performance of the WMA agent vary across different LLM architectures and sizes used for both the world model and the policy model? 2. What are the computational costs and limitations of scaling the transition-focused observation abstraction to more complex websites with dynamic content and user interactions? 3. Could the transition-focused observation abstraction approach be generalized to other sequential decision-making tasks beyond web navigation? |
| UCFE: A User-Centric Financial Expertise Benchmark for Large Language Models (Read more on arXiv or HuggingFace) | SP4595, Yueru1, wittenberg, amstrongzyf, TobyYang7 | This paper introduces UCFE, a benchmark designed to evaluate large language models’ (LLMs) ability to handle complex, real-world financial tasks. The methodology combines human expert evaluations with dynamic, task-specific interactions simulating evolving financial scenarios. Results showed a strong correlation (0.78 Pearson coefficient) between benchmark scores and human preferences. This implies UCFE effectively assesses LLM performance and user satisfaction in financial applications. Mid-sized LLMs (7B-14B parameters) performed well, balancing computational efficiency and domain expertise. Follow-up questions: 1. How does UCFE compare to existing financial benchmarks like FLARE in terms of task complexity and evaluation metrics? 2. Could the dynamic interaction component of UCFE be adapted to evaluate LLMs in other domains requiring specialized knowledge and evolving scenarios? 3. What specific improvements were observed in financial LLMs compared to their backbone models, and how can these improvements be attributed to the continued pre-training on financial corpora? |
| MagicTailor: Component-Controllable Personalization in Text-to-Image Diffusion Models (Read more on arXiv or HuggingFace) | gychen, jzwangcuhk, BryanW, jiancheng, donghao-zhou | a) The research introduces “component-controllable personalization,” a new task aiming to modify specific components of a visual concept during personalization of text-to-image (T2I) diffusion models. b) MagicTailor, the proposed framework, leverages Dynamic Masked Degradation (DM-Deg) to perturb unwanted visual semantics and Dual-Stream Balancing (DS-Bal) to balance learning of concept and component semantics. The model is fine-tuned using a masked diffusion loss and a cross-attention loss. c) MagicTailor achieved state-of-the-art performance in component-controllable personalization, reaching 56.5% in text alignment (CLIP-T) based on a user study, exceeding other personalization methods by at least 40 percentage points. d) AI practitioners can use MagicTailor to fine-tune T2I models for more nuanced and controlled image generation, enabling the customization of individual components of visual concepts from reference images. Follow-up questions: 1. What is the computational cost (time and resources) of training MagicTailor compared to baseline personalization methods like DreamBooth and Textual Inversion? 2. How does MagicTailor handle more complex concepts comprising multiple components or scenarios where the components overlap significantly in the reference images? 3. Could the DM-Deg and DS-Bal techniques be adapted to improve fine-grained control in other generative tasks, such as image editing or video generation? |
| NaturalBench: Evaluating Vision-Language Models on Natural Adversarial Samples (Read more on arXiv or HuggingFace) | zixianma, Nyandwi, Lilymelon7, zhiqiulin, BaiqiL | a) The research investigates whether current Vision-Language Models (VLMs) are truly effective, hypothesizing that they struggle with seemingly simple, natural image-question pairs. b) Researchers developed NaturalBench, a semi-automated benchmark with 10,000 human-verified VQA samples, using CLIP and ChatGPT to generate initial samples from natural image-text corpora, followed by human verification. A vision-centric design using question/image pairs with alternating answers prevents “blind” solutions. c) Evaluations of 53 state-of-the-art VLMs on NaturalBench demonstrate that even the best models, like GPT-40, perform significantly below human accuracy (over 90%), achieving only 39.6% group accuracy. d) NaturalBench provides a more robust evaluation for VLMs, highlighting areas for improvement by identifying biases and assessing diverse visio-linguistic skills. This necessitates focusing on debiasing techniques and improving models’ compositional reasoning abilities in visio-linguistic tasks for AI practitioners. Follow-up questions: 1. What specific debiasing techniques, beyond adjusting the prediction threshold (τ), were explored in the Appendix, and how effective were they in improving performance on NaturalBench without requiring knowledge of image-question pairings? 2. Can the NaturalBench benchmark generation methodology be adapted to create specialized datasets for evaluating specific visio-linguistic skills, allowing for targeted model improvement in areas like attribute binding or spatial reasoning? 3. Given the computational cost of fine-tuning large models like GPT-40, are there more efficient methods for mitigating the identified biases, such as incorporating debiasing strategies directly into the model architecture or training process? |
| SeerAttention: Learning Intrinsic Sparse Attention in Your LLMs (Read more on arXiv or HuggingFace) | Hayden Kwok-Hay So, tingcao, Daniel-Duda, CharyZeng, Retromonic | a) The paper investigates learning intrinsic attention sparsity in Large Language Models (LLMs) to improve efficiency, rather than relying on predefined patterns. b) The authors introduce SeerAttention, an attention mechanism with a learnable gate (AttnGate) that identifies important blocks in attention maps, enabling block-sparse computation via a custom FlashAttention kernel. AttnGate is trained using a max-pooled full attention map as ground truth, obtained through a modified FlashAttention kernel. c) SeerAttention achieves up to a 5.67x speedup compared to FlashAttention-2 at a 90% sparsity ratio and 32k context length, with minimal perplexity loss when integrated with YaRN for long-context fine-tuning. d) AI practitioners can leverage SeerAttention to significantly accelerate LLM inference, particularly for long sequences, without substantial accuracy degradation, by integrating this learned sparsity approach into existing or new models. Follow-up questions: 1. How easily can SeerAttention be integrated into existing LLM training frameworks and deployed to production environments? Are there specific hardware requirements or software dependencies? 2. The paper focuses on prefill attention; are there plans or insights into extending SeerAttention to the decoder phase of LLMs, and what performance gains might be expected? 3. What are the memory implications of using SeerAttention during training and inference compared to other sparse attention methods and dense attention? |
| Are AI Detectors Good Enough? A Survey on Quality of Datasets With Machine-Generated Texts (Read more on arXiv or HuggingFace) | Yury Chekhovich, Anastasia Voznyuk, German Gritsai, andriygav | a) The research investigated the quality of datasets used for training and evaluating AI-generated text detectors, questioning if high reported performance stems from dataset deficiencies. b) The authors evaluated multiple datasets using several detection methods (DeBERTa classifier, DetectGPT, Binoculars), topological time series analysis of text embeddings, and adversarial text perturbations (synonym replacement, sentence shuffling). c) On the HC3 dataset, the KL-divergence of topological time series distributions for human and machine-generated texts was 0.053, indicating some separability but also suggesting potential dataset limitations. d) AI practitioners should be cautious about relying solely on benchmark results for AI text detectors, as high performance might be due to biases or low generalizability of the evaluation datasets rather than true detector efficacy. The paper, however, does not provide clear guidelines or definitive criteria for assessing dataset quality for AI-generated text detection. Follow-up questions: 1. What specific criteria or thresholds should be used for the proposed dataset evaluation metrics (KLTTS, Ashift, KLshuffle) to determine whether a dataset is of sufficient quality for training and evaluating AI text detectors? 2. How can the proposed evaluation methods be extended or adapted to assess datasets for more complex tasks like hybrid writing detection or authorship attribution? 3. Can the authors elaborate on the limitations of KLTTS with short texts? What are the specific computational instability issues? How can those be addressed and applied for evaluating short generated texts? |
| Diffusion Curriculum: Synthetic-to-Real Generative Curriculum Learning via Image-Guided Diffusion (Read more on arXiv or HuggingFace) | Shweta Bhardwaj, Yijun Liang, zhoutianyi | a) This research investigates how to improve deep neural network training with low-quality or scarce data by addressing the distribution gap between synthetic and real data. b) The proposed “Diffusion Curriculum (DisCL)” leverages image guidance in diffusion models to generate a spectrum of synthetic-to-real interpolated data for hard samples. DisCL then uses curriculum learning strategies to select appropriate data from this spectrum for different training stages. c) On the iWildCam dataset, DisCL improved the out-of-distribution (OOD) and in-distribution (ID) macro-accuracy by 2.7% and 2.1%, respectively. On ImageNet-LT, it improved tail-class accuracy from 4.4% to 23.64%. d) AI practitioners can utilize DisCL to enhance the performance of image classifiers, particularly when dealing with challenging real-world datasets characterized by low quality or long-tailed class distributions. The demonstrated performance boost on tail classes suggests DisCL can significantly improve representation learning in data-scarce scenarios. Follow-up questions: 1. How does the computational cost of generating the synthetic data spectrum using DisCL compare to other data augmentation techniques, particularly for large datasets? 2. Could the adaptive curriculum selection strategy in DisCL be improved by incorporating other metrics beyond prediction score progress, such as feature diversity or uncertainty estimates? 3. The paper mentions limitations regarding the quality of generated data being dependent on the diffusion model and filtering model. What specific steps could be taken to mitigate these dependencies and improve the overall robustness of DisCL? |
| DAWN: Dynamic Frame Avatar with Non-autoregressive Diffusion Framework for Talking Head Video Generation (Read more on arXiv or HuggingFace) | dujun, Bazhu, page-xia, Limin-Lin, Hanbo-Cheng | a) The research aims to develop a faster, higher-quality method for generating talking-head videos from a single portrait image and an audio clip, addressing limitations of autoregressive and semi-autoregressive approaches. b) The proposed DAWN framework uses a non-autoregressive diffusion model (A2V-FDM) to generate motion representations, disentangling lip movements from head pose and blinks, which are generated separately by a Pose and Blink generation Network (PBNet). A two-stage curriculum learning strategy is employed for training. c) DAWN achieved state-of-the-art performance on the CREMA and HDTF datasets, including a Fréchet Inception Distance (FID) score of 9.60 and a Beat Align Score (BAS) of 0.281 on HDTF. d) AI practitioners can leverage DAWN for real-time or near real-time generation of dynamic-length talking head videos, potentially improving applications in virtual meetings, gaming, and film production by removing reliance on slow autoregressive methods. Follow-up questions: 1. How does the computational cost of DAWN during inference compare to autoregressive and semi-autoregressive methods, particularly for very long video sequences? 2. What are the limitations of the proposed disentanglement of lip movements, head pose, and blinks, and how might these limitations impact the realism of generated videos in complex scenarios with diverse head and facial movements? 3. Could the two-stage curriculum learning approach be generalized to other video generation tasks beyond talking heads, and what modifications might be necessary for effective application in these different contexts? |
| A Common Pitfall of Margin-based Language Model Alignment: Gradient Entanglement (Read more on arXiv or HuggingFace) | Yue Wu, leqiliu, Edify-Kd2024, yokey, huiyuan23 | This paper investigates the unintended consequences of using margin-based losses for preference optimization in language model alignment. The authors analyze the training dynamics of various margin-based methods, including Direct Preference Optimization (DPO), through theoretical analysis and empirical validation on text summarization and sentiment classification tasks. A key finding is the “gradient entanglement” effect, where changes in the chosen and rejected response log-probabilities are coupled through their gradient inner product. In experiments on a sentiment classification task, the chosen log probability increased with single-token responses, but decreased with longer suffix responses. This finding directly impacts alignment procedures as increasing the margin between preferred and dispreferred responses does not guarantee improved alignment and can even worsen performance on certain responses. Follow-up questions: 1. How can the proposed pairwise normalized gradient descent or sparsity regularized token masking methods be efficiently implemented in large-scale language model training? 2. What are the trade-offs between using margin-based methods versus alternative alignment strategies, especially in safety-critical applications where minimizing the probability of undesirable responses is paramount? 3. How does gradient entanglement influence the performance of reward models in traditional RLHF pipelines where reward modeling and policy optimization are distinct stages? |
| DPLM-2: A Multimodal Diffusion Protein Language Model (Read more on arXiv or HuggingFace) | Dongyu Xue, Fei Ye, Zaixiang Zheng, Xinyou Wang, thughost | a) The research aimed to develop a multimodal protein foundation model capable of simultaneously modeling, understanding, and generating both protein sequences and structures. b) DPLM-2 extends the discrete diffusion protein language model (DPLM) by incorporating structure information via a lookup-free quantizer (LFQ) tokenizer and training on experimental and synthetic structure data, using a warmup strategy from pre-trained DPLM and a self-mixup training strategy. c) DPLM-2 achieves competitive performance in unconditional structure-sequence co-generation, with a self-consistency TM-score (scTM) exceeding 0.9 for most generated proteins across various lengths. It also demonstrated competitive ability in folding, inverse folding, and motif scaffolding. d) AI practitioners can leverage DPLM-2 for various protein engineering tasks involving simultaneous sequence and structure generation or manipulation. The demonstration of effective multimodal training using discrete tokenized structure data provides a blueprint for other applications involving joint modeling of discrete and continuous data. Follow-up questions: 1. What are the limitations of the LFQ tokenizer regarding the potential loss of fine-grained structural information, and how might these limitations impact downstream applications requiring precise structural details? 2. How does the performance of DPLM-2’s structure-aware representations compare to existing dedicated structure-based models in downstream tasks beyond those presented in the paper, and what are the trade-offs between using DPLM-2 versus a specialized model for specific structure-related tasks? 3. Given the observed length extrapolation capabilities, what is the impact of training dataset length distribution and maximum length on the performance and stability of DPLM-2 when generating substantially longer sequences and structures exceeding those encountered during training? |
| Context is Key(NMF): Modelling Topical Information Dynamics in Chinese Diaspora Media (Read more on arXiv or HuggingFace) | Mette Thunø, Rebecca M. M. Hicke, Ross Deans Kristensen-McLachlan, kardosdrur | a) The research investigates potential PRC influence on European elections through Chinese diaspora media by analyzing how PRC narratives are represented and thus the objectives of PRC news media manipulation. b) The study uses a novel dynamic topic modeling pipeline combining KeyNMF, a transformer-based contextual embedding approach for topic extraction with Non-negative Matrix Factorization (NMF), and measures of novelty and resonance to analyze Chinese news articles. c) KeyNMF achieved higher external coherence scores compared to traditional and some contemporary topic models (e.g., LDA, NMF) on most of the tested corpora, exceeding LDA and NMF considerably. d) This research presents KeyNMF as a potentially more effective approach for topic modeling, especially in multilingual or data-scarce settings, offering AI practitioners a new tool for contextualized topic extraction and analysis of information dynamics. Follow-up questions: 1. How does KeyNMF’s performance compare to BERTopic or other dynamic topic models specifically in terms of computational cost and scalability for large datasets? 2. What are the limitations of using KeyNMF with other languages besides Chinese, considering the reliance on jieba tokenizer, a Chinese-specific tool? 3. Can the observed correlation between novelty/resonance signals and political events be used to predict future similar reactions or is further research needed to establish causality? |
| How Do Training Methods Influence the Utilization of Vision Models? (Read more on arXiv or HuggingFace) | Janis Keuper, Margret Keuper, Shashank Agnihotri, Paul Gavrikov | This research investigates how different training methods affect the criticality of layers in ResNet-50 ImageNet-1k classification models. The study randomized individual layer parameters and measured the cosine distance between the original and randomized output probability vectors to determine layer criticality. Results showed that training methods significantly influence layer criticality; for instance, a spatial convolution layer ([3.5] conv2) exhibited an average criticality of 36% but reached 95% when trained with PixMix. While some layers, like the initial stem convolution and classification head, were always critical, no layer was consistently auxiliary across all training methods. This implies that AI practitioners should consider training methodology when assessing the relative importance of different layers for a given task, as certain training methods may under-utilize specific layers, affecting potential optimization strategies like pruning or distillation. Follow-up questions: 1. How do these findings translate to other architectures beyond ResNet-50, such as vision transformers or ConvNeXt models? 2. The paper mentions a correlation between criticality and generalization suggested by prior work, but finds a weak correlation on their dataset. How might this correlation change with different datasets or evaluation metrics beyond ImageNet accuracy? 3. Could layer criticality analysis be integrated into the training process itself to dynamically adjust resource allocation or pruning strategies during training? |
Papers for 2024-10-18
| Title | Authors | Summary |
|---|---|---|
| MixEval-X: Any-to-Any Evaluations from Real-World Data Mixtures (Read more on arXiv or HuggingFace) | kcz358, fuzhao, Junhao233, dghosal, jinjieni | a) The research aimed to address inconsistencies and biases in current multi-modal AI evaluations and create a benchmark that better reflects real-world task distributions. b) MixEval-X was developed using a multi-modal benchmark mixture pipeline for understanding tasks and an adaptation-rectification pipeline for generation and agent tasks, both leveraging real-world user queries from Common Crawl. c) Meta-evaluations showed strong correlations between MixEval-X results and real-world user-facing evaluations, with Image2Text showing a 98.1% Spearman’s ranking correlation with Vision Arena. The paper does not provide information on the correlation between crowd-sourced evaluations and model-based evaluations of open-ended generation tasks beyond noting low correlation. d) MixEval-X offers AI practitioners a unified, real-world benchmark with diverse input-output modalities to facilitate more accurate and generalizable evaluations of multi-modal models and potentially different organizations. The paper does not detail how organizations are ranked or compared beyond a high-level overview in Figure 1. Follow-up questions: 1. Could you elaborate on the specific adaptation-rectification pipeline steps for MMG and agent tasks, including prompt examples and the impact of human review? 2. What are the specific metrics used for measuring the alignment between MixEval-X and real-world task distributions beyond visual representations and correlation with existing leaderboards? 3. What are the limitations of MixEval-X, especially regarding the evaluation of open-ended generation tasks, and what future research directions could address these limitations? |
| Movie Gen: A Cast of Media Foundation Models (Read more on arXiv or HuggingFace) | AnnLee, animeshsinha, androstj, amitz, adampo | a) The research aimed to develop a suite of foundation models (MovieGen) capable of generating and manipulating high-quality videos and audio, including personalization and editing. b) The team used transformer-based models trained with flow matching on large-scale image, video, and audio datasets, incorporating techniques like spatio-temporal compression, rich text embeddings, and post-training for personalization and editing. Multi-stage training with progressive resolution scaling and supervised fine-tuning was employed for video generation. c) MovieGen outperformed existing models on text-to-video generation, achieving a 35.02% net win rate against Runway Gen3 on overall video quality. It is unclear from the paper if these are cherry-picked examples or comprehensive benchmarks. d) AI practitioners can leverage MovieGen’s architecture and training techniques to develop high-quality video generation and editing models, pushing the state-of-the-art in media generation and manipulation. The focus on scaling data, model size, and compute resources highlights the importance of these factors for achieving superior results in generative AI for media. Follow-up questions: 1. The paper mentions using Flow Matching. What specific implementation details and hyperparameters were used for this objective function, and how were they tuned for optimal performance across different datasets and model sizes? 2. What specific metrics and evaluation protocols were used for assessing the quality of personalized videos, and how do these metrics address the potential biases introduced by using human evaluators? 3. Could you elaborate on the specifics of the “novel post-training procedure” used to produce MovieGen Edit and its advantages compared to other video editing training methods, including data augmentation techniques and loss functions? |
| Harnessing Webpage UIs for Text-Rich Visual Understanding (Read more on arXiv or HuggingFace) | Yuxiao Qu, Yifan Song, yuexiang96, oottyy, jeepliu | a) This research aims to improve text-rich visual understanding in multimodal large language models (MLLMs). b) The authors construct MultiUI, a 7.3-million-sample dataset synthesized from 1 million website UIs using text-based LLMs to generate multimodal instructions paired with UI screenshots. The dataset covers nine tasks across three categories: visual understanding and reasoning, text recognition, and grounding. Models are then trained on MultiUI and tested on both web UI and general multimodal benchmarks. c) Models trained on MultiUI achieve up to a 48% improvement on VisualWebBench and generalize to non-web UI domains like document understanding and chart interpretation, indicating the broader applicability of web UI data. d) AI practitioners can leverage web UI data as a powerful resource for training MLLMs in text-rich visual understanding, enabling models to perform well across a broader range of tasks beyond just web UI-specific scenarios. The surprising generalization to non-UI domains highlights the potential for cross-domain knowledge transfer when using this type of data. Follow-up questions: 1. What specific techniques were used to clean and process the accessibility trees to ensure they were suitable for LLM processing, and how did this impact the quality of the generated instructions? 2. While the paper demonstrates promising cross-domain generalization, what are the limitations of this approach, and what further research could be done to mitigate these limitations, particularly in domains with visually distinct characteristics from web UIs? 3. Could the methodology for creating synthetic training data from web UIs using LLMs be adapted or extended to create datasets for other multimodal tasks, such as video understanding or audio-visual scene analysis? |
| MobA: A Two-Level Agent System for Efficient Mobile Task Automation (Read more on arXiv or HuggingFace) | Yixuan Jiang, Kunyao Lan, Yansi Li, Hao Tang, JamesZhutheThird | a) The research aimed to improve mobile task automation by addressing the limitations of current mobile assistants, such as dependence on APIs and difficulty handling complex, dynamic GUI environments. b) The researchers developed MobA, a two-level agent system utilizing multimodal large language models (MLLMs) with a high-level Global Agent for planning and a low-level Local Agent for execution, incorporating a double-reflection mechanism and a multi-aspect memory module. c) Evaluated on MOBBENCH, a 50-task mobile scenario dataset, MobA achieved a 66.2% milestone score rate, surpassing the second-best baseline by over 17%. d) AI practitioners can leverage MobA’s two-level agent architecture, reflection mechanism, and memory modules to improve the efficiency and completion rate of MLLM-powered mobile assistants for complex real-world tasks. The significant improvement in milestone score rate achieved by MobA demonstrates the potential of this approach for building more robust and effective mobile automation systems. Follow-up questions: 1. How does MobA’s performance compare to other state-of-the-art MLLM-based agents on other benchmark datasets beyond MOBBENCH, and what are the key factors contributing to any performance differences? 2. What are the specific implementation details and computational costs associated with the double-reflection mechanism, and how can these be optimized for real-time performance on resource-constrained mobile devices? 3. How does the design of the memory module in MobA address the challenges of long-term memory management and retrieval in the context of mobile task automation, and what are the trade-offs between different memory retrieval strategies (relation-based vs. content-based)? |
| Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generation (Read more on arXiv or HuggingFace) | zdaxie, zizhpan, XCLiu, CNMaxwell, WuChengyue | a) The paper investigates whether decoupling visual encoding for multimodal understanding and generation tasks within a unified model improves performance compared to using a single visual encoder. b) The researchers developed Janus, a unified autoregressive transformer model employing separate visual encoders for understanding (SigLIP) and generation (VQTokenizer) tasks, trained in a three-stage process involving adaptor and image head training, unified pretraining, and supervised fine-tuning. c) Janus achieved 69.4 on the MMBench benchmark, outperforming other unified models of comparable size and even some larger, task-specific models. d) The results suggest that AI practitioners building unified multimodal models should consider decoupling visual encoding pathways to potentially improve performance, particularly in understanding tasks, without significant performance degradation in generation tasks. Follow-up questions: 1. What is the computational overhead of using two separate visual encoders compared to a single encoder, and how does this impact practical deployment? 2. Could other encoding methods besides SigLIP and VQTokenizer be more optimal for specific understanding or generation tasks within the Janus framework? 3. How does the performance of Janus scale with different LLM sizes, and what are the limitations of using smaller LLMs in this decoupled architecture? |
| MMed-RAG: Versatile Multimodal RAG System for Medical Vision Language Models (Read more on arXiv or HuggingFace) | Weijia Shi, Tianze Wang, Haoran Li, Kangyu Zhu, richardxp888 | This research addresses the issue of factual hallucinations in Medical Large Vision-Language Models (Med-LVLMs). The authors propose MMed-RAG, a multimodal Retrieval Augmented Generation (RAG) system incorporating domain-aware retrieval, adaptive context selection, and RAG-based preference fine-tuning. On medical Visual Question Answering (VQA) and report generation tasks across five datasets, MMed-RAG improved the factual accuracy of Med-LVLMs by an average of 18.5% for VQA and 69.1% for report generation compared to the original Med-LVLM. This suggests that MMed-RAG’s components effectively mitigate misalignment issues introduced by incorporating retrieved knowledge. AI practitioners can leverage MMed-RAG to improve the factuality and reliability of Med-LVLMs for real-world medical applications. Follow-up questions: 1. What are the specific architectural details of the domain identification module within the domain-aware retrieval mechanism, and how is its performance evaluated in isolation? 2. How does the computational cost of MMed-RAG during inference compare to the original Med-LVLM and other baseline methods, considering the overhead of retrieval and context selection? 3. How robust is MMed-RAG to noisy or incomplete retrieved contexts, and what mitigation strategies could be employed to further enhance its reliability in such scenarios? |
| A Unified View of Delta Parameter Editing in Post-Trained Large-Scale Models (Read more on arXiv or HuggingFace) | Keming Lu, Hongyu Lin, Bowen Yu, Le Yu, TangQiaoYu | a) This paper aims to establish a unified framework for understanding how various delta parameter editing operations (pruning, quantization, etc.) affect the performance of post-trained large-scale models. b) The research analyzes delta parameter editing through the lens of Riemann sum approximation of the loss function difference between post-trained and edited models. c) Experiments on ViT, LLaMA 3, Qwen 2, and Mistral models showed that DARE can eliminate up to 99% of delta parameters while maintaining competitive performance. The paper doesn’t provide enough quantitative detail to compare other editing operations besides DARE across all models and datasets tested. d) AI practitioners can use the Riemann sum approximation framework to predict the performance impact of different delta parameter editing techniques and to design new editing methods for improved model compression or performance enhancement. The impact is especially relevant for model compression, as demonstrated by the success of DARE in significantly reducing model size without substantial performance loss. Follow-up questions: 1. How does the choice of the constant C in the Riemann sum approximation affect the accuracy of the performance predictions for different model architectures and datasets? 2. Can the proposed framework be extended to analyze the effects of delta parameter editing in the context of parameter-efficient fine-tuning methods? 3. Beyond the average magnitude, what other holistic statistics of delta parameters could be explored in the quantization approach, and how can we systematically evaluate their effectiveness? |
| PopAlign: Diversifying Contrasting Patterns for a More Comprehensive Alignment (Read more on arXiv or HuggingFace) | Ke Xu, Jiaheng Liu, Shawn Wang, Zekun Moore Wang, kangz | a) The research investigates how to construct more comprehensive and diversified contrasting patterns to enhance preference data for large language model (LLM) alignment and verifies the impact of diversifying these patterns. b) PopAlign, a framework integrating six contrasting strategies across prompt, model, and pipeline levels, is proposed to synthesize preference-contrastive data without additional feedback labeling. The models are then trained using Direct Preference Optimization (DPO). c) PopAlign achieved a 19.0% win rate against GPT-3.5 on AlpacaEval 2.0 (length-controlled), compared to 11.8% for the base Yi-6B-Chat model. d) AI practitioners can leverage PopAlign to create more comprehensive alignment datasets, potentially leading to more robust and less susceptible LLMs by distilling diversified contrasting patterns across the response generation workflow. The paper suggests “Elicitive Contrast” is particularly effective. e) The paper mentions using Yi-34B-Chat and Vicuna-33B for Leaderboard Contrast, citing a training data quality gap as the main performance differentiator. It is unclear whether other factors (e.g., architecture, training methodology) were controlled for. Follow-up questions: 1. How does PopAlign’s performance scale with larger LLMs and datasets, and what are the computational resource implications? 2. Can the “Elicitive Contrast” strategy be further optimized or adapted for different LLM architectures or tasks? 3. How robust is PopAlign to adversarial attacks aimed at exploiting specific contrasting patterns? |
| MoH: Multi-Head Attention as Mixture-of-Head Attention (Read more on arXiv or HuggingFace) | Shuicheng Yan, Li Yuan, Bo Zhu, Chat-UniVi | This research aims to improve the efficiency of multi-head attention in Transformer models while maintaining or exceeding accuracy. The authors propose Mixture-of-Head attention (MoH), which uses a router to select a subset of attention heads for each token and employs a weighted summation of the selected heads’ outputs. Experiments with MoH-LLaMA3-8B showed an average accuracy of 64.0% across 14 benchmarks, a 2.4% improvement over LLaMA3-8B while using only 75% of the attention heads. This implies that MoH can enable more efficient use of computational resources in attention-based models without sacrificing performance. The paper doesn’t specify the proportion of shared versus routed heads used in MoH-LLaMA3-8B. Follow-up questions: 1. What are the computational costs and latency implications of the routing mechanism in MoH compared to standard multi-head attention, and how do these scale with model size? 2. How does the performance of MoH change when different criteria are used for selecting shared attention heads (besides simply selecting the first n heads)? 3. Could the two-stage routing strategy be further optimized for different modalities, like vision or audio, and how would this impact performance and efficiency? |
| DreamVideo-2: Zero-Shot Subject-Driven Video Customization with Precise Motion Control (Read more on arXiv or HuggingFace) | Haonan Qiu, Xiang Wang, Hangjie Yuan, Shiwei Zhang, Yujie Wei | a) The research aimed to develop a zero-shot video customization framework capable of generating videos with user-specified subjects and motion trajectories, without test-time fine-tuning. b) DreamVideo-2 utilizes reference attention for subject learning from a single image and a mask-guided motion module (spatiotemporal encoder + ControlNet) for motion control from bounding box sequences. Masked reference attention and a reweighted diffusion loss are introduced to balance subject learning and motion control. c) On a curated single-subject video dataset, DreamVideo-2 achieved a mean Intersection over Union (mIoU) of 0.670 for motion control, outperforming baseline methods. The paper does not provide specifics on the dataset’s size or composition besides mentioning 230,160 training videos and a test set with 50 subjects and 36 bounding boxes. d) AI practitioners can use DreamVideo-2 to efficiently generate customized videos without requiring computationally expensive fine-tuning, simplifying the process of subject-driven video creation. The balance achieved between subject fidelity and motion control offers greater customization control. Follow-up questions: 1. What are the computational requirements (e.g., GPU memory, training time) of DreamVideo-2 compared to fine-tuning based approaches like DreamVideo and MotionBooth? 2. How does DreamVideo-2 handle complex motion patterns or occlusions of the subject during video generation, and what limitations exist in its motion control capabilities? 3. What is the license of the created dataset and the trained models, and are there any restrictions on usage, especially for commercial use-cases? |
| VidPanos: Generative Panoramic Videos from Casual Panning Videos (Read more on arXiv or HuggingFace) | Shiran Zada, Roni Paiss, Erika Lu, Jingwei Ma, fcole | a) The research aims to synthesize coherent panoramic videos from casually captured panning videos of dynamic scenes. b) The method projects input video frames onto a panoramic canvas, then completes spatiotemporal gaps using diffusion-based (Lumiere) and token-based (Phenaki) generative video models adapted with coarse-to-fine synthesis and spatial aggregation to overcome limited context windows. c) On a synthetic dataset with ground truth, the Lumiere-based method achieves a lower LPIPS score (0.05/0.09 on static/dynamic regions) compared to the best baseline (ProPainter with 0.10/0.19). d) AI practitioners can leverage this technique to generate immersive panoramic videos from limited-FOV panning inputs, enabling novel video creation and viewing experiences. The significant improvement in LPIPS compared to existing inpainting techniques suggests improved perceptual quality for generating realistic and temporally consistent panoramic videos. e) The paper lacks specific quantitative results on real-world panning videos, relying primarily on qualitative comparisons. Follow-up questions: 1. How does the performance of the proposed method compare to baseline methods on metrics besides LPIPS, such as FID, particularly on real-world video datasets? 2. What are the computational resource requirements and runtimes for generating panoramic videos of varying lengths and resolutions using the proposed method with the different generative video models? 3. How robust is the method to variations in camera motion beyond pure panning, such as zooming or tilting, and what are the failure modes in these scenarios? |
| Retrospective Learning from Interactions (Read more on arXiv or HuggingFace) | Anne Wu, Gloria Geng, Yiwei Chen, Mustafa Omer Gul, Zizhao Chen | a) This research investigates whether implicit feedback signals in multi-turn human-LM interactions can be used to improve LM performance without explicit annotations. b) The RESPECT method decodes implicit feedback (positive, neutral, or negative) from past interactions using the LLM itself and retrains the LLM using supervised learning, REINFORCE-style policy gradient, or KTO. This is deployed in MULTIREF, a multi-turn referential game with abstract images. c) In a live deployment setting, the best-performing system (B-SUP, binary feedback with supervised learning) improved task completion rate from 31% to 82% over six rounds of interaction and retraining. d) This implies that AI practitioners can leverage implicit feedback signals present in user interactions to continually improve LLM performance in deployed systems without requiring costly explicit annotations. The effectiveness of leveraging negative feedback, however, remains unclear and requires further investigation. Follow-up questions: 1. How does the performance of RESPECT compare to traditional RLHF methods in terms of both effectiveness and cost efficiency, considering the annotation effort involved in each? 2. What are the limitations of the current feedback decoder, and what strategies can be explored to improve its accuracy and robustness, especially in handling more complex and nuanced feedback signals? 3. How does the choice of the underlying LLM architecture and size impact the effectiveness of RESPECT, and is there an optimal LLM configuration for this retrospective learning approach? |
| FlatQuant: Flatness Matters for LLM Quantization (Read more on arXiv or HuggingFace) | Kang Zhao, Han Bao, Haoli Bai, Yuxuan Sun, lianlio | a) The paper investigates the impact of weight and activation flatness on the effectiveness of Large Language Model (LLM) quantization and proposes a method to improve it. b) The authors introduce FLATQUANT, a post-training quantization approach employing learnable affine transformations with Kronecker decomposition and a lightweight training objective to enhance flatness. An efficient kernel fuses affine transformations and quantization into a single operation for reduced overhead. c) FLATQUANT achieved less than 1% accuracy drop for 4-bit weight and activation quantization on LLaMA-3-70B, surpassing SpinQuant by 7.5% in accuracy. d) AI practitioners can leverage FLATQUANT to significantly reduce the memory footprint and accelerate inference of large language models with minimal accuracy degradation, enabling deployment on resource-constrained hardware. The key impact is the ability to deploy larger, more accurate LLMs with significantly improved inference speed thanks to efficient quantization. Follow-up questions: 1. How does FLATQUANT’s performance compare to other quantization techniques in terms of memory savings and computational efficiency on different hardware platforms besides the RTX3090? 2. What is the impact of different calibration dataset sizes and compositions on FLATQUANT’s performance, particularly for domain-specific LLMs? 3. Does FLATQUANT’s effectiveness generalize to other model architectures beyond the LLaMA family, such as Mixture-of-Experts models? |
| MedMobile: A mobile-sized language model with expert-level clinical capabilities (Read more on arXiv or HuggingFace) | Eric Karl Oermann, Daniel Alexander Alber, Anton Alaykin, Jaden Stryker, KrithikV | a) This research aimed to develop a mobile-sized language model (LM) with expert-level clinical capabilities, addressing computational cost and privacy barriers associated with larger LMs. b) The researchers fine-tuned the 3.8B parameter phi-3-mini LM on the UltraMedical dataset, employing chain-of-thought (CoT) prompting, ensembling, and supervised fine-tuning (SFT). c) The resulting model, MedMobile, achieved 75.7% accuracy on MedQA (USMLE), surpassing the passing threshold for physicians (~60%) and outperforming prior sub-5B parameter models by over 20 percentage points. d) AI practitioners can leverage the findings to develop and deploy smaller, more efficient LMs for specific domains, demonstrating that expert-level performance can be achieved with significantly fewer parameters and thus reduced computational resources. However, the paper lacks details on specific hardware testing for mobile deployment, although it references prior work demonstrating the feasibility of running such sized models on mobile hardware. Follow-up questions: 1. What are the specific latency and power consumption metrics of MedMobile on representative mobile devices during inference, and how do these compare to larger LMs? 2. What are the specific privacy implications of deploying MedMobile on mobile devices, and what mitigation strategies are recommended for handling sensitive patient data within this context? 3. Given that retrieval augmentation did not improve performance, what alternative techniques could be explored to further enhance MedMobile’s clinical knowledge and reasoning capabilities while remaining within mobile-size constraints? |
| Failing Forward: Improving Generative Error Correction for ASR with Synthetic Data and Retrieval Augmentation (Read more on arXiv or HuggingFace) | Jian Xue, Peidong Wang, Michael Levit, Mohammad Sadegh Rasooli, Sreyan Ghosh | This research investigates the limited generalization ability of Generative Error Correction (GEC) models for Automatic Speech Recognition (ASR). The authors propose DARAG (Data- and Retrieval-Augmented Generative Error Correction), which augments GEC training with synthetic speech-transcript pairs generated by LLMs and TTS models and incorporates retrieval-augmented correction for named entities using a datastore. Experiments across five ASR datasets show DARAG improves WER by 8%-30% in in-domain settings and 10%-33% in out-of-domain settings. This implies that AI practitioners can significantly improve ASR performance by training GEC models on a diverse and consistent set of errors similar to those encountered during testing, including explicit NE knowledge. Follow-up Questions: 1. What are the computational costs and infrastructure requirements for implementing DARAG, especially for very large datasets or low-resource languages? 2. How does the choice of specific LLM and TTS models used for synthetic data generation affect DARAG’s performance and potential biases? 3. Can the proposed phoneme-aware NE retrieval method be further elaborated, and are there any comparative evaluations against other retrieval techniques for this specific use-case? |
| LoLDU: Low-Rank Adaptation via Lower-Diag-Upper Decomposition for Parameter-Efficient Fine-Tuning (Read more on arXiv or HuggingFace) | Chengwei Sun, Ran Ran, Yujia Wu, Jiwei Wei, Shiym | a) The research aims to develop a more parameter-efficient fine-tuning (PEFT) method than existing techniques like Low-Rank Adaptation (LoRA). b) The proposed method, LoLDU, leverages Lower-Diag-Upper (LDU) decomposition to initialize and constrain low-rank matrices, optimizing a diagonal matrix for scaling transformations during fine-tuning. c) Experiments across various tasks and model architectures (including LLaMA2, RoBERTa, ViT, and Stable Diffusion) show LoLDU achieves comparable performance to LoRA while using significantly fewer parameters; for example, on image classification using ViT-Base, LoLDU achieves 82.79% mean accuracy with 0.21% of the parameters, while LoRA achieves 76.22% with 6.77%. d) LoLDU offers AI practitioners a more computationally and memory-efficient method for fine-tuning large models, particularly beneficial in resource-constrained environments, without significant performance degradation. Follow-up questions: 1. The paper mentions heuristic initialization for the diagonal matrix. What is the specific impact of different heuristic initialization methods (e.g., constant, uniform, normal) on the performance and stability of LoLDU across different model architectures and datasets? 2. How does the computational cost of the initial LDU decomposition compare to the overall training time saved by LoLDU, particularly for very large models? Does the one-time cost of LDU decomposition become negligible as training progresses? 3. Could the authors elaborate on the integration of LoLDU within different deep learning frameworks and the practical considerations for implementing it in real-world production settings? |
| BenTo: Benchmark Task Reduction with In-Context Transferability (Read more on arXiv or HuggingFace) | Lichao Sun, Ming Li, Hongyu Zhao, zhoutianyi | a) The paper investigates how to reduce the number of tasks in large language model (LLM) benchmarks without significantly impacting evaluation quality. b) The authors propose In-Context Transferability (ICT), a training-free method using in-context learning to estimate task transferability, and Benchmark Task Reduction (BENTO), which formulates task selection as a facility location problem based on the ICT similarity matrix. c) BENTO can reduce the Massive Multitask Language Understanding (MMLU) benchmark to 5% of its original size (3 out of 57 tasks) while inducing only a <4% difference in evaluation accuracy compared to the full benchmark, averaged across nine LLMs. d) This method offers AI practitioners a cost-efficient way to evaluate LLMs, reducing computational overhead while maintaining evaluation reliability. It allows more rapid model assessment by using a smaller, representative subset of benchmark tasks. Follow-up questions: 1. How does the performance of BENTO vary with different hyperparameter settings for in-context learning (number of exemplars, number of trials), particularly when applied to other benchmarks beyond MMLU and FLAN? 2. Given the identified clustering structure of benchmark tasks, could ICT and BENTO be adapted to create more specialized, smaller benchmarks focused on specific LLM capabilities or domains, rather than general-purpose evaluation? 3. How robust is the BENTO-reduced benchmark to adversarial attacks compared to the full benchmark, and are there strategies to mitigate this potential vulnerability while retaining the efficiency gains of task reduction? |
| AERO: Softmax-Only LLMs for Efficient Private Inference (Read more on arXiv or HuggingFace) | Brandon Reagen, Nandan Kumar Jha | a) The paper investigates architectural optimizations for transformer-based decoder-only language models (LLMs) to improve the efficiency of private inference (PI). b) The authors propose AERO, a four-stage framework involving removing LayerNorm and GELU, substituting ReLU, designing a Softmax-only model with reduced FLOPs, and introducing entropy regularization. c) AERO achieved up to 4.23x communication reduction and 1.94x latency improvement for a GPT-2 model (L=12, H=12, d=768) trained on the CodeParrot (Face) dataset with a context length of 128. d) AI practitioners working on private inference can utilize AERO to significantly reduce the communication and latency overheads associated with nonlinear operations in transformer-based LLMs, making PI more practical. The most impactful finding is the effectiveness of the Softmax-only architecture, as it drastically reduces computational overhead while maintaining reasonable performance, demonstrating a promising direction for efficient PI. Follow-up questions: 1. How does the performance of AERO on downstream tasks, such as text classification or question answering, compare to baseline models and other PI-optimized architectures, and does the reduction in nonlinearity affect the model’s ability to generalize? 2. Could the entropy regularization technique be adapted or generalized for other architectures beyond transformer-based LLMs, or for other applications that experience similar issues with entropic overload or collapse? 3. What are the memory implications of AERO during training and inference, particularly for larger models and context lengths, compared to the baselines and SOTA, and how does AERO scale with model size during training and inference in a PI setting? |
| Long-LRM: Long-sequence Large Reconstruction Model for Wide-coverage Gaussian Splats (Read more on arXiv or HuggingFace) | Fujun Luan, Sai Bi, Kai Zhang, Hao Tan, arthurhero | a) The research aims to enable fast and accurate Gaussian Splat (GS) reconstruction of large scenes with wide viewing coverage from long sequences of input images, avoiding per-scene optimization. b) Long-LRM, a novel GS-based Large Reconstruction Model (LRM), is proposed, leveraging a hybrid architecture combining Mamba2 blocks and transformer blocks for efficient long-context reasoning. It also incorporates token merging and Gaussian pruning for improved memory efficiency. c) Long-LRM reconstructs scenes from 32 images at 960x540 resolution in 1.3 seconds on a single A100 80G GPU, achieving a PSNR of 23.86 on the DL3DV-140 benchmark, comparable to optimization-based 3D GS which takes 13 minutes. d) AI practitioners can now leverage a feed-forward model for rapid large-scale scene reconstruction, significantly accelerating applications in 3D content creation and novel view synthesis. The demonstrated ability to process long sequences of high-resolution images efficiently opens possibilities for improved real-time 3D applications. Follow-up questions: 1. What are the limitations of Long-LRM in terms of generalizability to scenes with different fields of view and its performance scaling beyond 32 input images? 2. How does the hybrid architecture’s balance of Mamba2 and transformer blocks impact the trade-off between reconstruction quality and computational efficiency compared to using only transformers or only Mamba2 blocks at different input sequence lengths and resolutions? 3. What are the specific details of the Gaussian pruning strategy employed during training and inference, and how does it impact rendering quality and memory usage at different pruning thresholds? |
| Remember, Retrieve and Generate: Understanding Infinite Visual Concepts as Your Personalized Assistant (Read more on arXiv or HuggingFace) | Xiangyu Yue, Yu-Feng Li, Changsheng Li, Jiaming Han, Hoar012 | a) The paper aims to personalize Multimodal Large Language Models (MLLMs) by enabling them to remember, retrieve, and utilize user-specific visual concepts without continuous retraining. b) The researchers introduce a Retrieval Augmented Personalization (RAP) framework, involving a key-value database to store concept information (image and description), a multimodal retriever, and integration of retrieved information into MLLM input for personalized generation. They also create a specialized dataset for personalized training, leveraging data augmentation and iterative question generation. c) On a personalized image captioning task, RAP-LLaVA achieved an F1-score of 94.97, outperforming finetuning and other personalization baselines. d) AI practitioners can utilize the RAP framework to develop personalized MLLM-based applications that adapt to individual users and their unique visual concepts without requiring model retraining for each new concept. This significantly reduces the computational cost and complexity associated with personalized MLLM development. Follow-up questions: 1. The paper mentions using low-rank adapters for training. How does the choice of adapter method impact the performance and efficiency trade-offs for different-sized MLLMs within the RAP framework? 2. What are the specific architectural details of the multimodal retriever used in RAP, and how does its performance compare to alternative retrieval methods (e.g., different visual encoders, retrieval strategies) on various personalized tasks? 3. What are the privacy implications of storing user-specific data, particularly images and descriptions, within the personalized database, and how does RAP address these concerns? |
| MuVi: Video-to-Music Generation with Semantic Alignment and Rhythmic Synchronization (Read more on arXiv or HuggingFace) | Shengpeng Ji, Ziang Zhang, Xize Cheng, Siqi Zheng, Ruiqi Li | a) The research aims to generate music soundtracks for videos that exhibit both semantic alignment with the video content and rhythmic synchronization with visual dynamics. b) MuVi, a novel framework, uses a non-autoregressive encoder-decoder architecture with a visual adaptor for feature compression and a contrastive music-visual pre-training scheme to enhance rhythmic synchronization. The music decoder is adapted from a pre-trained flow-matching-based music generator. c) MuVi achieved a SIM score of 19.18% for semantic synchronization, outperforming the M²UGen baseline’s 1.41% and a self-baseline trained from scratch (10.71%). d) AI practitioners can leverage MuVi’s architecture and pre-training strategy for generating higher-quality music for videos, enhancing the user experience in multimedia applications by improving the cohesion between audio and visual elements. The paper suggests potential scalability to larger model sizes. Follow-up questions: 1. The paper mentions in-context learning capabilities but reports degraded performance when using them. What specific modifications to the in-context learning approach could improve these results without sacrificing synchronization quality? 2. What are the computational resource requirements and inference latency of MuVi, and how could these be optimized for real-time or near real-time music generation in practical applications? 3. What is the process for collecting and validating the web-crawled video dataset used for training the V2M model, and how does this dataset differ from publicly available datasets claimed to be “insufficient” for this task? More detail on the specifics of this dataset is needed. |
| Do LLMs Have Political Correctness? Analyzing Ethical Biases and Jailbreak Vulnerabilities in AI Systems (Read more on arXiv or HuggingFace) | Isack Lee, hbseong | a) This research investigates whether intentional biases in Large Language Models (LLMs), introduced for safety alignment, create vulnerabilities to jailbreak attacks, and how these vulnerabilities differ across demographic groups. b) The researchers developed PCJailbreak, a method using LLM-generated keyword pairs representing privileged and marginalized groups in conjunction with harmful prompts, to measure jailbreak success rates across different LLMs. They also proposed PCDefense, a prompt-based defense mechanism to mitigate jailbreak attacks without additional inference. c) In GPT-40, jailbreaking success rates differed by 20% between non-binary and cisgender keywords and 16% between white and black keywords, even with identical prompt structures beyond the keywords. d) LLM developers must carefully consider the potential for safety-induced biases to be exploited by malicious actors, necessitating the development and implementation of more robust defense mechanisms against jailbreak attacks, such as prompt-based mitigation techniques that don’t require significant additional compute resources. e) The paper mentions a learning-based jailbreak method, GCG, but doesn’t clearly explain the details of its implementation within their comparative analyses, leaving some ambiguity in how directly their proposed approach compares to established methods. Follow-up questions: 1. How does PCDefense compare in effectiveness to existing defense mechanisms like Guard Models, considering the trade-off between computational cost and robustness? 2. The paper mentions the LLM-generated keywords - what specific prompts were used to generate these keywords, and what is the degree of variation in the generated keywords between different LLMs? 3. Could the observed discrepancies in jailbreak success rates be attributed to factors other than intentional bias, such as differences in the frequency or context of these keywords within the training data? |
| SBI-RAG: Enhancing Math Word Problem Solving for Students through Schema-Based Instruction and Retrieval-Augmented Generation (Read more on arXiv or HuggingFace) | Tim Oates, pdx97 | a) The research aimed to enhance math word problem (MWP) solving by improving reasoning clarity and accuracy through schema-based instruction and retrieval-augmented generation (RAG). b) A schema classifier (DistilBERT) predicted problem schema, guiding schema-specific prompt generation for RAG using a Llama 3.1 LLM; solutions were compared against GPT-3.5-Turbo and GPT-4 using a novel “reasoning score” and LLM-as-a-Judge evaluations. c) The SBI-RAG system achieved a higher average reasoning score (0.588) compared to GPT-4 (0.491) and GPT-3.5-Turbo (0.290). d) AI practitioners can leverage schema-guided RAG and structured prompts to improve the transparency and reasoning capabilities of LLMs for educational applications like MWP solving. The impactful finding of improved reasoning scores suggests potential for enhanced educational effectiveness through structured, schema-driven prompting. Follow-up questions: 1. What were the specific hyperparameters used for fine-tuning the DistilBERT schema classifier, and how was its performance validated beyond accuracy (e.g., using cross-validation)? The paper provides limited details on the training configuration and evaluation. 2. How was the “reasoning score” metric precisely calculated? While the general concept is explained, details on weighting, normalization, and specific implementation are unclear. 3. What was the composition and size of the document set used for context retrieval, and how did its content specifically relate to the GSM8K dataset? More detail on the context source would be beneficial. |
| $γ-$MoD: Exploring Mixture-of-Depth Adaptation for Multimodal Large Language Models (Read more on arXiv or HuggingFace) | Xiaoshuai Sun, Yiyi Zhou, Jiayi Ji, Gen Luo, YaxinLuo | a) The paper investigates how to reduce the computational cost of Multimodal Large Language Models (MLLMs) while maintaining performance, focusing on minimizing “activated tokens” rather than parameters. b) The authors propose γ-MoD, a plug-and-play adaptation strategy integrating Mixture-of-Depths (MoDs) into existing MLLMs. A novel metric called Rank of Attention Maps (ARank) guides MoD layer placement, complemented by a shared vision-language router and masked routing learning to optimize token skipping. c) γ-MoD achieved a 51.6% reduction in FLOPs and a 53.2% inference time speedup on LLaVA-HR with an average performance decrease of only 1.5% across four benchmark datasets (GQA, SQA, MMMU, TextVQA). d) AI practitioners can use γ-MoD to significantly improve the efficiency of existing MLLMs during both training and inference with minimal performance trade-offs, facilitating deployment in resource-constrained environments. The plug-and-play nature and demonstrated generalizability across different MLLM architectures and sizes simplify integration into existing workflows. Follow-up questions: 1. How does the performance of γ-MoD compare to other sparsity techniques like MoEs when applied to other, more complex MLLM architectures, particularly those designed for high-resolution image inputs? 2. The paper mentions ARank being calculated after pre-training. Could ARank be dynamically updated during fine-tuning or even inference to further adapt to specific tasks or input distributions? What are the computational implications of such dynamic ARank updates? 3. What are the memory access patterns and implications of using γ-MoD, and how could these be optimized for specific hardware architectures like GPUs to maximize the realized efficiency gains? |
| Toward Guidance-Free AR Visual Generation via Condition Contrastive Alignment (Read more on arXiv or HuggingFace) | Jun Zhu, Peize Sun, Hang Su, ChenDRAG | a) The research aims to improve autoregressive (AR) visual generation by removing the reliance on computationally expensive classifier-free guidance (CFG) while maintaining high sample quality. b) The paper proposes Condition Contrastive Alignment (CCA), a fine-tuning method that contrasts positive and negative image-condition pairs to align pretrained AR models to a target sampling distribution equivalent to that achieved by CFG. c) CCA significantly improves the FID score of a LlamaGen-L (343M parameter) model from 19.07 to 3.41 and the IS score from 64.3 to 288.2 after one epoch of fine-tuning on ImageNet, achieving near-CFG performance without guided sampling. d) AI practitioners can use CCA to reduce the computational cost of AR visual generation by approximately half compared to CFG, potentially simplifying the implementation and deployment of these models. Follow-up questions: 1. How does CCA’s performance compare to CFG when evaluated on other datasets beyond ImageNet, particularly those with more complex scenes or different image resolutions? 2. While CCA eliminates the need for a separate unconditional model during sampling, it still appears to require one during training. Could the training procedure be modified to completely remove this dependency? 3. The paper mentions combining CCA with CFG. Are there specific guidelines for selecting hyperparameters in this combined approach to achieve optimal performance, and what are the practical computational cost implications of this hybrid method? |
| Can MLLMs Understand the Deep Implication Behind Chinese Images? (Read more on arXiv or HuggingFace) | Xinrun Du, Yuelin Bai, Xi Feng, zhangysk, MING-ZCH | a) The research evaluates the ability of Multimodal Large Language Models (MLLMs) to understand higher-order implications and cultural nuances within Chinese images. b) A new benchmark, CII-Bench, containing 698 Chinese images and 800 multiple-choice questions across six domains, was created and used to evaluate several MLLMs and LLMs with varying prompt configurations. Human evaluation was also included for comparison. c) The highest accuracy achieved by an MLLM on CII-Bench was 64.4%, significantly lower than the average human accuracy of 78.2%. d) MLLMs struggle with complex cultural elements in Chinese imagery and emotion understanding, significantly impacting their performance in accurately interpreting implicit meanings; therefore, AI practitioners should focus on improving MLLMs’ ability to process complex cultural context and nuanced emotional information within visual content. Follow-up questions: 1. What specific architectural modifications or training strategies could be employed to enhance MLLMs’ understanding of culturally specific imagery and symbolism? 2. How can the evaluation metric based on GPT-4 for Chinese traditional paintings be further refined to provide more granular insights into the specific areas where MLLMs struggle with cultural understanding? 3. Does the paper offer any insight into the transferability of these findings to other cultures or languages with visually rich and implicit communication styles? |
| Minimum Tuning to Unlock Long Output from LLMs with High Quality Data as the Key (Read more on arXiv or HuggingFace) | Yunlin Mao, Jintao Huang, Daoze, wangxingjun778, Yingda | This research investigates how data quality impacts the tuning of large language models (LLMs) for generating long-form text outputs. The authors curated a high-quality dataset (LongWriter-6K-filtered) by removing entries from an existing dataset (LongWriter-6K) that lacked output length specifications or had large discrepancies between requested and actual output length. Tuning Qwen2-7B-Instruct with the curated 666-sample dataset resulted in a 9.22 point improvement in the combined length and quality score compared to using the original LongWriter-6K dataset. This indicates that high-quality, task-aligned data is crucial for efficiently tuning LLMs for long output generation, enabling comparable performance improvements with significantly less training data. The authors do not clearly specify how the 9.22-point improvement is calculated or what the absolute starting score was. Follow-up questions: 1. How is the combined length and quality score (S) calculated, and what were the baseline S scores for the untuned models used in the experiments? 2. Could the authors elaborate on the computational cost savings achieved using the smaller, curated dataset compared to the larger, original dataset, and how this translates into practical benefits for LLM deployment? 3. What specific techniques were used for data cleansing beyond removing entries based on missing length or length discrepancies, and how were these chosen? |
| TransAgent: Transfer Vision-Language Foundation Models with Heterogeneous Agent Collaboration (Read more on arXiv or HuggingFace) | Yali Wang, Yu Qiao, Kunchang Li, Shaobin Zhuang, markywg | a) The research aims to improve the generalization ability of vision-language foundation models (VLMs), such as CLIP, in low-shot transfer learning scenarios. b) TransAgent, a framework leveraging multi-source knowledge distillation, transfers knowledge from 11 heterogeneous vision, language, and multi-modal “agents” (pre-trained models) to enhance CLIP. This is achieved through layer-wise feature distillation, class-specific feature distillation, and score distillation, combined with a mixture-of-agents gating mechanism for knowledge integration. c) On 11 visual recognition benchmarks under a base-to-novel generalization setting, TransAgent, using CLIP ViT-B/16, outperforms CoOp by approximately 10% on average and 20% on EuroSAT. d) AI practitioners can leverage TransAgent to improve the performance of CLIP-like models in diverse downstream tasks, particularly under low-shot conditions, without incurring additional computational cost in the inference phase due to the distillation approach. The paper does not explicitly detail the computational cost of the training/distillation phase. Follow-up questions: 1. What is the computational overhead of the TransAgent training process compared to standard prompt tuning methods, and what are the trade-offs in terms of resource utilization? 2. How does the performance of TransAgent scale with the number and diversity of the incorporated agent models, and are there limitations to integrating an even wider range of agents? 3. Could the TransAgent framework be adapted for other VLM architectures beyond CLIP, and what modifications would be necessary? |
Papers for 2024-10-17
| Title | Authors | Summary |
|---|---|---|
| HumanEval-V: Evaluating Visual Understanding and Reasoning Abilities of Large Multimodal Models Through Coding Tasks (Read more on arXiv or HuggingFace) | Xiao Li, Guancheng Lin, Huiyu Bai, Linquan Wu, zfj1998 | a) The paper investigates the visual understanding and reasoning abilities of Large Multimodal Models (LMMs) in coding tasks that require visual context. b) The researchers created HumanEval-V, a benchmark of 108 Python coding tasks adapted from existing problems and requiring LMMs to generate code solutions based on images and function signatures, evaluated using pass@k metrics. c) State-of-the-art LMMs performed below expectations, with even proprietary models like GPT-4o achieving only 13% pass@1 on HumanEval-V. d) AI practitioners developing LMMs should focus on improving models’ visual understanding and reasoning as well as coding proficiencies, as current models demonstrate significant weaknesses in integrating these skills. e) The paper notes a consistent performance degradation in open-weight LMMs compared to their language-only decoder counterparts on coding benchmarks, highlighting a need for further improvement in multimodal training strategies. Follow-up questions: 1. The paper mentions “hallucination errors” due to overfitting. Could the authors elaborate on the specific types of hallucinations observed and how they relate to the adaptation process used in creating HumanEval-V? 2. Given the limited improvement from zero-shot Chain-of-Thought prompting, what other reasoning or prompting techniques could be explored to better assist LMMs in solving these visual coding tasks? 3. What specific architectural changes or training strategies could be implemented to address the performance degradation observed in open-weight LMMs compared to their decoder counterparts on coding tasks? |
| VidEgoThink: Assessing Egocentric Video Understanding Capabilities for Embodied AI (Read more on arXiv or HuggingFace) | Sicheng Zhou, Yangyang Yu, Kechen Fang, yetian, SijieCheng | a) The research assesses the capabilities of Multi-modal Large Language Models (MLLMs) in understanding egocentric videos for application in Embodied AI tasks. b) A new benchmark, VidEgoThink, was created with four interrelated tasks: video question-answering, hierarchy planning, visual grounding, and reward modeling; data was generated using Ego4D and GPT-40, then filtered by human annotators; and 14 MLLMs across three categories (API-based, open-source image-based, and open-source video-based) were evaluated. c) MLLMs performed poorly across all tasks, with the best average accuracy on video question-answering reaching only 32.82% across all dimensions. d) The findings indicate current MLLMs require significant improvement for effective application in first-person scenarios in Embodied AI, particularly in understanding temporal dynamics and generating actionable outputs, despite having certain potential for advancement. Follow-up Questions: 1. Given the poor performance on temporal reasoning tasks, what specific architectural modifications or training strategies could be explored to improve MLLMs’ ability to understand action sequences and temporal relations in egocentric videos? 2. The paper mentions an automatic data generation pipeline; it would be useful to know more specific details of this pipeline. Could the authors elaborate on the specific prompts used for GPT-40 and the filtering criteria employed by the human annotators to improve replicability and allow further exploration of this data generation approach? 3. The paper briefly mentions future work on developing egocentric foundation models for robotics. What specific robotic tasks are the authors envisioning these models being applied to, and what are the key challenges they anticipate in adapting VidEgoThink or similar benchmarks for evaluating these specialized models? |
| The Curse of Multi-Modalities: Evaluating Hallucinations of Large Multimodal Models across Language, Visual, and Audio (Read more on arXiv or HuggingFace) | Hang Zhang, Yang Zhou, Yun Xing, Sicong Leng, ClownRat | a) This paper investigates the causes and prevalence of hallucinations in Large Multimodal Models (LMMs) processing language, visual, and audio data. b) A new benchmark called “The Curse of Multi-Modalities” (CMM) was created, using object/event-level probing questions in a binary classification framework to evaluate LMM performance across various multimodal contexts and hallucination subcategories. c) LMMs exhibit significant vulnerabilities to Audio-Language (AL) hallucinations, with Gemini-1.5-pro achieving only a 14.5% Hallucination Resistance (HR) score in this category. d) AI practitioners should prioritize addressing spurious inter-modality correlations, especially those involving audio, and mitigate the overreliance on unimodal priors when developing and deploying LMMs. The specific training strategies mentioned (balanced multi-modal training data, advanced cross-modal fusion, mitigating linguistic priors, and refined safety alignment) could be beneficial. Follow-up Questions: 1. The paper highlights the limited availability of visual-audio-language datasets as a potential reason for stronger AL correlations. Are there recommended strategies or resources for constructing or augmenting such datasets to improve AL hallucination resistance? 2. Could the authors elaborate on the specific implementation details of the “dynamic fusion strategies” mentioned as a potential improvement for cross-modal fusion? What are some promising architectures or approaches for achieving more context-aware modality integration? 3. The paper identifies varying response tendencies in different LMMs (overconfidence vs. excessive caution). Are there specific evaluation metrics or techniques beyond PA and HR that could be used to better characterize and compare these tendencies, enabling a more nuanced understanding of their impact on downstream tasks? |
| Revealing the Barriers of Language Agents in Planning (Read more on arXiv or HuggingFace) | Kai Zhang, Siyu Yuan, jiangjiechen, kexunz, hsaest | This paper investigates why language agents struggle with planning tasks. Permutation Feature Importance (PFI) analysis of constraint and question components within prompts was used. The results show that constraints have a limited role, and the influence of the question decreases with increasing planning horizon; OpenAI’s 01 model achieves only 15.6% on the TravelPlanner benchmark. This implies that current memory updating strategies for language agents, while offering some improvements, resemble “shortcut learning” and do not fully address the core issues of constraint integration and long-horizon goal maintenance. Follow up questions: 1. How does the PFI analysis method account for the variability in the natural language generation process of LLMs across different prompts and trials? 2. How can the insights regarding the limitations of episodic and parametric memory updating inform the development of more effective memory mechanisms for language agents specifically aimed at improving planning performance? 3. Can the observed weakness in constraint handling be addressed by incorporating symbolic planning techniques within the LLM framework for agent planning? |
| DocLayout-YOLO: Enhancing Document Layout Analysis through Diverse Synthetic Data and Global-to-Local Adaptive Perception (Read more on arXiv or HuggingFace) | Conghui He, Bin Wang, Hengrui Kang, Zhiyuan Zhao | a) The research aims to improve the speed and accuracy of Document Layout Analysis (DLA) by addressing the trade-off between multimodal and unimodal methods. b) The authors introduce DocLayout-YOLO, which uses a synthetic dataset (DocSynth-300K) generated by their Mesh-candidate BestFit algorithm and integrates a Global-to-Local Controllable Receptive Module (GL-CRM) within a YOLOv10 architecture. c) DocLayout-YOLO achieved 78.8% mAP on the DocStructBench dataset with an inference speed of 85.5 frames per second (FPS). d) AI practitioners can leverage DocLayout-YOLO for real-time, accurate DLA in applications such as document parsing, information retrieval, and knowledge extraction, benefiting from its improved speed and accuracy compared to previous methods. Follow-Up Questions: 1. What are the details of the GL-CRM’s integration with the YOLOv10 architecture, and how does this module specifically contribute to the improved handling of multi-scale elements? 2. While the paper mentions that DocSynth-300K offers improved diversity, what are the limitations of this synthetic dataset, particularly when dealing with extremely complex or unusual document layouts not well-represented in the training data? 3. Can the Mesh-candidate BestFit algorithm be adapted for other layout generation tasks beyond document layout analysis, such as webpage layout or UI design? |
| Exploring Model Kinship for Merging Large Language Models (Read more on arXiv or HuggingFace) | Huajun Chen, Shumin Deng, Ningyu Zhang, Yunzhi Yao, Yedi Hu | a) This research investigates whether a metric called “model kinship” (similarity between LLMs based on weight differences from a base model) can guide and improve the performance of iterative LLM merging. b) The researchers analyzed open-source LLMs using Pearson Correlation, Cosine Similarity, and Euclidean Distance to calculate model kinship, correlating it with merging performance gains and examining its behavior across different merging stages. They also proposed a “Top-k Greedy Merging with Model Kinship” strategy that incorporates kinship into model selection for merging. c) A statistically significant correlation was found between the absolute value of merge gain and model kinship. Using the kinship-guided merging strategy, the researchers achieved an average task performance of 69.13 across six tasks, compared to 68.72 using a standard greedy strategy. It is unclear why the results focus on absolute merge gain rather than merge gain itself, and the choice and impact of merging six specific tasks is also not explained. d) AI practitioners can utilize model kinship to guide model selection during iterative merging, potentially escaping local optima and achieving higher performance gains on multi-task learning benchmarks. Using model kinship also offers potential as an early stopping criterion in iterative merging, improving resource efficiency. Follow-up questions: 1. How does the choice of the base model affect the calculation and interpretation of model kinship, and what are best practices for base model selection? 2. Beyond the six tasks used in this study, how does model kinship generalize to broader sets of tasks or different task domains, and what are the limitations of its applicability? 3. Can the concept of model kinship be extended to guide other LLM combination techniques beyond simple weight averaging, such as knowledge distillation or parameter fusion? |
| Large Language Model Evaluation via Matrix Nuclear-Norm (Read more on arXiv or HuggingFace) | Yi Chang, Yahan Li, WhiteCatY, xiatingyu | This research aimed to develop a more computationally efficient metric for evaluating information compression and redundancy reduction in Large Language Models (LLMs). The researchers proposed using the Matrix Nuclear-Norm, approximated by the L1,2-norm, as a computationally less expensive alternative to Matrix Entropy. Results showed the Matrix Nuclear-Norm achieved speeds 8 to 24 times faster than Matrix Entropy for the CEREBRAS-GPT model with increasing sizes from 111M to 6.7B parameters. This improvement allows AI practitioners to more efficiently evaluate LLMs, especially as model sizes continue to scale, making the Matrix Nuclear-Norm a potentially practical choice for assessing compression capabilities. The paper does not definitively state whether Matrix Nuclear-Norm and Matrix Entropy yield comparable evaluation accuracy despite the stated claim of “comparable accuracy”. Follow-up questions: 1. While the paper demonstrates computational efficiency gains, how does the Matrix Nuclear-Norm’s correlation with downstream task performance compare to Matrix Entropy’s? 2. The paper mentions anomalies in Matrix Nuclear-Norm values for certain model sizes (2.7B and 13B). What are the potential underlying reasons for these anomalies and how might they affect the metric’s reliability in evaluating these specific models? 3. How sensitive is the Matrix Nuclear-Norm to the choice of L1,2-norm approximation, and are there alternative approximations that might improve its accuracy or stability further? |
| ProSA: Assessing and Understanding the Prompt Sensitivity of LLMs (Read more on arXiv or HuggingFace) | Dahua Lin, Xinyu Fang, KennyUTC, zsytony, JingmingZ | a) The research aimed to evaluate and understand prompt sensitivity in large language models (LLMs) at the instance level. b) ProSA, a framework incorporating the PromptSensiScore (PSS) metric and leveraging decoding confidence, was developed. c) Results across multiple datasets and models revealed variations in prompt sensitivity, with Llama3-70B-Instruct exhibiting the highest robustness and Qwen1.5-14B-Chat demonstrating the most serious prompt sensitivity on the MATH dataset. d) Higher model confidence correlated with increased prompt robustness, suggesting prompt sensitivity reflects the model’s decoding logic. This finding provides a new metric for evaluating LLM robustness and emphasizes the importance of considering prompt engineering and selection strategies in development and applications. Follow-up Questions: 1. How does the ProSA framework compare with existing methods for evaluating prompt sensitivity in terms of computational cost and insights provided? 2. Could the decoding confidence be used as a signal for automated prompt optimization or selection? 3. How does the observed correlation between model size and prompt sensitivity vary across different model architectures (e.g., decoder-only vs. encoder-decoder)? |
| ZipVL: Efficient Large Vision-Language Models with Dynamic Token Sparsification and KV Cache Compression (Read more on arXiv or HuggingFace) | Wenqi Shao, Jing Liu, Feng Chen, Yefei He, kpzhang996 | a) The research aims to improve the efficiency of Large Vision-Language Models (LVLMs) by addressing computational bottlenecks in the prefill phase and memory bottlenecks in the decoding phase. b) ZipVL employs a dynamic, layer-wise adaptive ratio assignment for important tokens based on attention score distribution, combined with token-level sparse attention in the prefill phase and mixed-precision KV cache quantization in the decoding phase. c) Experiments demonstrate a 2.6× speedup in the prefill phase and a 50.0% reduction in GPU memory usage on the LongVA-7B model for the Video-MME benchmark, with a 0.2% accuracy reduction. d) AI practitioners can leverage ZipVL to significantly improve the inference speed and reduce the memory footprint of LVLMs, facilitating their deployment in resource-constrained environments. The dynamic ratio assignment, in particular, offers a more robust and adaptive approach compared to fixed sparsity methods. Follow-up Questions: 1. What are the specific implementation details regarding the integration of ZipVL with different fast attention mechanisms besides FlashAttention? 2. How does the performance of ZipVL scale with increasing video lengths or image resolutions, particularly with regards to the trade-off between computational cost and accuracy? 3. Could the dynamic ratio allocation strategy be further improved by incorporating factors beyond attention scores, such as textual context or visual saliency? |
| Improving Long-Text Alignment for Text-to-Image Diffusion Models (Read more on arXiv or HuggingFace) | Chongxuan Li, Zehan Wang, Tianyu Pang, Chao Du, luping-liu | a) This research addresses the challenge of aligning text-to-image (T2I) diffusion models with long, complex text prompts, which often exceed the token limits of standard encoders like CLIP and result in incomplete or inaccurate image generation. b) The authors propose LongAlign, combining segment-level encoding, which divides long text into segments and processes them individually, with a decomposed preference optimization method that fine-tunes diffusion models using a reweighted combination of text-relevant and text-irrelevant preference scores derived from a modified CLIP-based model. c) The fine-tuned Stable Diffusion (SD) v1.5 model, after 20 hours of training using LongAlign on 6 A100 GPUs, achieves a FID score of 19.63 on a 5k image dataset, outperforming baseline foundation models like PixArt-a and Kandinsky v2.2 in long-text alignment. d) AI practitioners can leverage LongAlign to improve the fidelity of T2I generation from detailed text prompts by overcoming input length limitations and enhancing alignment between text and generated images. The decomposition of preference scores during fine-tuning helps mitigate overfitting, a common issue in reward-based optimization of diffusion models. Follow-up questions: 1. What are the specific implementation details for merging the segment embeddings in LongAlign, especially regarding the choice of concatenation versus other aggregation methods, and how does this impact the computational complexity? 2. How does the reweighting factor w in the gradient-reweight reward fine-tuning affect the trade-off between text alignment and visual quality (e.g., aesthetics, photorealism), and is there a systematic method for determining the optimal w value for different datasets and models? 3. How robust is LongAlign to variations in text segmentation strategies (e.g., sentence-level versus semantic chunk-level segmentation), and what preprocessing steps are necessary to ensure consistent performance across diverse text formats and domains? |
| Simplifying, Stabilizing and Scaling Continuous-Time Consistency Models (Read more on arXiv or HuggingFace) | Yang Song, Cheng Lu | a) This research aims to improve the training stability and scalability of continuous-time consistency models (CMs) for fast generative sampling. b) The authors introduce TrigFlow, a simplified theoretical framework unifying diffusion and CM formulations, alongside improved network architecture, time-conditioning, and training objectives incorporating tangent normalization and adaptive weighting. They also enhance Jacobian-vector product computation for Flash Attention to improve training efficiency. c) The resulting simplified CMs (sCMs) achieved a 2-step FID score of 1.88 on ImageNet 512x512 with 1.5 billion parameters, narrowing the gap to state-of-the-art diffusion models to within 10%. d) AI practitioners can leverage these stabilized and scalable continuous-time CMs for high-quality image generation with significantly reduced sampling compute compared to traditional diffusion models. The simplification provided by TrigFlow could also make CMs more accessible for development and analysis. Follow-up questions: 1. Could the TrigFlow framework be adapted for other data modalities beyond images, such as audio or 3D models, and what modifications might be necessary? 2. What are the practical memory and compute requirements for training sCMs at the reported scale, and how do they compare to training comparable diffusion models? 3. How sensitive are the sCM results to the hyperparameters introduced for tangent normalization and adaptive weighting, and are there recommended starting points for tuning these on new datasets? |
| Insights from the Inverse: Reconstructing LLM Training Goals Through Inverse RL (Read more on arXiv or HuggingFace) | Sonali Parbhoo, Arjun Jagota, Jared Joselowitz, skrishna | This research investigated whether Inverse Reinforcement Learning (IRL) can recover the reward functions underlying the training of Large Language Models (LLMs) fine-tuned with Reinforcement Learning from Human Feedback (RLHF). The researchers applied a Max-Margin IRL algorithm to extract reward models from toxicity-aligned LLMs of varying sizes (70M and 410M parameters), trained on a subset of the Jigsaw toxicity dataset. The extracted reward model for the 70M parameter LLM achieved 80.40% accuracy in predicting human preferences on a held-out test set. This indicates that, at least for smaller models and specific tasks, IRL can extract reward models that capture key aspects of the original RLHF objective, which has implications for interpretability and potential vulnerability analysis. The paper mentions challenges with the non-identifiability of reward functions and potential scalability issues for larger LLMs but does not fully elaborate on mitigations or solutions. Follow-up questions: 1. How does the performance of the proposed Max-Margin IRL method compare to other IRL techniques, such as Max-Entropy or adversarial IRL, in extracting reward models from RLHF-trained LLMs, especially for larger models and more complex reward structures? 2. What specific mitigation strategies are proposed to address the non-identifiability of the recovered reward functions, and how do these impact the reliability and interpretability of the extracted models for practical applications like debugging or bias detection? 3. Given the potential for misuse of extracted reward models, what concrete recommendations would the researchers offer for responsible disclosure and use of these models within the broader AI community? |
| Neural Metamorphosis (Read more on arXiv or HuggingFace) | Xinchao Wang, Xingyi Yang | This paper aims to create self-morphable neural networks adaptable to various sizes without retraining. The key methodology involves training a neural implicit function (INR) as a hypernetwork to learn the continuous weight manifold of neural networks, incorporating strategies for intra- and cross-network smoothness. On CIFAR10 image classification, the proposed method, NeuMeta, achieved 91.76% accuracy with a full-sized ResNet20 and 89.56% accuracy at a 75% compression rate, often outperforming individually trained models at smaller sizes. This implies that AI practitioners could potentially achieve significant model compression without retraining or substantial performance loss. Follow-up questions: 1. How does the computational cost of using the INR to generate weights compare to the cost of fine-tuning a pruned model or training a smaller model from scratch, especially for very large networks? 2. The paper mentions limitations in the INR’s representational ability for complex tasks like segmentation; how might these limitations be addressed to improve performance on such tasks at higher compression rates? 3. Could NeuMeta be extended to enable dynamic morphing of network architectures during inference based on resource availability or input characteristics? |
| WorldMedQA-V: a multilingual, multimodal medical examination dataset for multimodal language models evaluation (Read more on arXiv or HuggingFace) | Juan Carlos Climent Pardo, Yingya Li, Siena Placino, João Matos, shanchen | a) The research aimed to create and evaluate a multilingual, multimodal benchmark dataset to assess vision-language models (VLMs) in healthcare question answering (QA). b) Researchers collected multiple-choice medical exam questions from Brazil, Israel, Japan, and Spain, pairing them with images and validating English translations. They then evaluated the performance of 10 open and closed-source VLMs with and without image input, using accuracy as the metric, and calculated Cohen’s kappa for cross-linguistic consistency. c) GPT4o achieved the highest accuracy across most datasets, but only reached 58% accuracy on the Hebrew version of the Israeli dataset. d) The results indicate a need for improvement in VLMs’ ability to handle diverse languages, especially those underrepresented in training data, as demonstrated by lower performance in non-Roman alphabet languages like Hebrew. The impact of image input varied significantly across model families, with Gemini models showing the largest performance gains. Follow-up questions: 1. What specific pre-training datasets were used for the evaluated VLMs, and what is their representation of different languages and medical concepts? 2. How does the performance of the VLMs on this multiple-choice dataset compare to their performance on other medical QA tasks, such as free-text generation or information retrieval? 3. Beyond accuracy and Cohen’s Kappa, what other metrics (e.g., calibration, robustness, fairness) would be relevant to evaluate VLMs in this context, and were they examined in the research? |
| OMCAT: Omni Context Aware Transformer (Read more on arXiv or HuggingFace) | Andrew Tao, Rafael Valle, Matthieu Le, Karan Sapra, goarushi27 | a) This research aims to improve cross-modal temporal understanding in multimodal Large Language Models (LLMs), particularly the ability to correlate events across audio and video streams. b) The authors introduce a new dataset, OCTAV (Omni Context and Temporal Audio Video), designed to capture event transitions across audio and video, and a new model, OMCAT (Omni Context Aware Transformer), which leverages Rotary Time Embeddings (ROTE) for enhanced temporal grounding. OMCAT is trained using a three-stage pipeline: feature alignment, instruction tuning, and OCTAV-specific training. c) OMCAT achieves state-of-the-art performance on Audio-Visual Question Answering (AVQA) tasks, outperforming existing models by a substantial margin on the OCTAV benchmark (19.0% Recall@1 IoU 0.7 on OCTAV-ST-ActivityNet for OMCAT vs 1.57% for GroundingGPT). It also shows competitive results in zero-shot settings. d) AI practitioners can leverage OMCAT and the OCTAV dataset to develop more robust multimodal applications requiring fine-grained temporal understanding, such as video analysis, content creation, and interactive media. The improved performance on time-anchored tasks directly enhances the ability of LLMs to understand and generate temporally consistent responses in multimodal contexts. Follow-up questions: 1. What are the computational costs and scalability implications of ROTE compared to other temporal embedding methods, especially when applied to longer videos or higher-resolution data? 2. How does the performance of OMCAT degrade with noisier or more ambiguous audio-visual data, which is common in real-world scenarios not represented in the artificially constructed OCTAV dataset? 3. Can the ROTE embeddings be effectively generalized to other multimodal tasks beyond audio-visual understanding, such as integrating text, images, and sensor data with time dependencies? |
| Tracking Universal Features Through Fine-Tuning and Model Merging (Read more on arXiv or HuggingFace) | Desmond Elliott, nilq | a) This research investigates how features in one-layer Transformer language models evolve (emerge, disappear, persist) during fine-tuning to new domains and model merging via spherical linear interpolation. b) The study uses small-scale Mistral-like Transformers trained on English text and programming code (Python and Lua), with feature extraction performed using sparse autoencoders analyzing MLP activations. c) Few features persist across fine-tuning and merging, though persistent features often correspond to generic text properties like punctuation and formatting (e.g., a variable assignment feature maintained an average 85.1% cross-correlation across models). d) AI practitioners can leverage these findings to understand feature dynamics when adapting existing models for new domains or tasks using fine-tuning and merging techniques. The low feature persistence suggests that substantial feature change is expected when applying these techniques, and monitoring/analysis of these changes may be crucial. Follow-up Questions: 1. How do the findings generalize to larger, more complex Transformer models used in real-world applications? 2. Are there alternative merging techniques or hyperparameter settings that could improve feature retention during merging? 3. Could controlling or manipulating these evolving features during fine-tuning and merging lead to more robust and adaptable models? |
| DyVo: Dynamic Vocabularies for Learned Sparse Retrieval with Entities (Read more on arXiv or HuggingFace) | Jeff Dalton, Iain Mackie, Sean MacAvaney, Shubham Chatterjee, Thong Nguyen | This paper investigates whether incorporating entities into learned sparse retrieval (LSR) improves its effectiveness. The researchers introduce a Dynamic Vocabulary (DyVo) head, which uses entity embeddings and an entity retrieval component to generate entity weights, merged with word piece weights to create joint representations. On the CODEC dataset, DyVo with GPT-4 generated entity candidates achieves an nDCG@10 of 56.46, compared to 52.61 for LSR without entities. This implies that augmenting LSR with dynamically retrieved entities can improve retrieval effectiveness, especially in entity-rich datasets. AI practitioners working with LSR can use the DyVo head to expand vocabularies with entities from external knowledge bases, potentially increasing performance. Follow-up questions: 1. What is the computational overhead of the entity retrieval component, especially at scale with large knowledge bases? 2. How robust is the method to different entity embedding sources, and how can embedding quality be efficiently evaluated within this framework? 3. What strategies could be employed to further reduce the dependence on computationally expensive large language models for candidate generation during training and inference? |
Papers for 2024-10-16
| Title | Authors | Summary |
|---|---|---|
| MLLM can see? Dynamic Correction Decoding for Hallucination Mitigation (Read more on arXiv or HuggingFace) | Haoming Xu, Bozhong Tian, Xiang Chen, Chenxi Wang, Ningyu | a) This research investigates the mechanism of hallucinations in Multimodal Large Language Models (MLLMs) and proposes a mitigation method. b) The authors analyze MLLM behavior through object probing, probability analysis across transformer layers, and early exit experiments, then introduce Dynamic Correction Decoding with preCeding-Layer Knowledge (DeCo). DeCo dynamically selects preceding layers with higher ground truth token confidence and integrates their knowledge into the final layer output logits. c) DeCo reduces hallucination rates on the CHAIR benchmark by an average of 10.8% compared to baselines across various MLLMs and decoding strategies. d) AI practitioners can use DeCo as a training-free decoding method to mitigate hallucinations in MLLMs during inference, potentially improving the reliability of generated content in image captioning and VQA tasks. This is particularly relevant for applications where factual accuracy is critical. Follow-up questions: 1. How does DeCo’s performance compare to existing training-based hallucination mitigation methods in terms of both accuracy and computational cost? 2. Can DeCo be effectively combined with other decoding strategies or post-processing methods for further hallucination reduction? 3. What are the limitations of DeCo in handling other types of hallucinations beyond object hallucinations, such as incorrect attribute assignment or relationship descriptions? |
| MTU-Bench: A Multi-granularity Tool-Use Benchmark for Large Language Models (Read more on arXiv or HuggingFace) | Xiaoshuai Song, Jiaheng Liu, Zekun Wang, Yanan Wu, Pei Wang | a) This research aimed to create a benchmark for evaluating Large Language Model (LLM) performance on diverse real-world tool-use tasks. b) The authors developed MTU-Bench, consisting of MTU-Instruct (a training dataset derived from existing dialogue datasets and synthesized tool calls) and MTU-Eval (an automatic evaluation framework with fine-grained metrics). c) Their fine-tuned model, MTU-LLaMA, achieved a tool selection accuracy of 92.31% on single-turn, single-tool tasks in the normal test set. d) AI practitioners can use MTU-Bench to more comprehensively evaluate and improve the tool-use capabilities of LLMs, particularly in complex multi-turn and multi-tool scenarios. The demonstrated superior performance of MTU-LLaMA across multiple settings indicates its potential for more robust tool integration in real-world applications. Follow-up questions: 1. How does the performance of MTU-LLaMA compare to other state-of-the-art tool-learning models on benchmarks beyond MTU-Bench? 2. What specific types of errors are most prevalent in the hard test set, and how can these insights guide future model development to improve robustness? 3. Could the automated data synthesis pipeline be adapted for other types of tasks beyond tool use, such as code generation or reasoning? |
| LLM$\times$MapReduce: Simplified Long-Sequence Processing using Large Language Models (Read more on arXiv or HuggingFace) | Yu Chao, Xinyi Chen, Chong Li, Zihan Zhou, shuo-hf | a) The research aims to improve long-text processing in Large Language Models (LLMs) by mitigating the loss of long-range information when using divide-and-conquer strategies. b) The proposed LLM×MapReduce framework employs a three-stage process (map, collapse, reduce) augmented by a structured information protocol and in-context confidence calibration. c) On the InfiniteBench benchmark, LLM×MapReduce achieved an average score of 68.66%, outperforming closed-source models like GPT-4 (57.34%) and other open-source models. d) AI practitioners can utilize this training-free method to extend the effective context window of LLMs, enhancing performance on tasks requiring the comprehension of long sequences without needing extensive computational resources or retraining. The significant performance improvement over existing methods makes LLM×MapReduce a viable solution for long-text applications. Follow-up questions: 1. What are the specific prompt engineering techniques used in each stage (map, collapse, reduce) of LLM×MapReduce, and how can these be adapted for different downstream tasks? 2. How does the computational cost of LLM×MapReduce, including the multiple inference calls, compare to the cost of training LLMs with extended context windows using methods like LongLoRA or adjusting RoPE frequencies? What are the tradeoffs? |
| SecCodePLT: A Unified Platform for Evaluating the Security of Code GenAI (Read more on arXiv or HuggingFace) | Wenbo Guo, Yuheng Tang, Zhun Wang, Yuzhou Nie, yuyangy | a) The research aims to develop a comprehensive platform for evaluating the security risks of code generation AI models in both insecure code generation and facilitation of cyberattacks. b) SECCODEPLT utilizes a two-stage data creation pipeline involving expert-crafted seed examples and automated mutation for insecure code evaluation, alongside a real-world attack environment with dynamic metrics for cyberattack helpfulness assessment. They compared their benchmark with CYBERSECEVAL using LLM-based judgement on prompt security relevance and faithfulness. c) SECCODEPLT achieved near 100% in both security relevance and prompt faithfulness, while CYBERSECEVAL scored 67.81% and 42% respectively. When testing against SOTA models, GPT-4 performed best in secure coding, with a 52% secure code rate on instruction generation without security policies, though still demonstrating a need for improvement. d) AI practitioners developing or deploying code generation models should leverage SECCODEPLT for more robust security risk assessments and prioritize safety alignment strategies to mitigate the risks of generating insecure code and facilitating cyberattacks. It is unclear whether human verification was used on the automatically generated data used in the large-scale data generation process. Follow-up questions: 1. How does the performance of the rule-based detection compare to the dynamic detection methods in identifying insecure code generated by the models on SECCODEPLT? Does the paper report on the false positive/negative rates? 2. What are the specific details of the attack environment construction, and how scalable is it for evaluating different types of attacks beyond the ones presented in the paper? 3. What specific mitigation strategies, beyond general safety alignment, can be derived from the SECCODEPLT findings for improving the security of code generation models? |
| LVD-2M: A Long-take Video Dataset with Temporally Dense Captions (Read more on arXiv or HuggingFace) | Zhijie Lin, Daquan Zhou, Yuqing Wang, XihuiLiu, YuuTennYi | a) The research aimed to create a high-quality dataset of long videos with dense captions to facilitate the training of long-form video generation models. b) The authors developed a pipeline involving automated video filtering (using scene cut detection, optical flow, and multi-modal large language models) and a hierarchical captioning approach (using image grids and large language models). c) The resulting LVD-2M dataset contains 2 million long-take videos (over 10 seconds each) with temporally dense captions, achieving a long-take video ratio of 86.8% based on human evaluation. d) AI practitioners working on video generation can utilize LVD-2M to fine-tune models for generating longer, more dynamic, and semantically consistent videos, potentially improving metrics like dynamic degree and object class recognition as measured by VBench. The paper notes limitations in dataset size and potential for misuse of generated videos, which practitioners should consider. Follow-up questions: 1. What specific technical details were used in the hierarchical captioning pipeline with LLaVA and Claude3-Haiku, including prompt engineering and parameter settings? How were inconsistencies or hallucinations in the generated captions addressed? 2. While the paper mentions fine-tuning on a 7B LM-based video generation model and a 1.8B parameter diffusion-based I2V model, what are the computational requirements for fine-tuning these models on LVD-2M, and how can these resources be optimized for practical use by AI practitioners? 3. How can the filtering process be further refined to eliminate subtle jump cuts, which were identified as a major remaining challenge, potentially utilizing more advanced scene change detection algorithms or incorporating visual coherence metrics? |
| What Matters in Transformers? Not All Attention is Needed (Read more on arXiv or HuggingFace) | Zheyu Shen, Guoheng Sun, Shwai He, charleslipku | a) This paper investigates the redundancy of different modules (Blocks, MLP layers, Attention layers) within Transformer-based large language models (LLMs). b) The authors use a similarity-based metric to assess module redundancy and propose techniques like “Attention Drop” and “Joint Layer Drop” to prune redundant layers. c) Dropping 50% of the Attention layers in Llama-2-70B resulted in a 48.4% speedup with only a 2.4% performance drop. d) AI practitioners can significantly improve the efficiency of LLMs, particularly regarding inference speed and memory usage (KV-cache), by strategically pruning redundant Attention layers, often without substantial performance degradation. Follow-up Questions: 1. How does the proposed “Joint Layer Drop” method compare with other structured pruning techniques, such as filter pruning or layer-wise magnitude pruning, in terms of performance-efficiency trade-off on different LLM architectures and sizes? 2. Could the “Attention Drop” method be adapted for efficient training of large language models, given that the paper demonstrates consistent redundancy in attention layers throughout the training process? 3. What are the potential implications of this work for hardware design, particularly considering the reduction in KV-cache memory usage achieved by pruning attention layers? |
| Efficiently Democratizing Medical LLMs for 50 Languages via a Mixture of Language Family Experts (Read more on arXiv or HuggingFace) | Yuping Zheng, Nuo Chen, Juhao Liang, Xidong Wang, Guorui Zheng | a) This research aims to develop a multilingual medical Large Language Model (LLM) accessible in numerous languages, addressing data scarcity challenges, particularly for low-resource languages. b) The researchers construct a multilingual medical dataset, analyze LLM information flow using a circuits-based routing analysis within a Mixture of Experts (MoE) framework, and introduce the concept of “language family experts” to scale the model to 50 languages efficiently. c) The 2B parameter Apollo-MoE model achieved 54.8% accuracy on a 12-language medical benchmark and 44.9% accuracy on a 38 low-resource language benchmark. d) AI practitioners can leverage the “language family experts” approach within a Post-MoE architecture to scale multilingual LLMs efficiently without proportionally increasing parameters, facilitating the development of language-inclusive medical AI applications. The most impactful finding is the “Spread Out in the End” phenomenon observed in the information flow circuits, which directly led to the development of Post-MoE architecture applying MoE only in later layers and improving low-resource language performance without additional training. Follow-up questions: 1. How does the performance of Apollo-MoE compare to existing state-of-the-art multilingual LLMs in zero-shot or few-shot settings across different medical tasks beyond the presented benchmarks? 2. What specific linguistic features are used to define the language families, and how was the effectiveness of this grouping validated for the MoE routing? 3. What are the computational resource requirements (e.g., GPU memory, training time) for different Apollo-MoE model sizes, and how do they scale with the number of languages? |
| GS^3: Efficient Relighting with Triple Gaussian Splatting (Read more on arXiv or HuggingFace) | Xiang Feng, Fan Pei, Yixin Zeng, Zoubin Bi, NCJ | a) This research aims to develop a real-time, high-quality novel lighting-and-view synthesis method from multi-view point-lit images. b) The approach utilizes a spatial and angular Gaussian-based representation with a triple splatting process: angular Gaussian splatting for appearance, shadow splatting for self-shadowing, and Gaussian splatting for combining these with residual effects predicted by an MLP. The representation is optimized end-to-end by minimizing the difference between rendered and input photographs. c) The method achieves a rendering speed of over 90 frames per second on a single commodity GPU and a training time of 40-70 minutes. d) AI practitioners can leverage this approach for efficient and high-quality relighting of complex objects and scenes, potentially impacting applications like virtual reality, augmented reality, and visual effects. The paper demonstrates successful reconstruction of a wide range of challenging appearance characteristics like anisotropic reflectance. Follow-up questions: 1. The paper mentions the possibility of using separate sets of angular Gaussians for each spatial Gaussian if sufficient input data is available. Could more details be provided on the trade-off between quality and computational cost when using this approach? How much improvement in quality is observed in practice? 2. What specific hardware configuration constitutes the “single commodity GPU” referenced for the 90fps rendering speed? How does performance scale with the number of spatial and angular Gaussians? 3. What are the limitations of the current shadow splatting method, and what alternative approaches could be explored to improve shadow quality in cases where it is not as crisp as desired? |
| Your Mixture-of-Experts LLM Is Secretly an Embedding Model For Free (Read more on arXiv or HuggingFace) | Ziyue Li, zhoutianyi | a) This research investigates whether the routing weights (RW) in Mixture-of-Experts (MoE) LLMs can function as effective embedding models without further training. b) The study analyzes RW in comparison to hidden state (HS) embeddings, proposing a combined embedding method called MoE Embedding (MOEE) that concatenates or performs a weighted sum of similarities calculated from RW and HS embeddings. c) MOEE (sum), using a weighted sum of similarities from RW and HS, achieved a 22.45% improvement over HS on the DeepSeekMoE-16B model in the Massive Text Embedding Benchmark (MTEB), averaging across all tasks without prompts. d) AI practitioners can leverage the readily available RW in MoE LLMs as effective embedding models without the computational expense of further training or fine-tuning, enhancing performance in various downstream tasks like semantic textual similarity and classification. Follow-up questions: 1. How does the performance of MOEE compare to other state-of-the-art embedding methods that do require training, especially considering the trade-off between computational cost and accuracy? 2. What are the specific implementation details for calculating the weighted sum in MOEE (sum), including the choice of weighting factor (α) and similarity metric, and how can these be optimized for different downstream tasks? 3. Could the observed complementarity between RW and HS embeddings be leveraged for other applications beyond embedding, such as model interpretability or knowledge distillation? |
| SimBa: Simplicity Bias for Scaling Up Parameters in Deep Reinforcement Learning (Read more on arXiv or HuggingFace) | Jun Jet Tai, Hyunseung Kim, Donghu Kim, Hojoon Lee, godnpeter | This research investigates whether incorporating a simplicity bias into network architecture enables effective parameter scaling in deep reinforcement learning (RL). The authors introduce SimBa, a novel RL network architecture combining running statistics normalization, a residual feedforward block, and post-layer normalization. Experiments across various RL algorithms and 51 continuous control tasks show SimBa consistently improves sample efficiency. Specifically, SimBa with Soft Actor-Critic (SAC) matches or surpasses state-of-the-art methods on the DMC, MyoSuite, and HumanoidBench benchmarks, achieving an average return of 706 points on the DMC Hard benchmark. This suggests that, for RL practitioners, simply modifying network architecture to SimBa can improve performance and scalability without computationally expensive add-ons like self-supervised objectives or planning. Follow-up questions: 1. How does SimBa’s performance compare to other architecture scaling methods like BroNet or SpectralNet when using algorithms besides SAC, such as TD7 or DreamerV3, given the paper’s focus on SAC? 2. The paper mentions SimBa’s effectiveness in high-dimensional input spaces. What is the threshold where SimBa’s benefits become particularly significant compared to a standard MLP, and how does this relate to the choice of environment? 3. While the paper analyzes plasticity, it doesn’t explicitly connect it to the generalization capabilities of the learned policies. Are there further investigations planned or insights available on how SimBa’s impact on plasticity affects generalization in dynamic RL environments? |
| Efficient Diffusion Models: A Comprehensive Survey from Principles to Practices (Read more on arXiv or HuggingFace) | Liangliang Zhao, Guoli Jia, Yuzhu Zhang, Zhiyuan Ma, iseesaw | a) This survey paper aims to comprehensively review advancements in efficient diffusion models (DMs) covering architectural designs, training, inference, and deployment to facilitate broader understanding and application. b) The authors organize existing literature into a taxonomy of six categories: principles, architecture, training/fine-tuning, sampling/inference, deployment, and applications, analyzing and comparing the performance of various efficient DM techniques. The survey also compares different approaches such as U-Net, Transformer, and SSM-based backbones. c) The survey presents various techniques to improve DM efficiency, including SnapFusion which reduced mobile text-to-image generation time to under 2 seconds on an iPhone 14 Pro. It lacks specific quantitative benchmarks comparing the different architectural designs and training methods mentioned. d) AI practitioners can use this survey as a roadmap to understand the core principles and practical strategies for developing and deploying efficient DMs across various tasks like image/video generation and editing, 3D synthesis, and medical/bioinformatics applications. The survey’s organization can guide practitioners in selecting appropriate efficient DM techniques based on task requirements. Follow-up questions: 1. Could you provide a more detailed comparative analysis of the different network backbones (U-Net, Transformer, SSM, RWKV, etc.) in terms of computational cost, memory footprint, and performance trade-offs for specific tasks like high-resolution image synthesis and long video generation? 2. The survey mentions the scalability dilemma of DMs compared to LLMs. What are the current most promising research directions to overcome this limitation and enable the emergence of powerful capabilities in DMs similar to those observed in large language models? 3. What are the best practices for deploying and optimizing DM inference in resource-constrained environments, particularly for real-time applications on mobile and web platforms? Can the survey provide more detailed guidance or examples? |
| Towards Synergistic, Generalized, and Efficient Dual-System for Robotic Manipulation (Read more on arXiv or HuggingFace) | Jia Zeng, Jisong Cai, Li Chen, Hongyang Li, qwbu | a) The paper aims to develop a synergistic dual-system framework, RoboDual, to improve robotic manipulation by combining the generalization capabilities of a large-scale pre-trained generalist policy (OpenVLA) with the efficiency and adaptability of a specialist policy. b) RoboDual uses a diffusion transformer-based specialist policy conditioned on multimodal sensory inputs and outputs (latent representations and discretized actions) from the generalist policy. The generalist and specialist are trained separately with potentially different datasets. c) RoboDual achieved a 12% performance improvement on CALVIN and a 20% increase over the most competitive baseline in a real-world setting across a range of manipulation tasks. It also maintained strong performance with only 5% of demonstration data and enabled a 3.8x higher control frequency compared to the generalist alone. d) AI practitioners can leverage RoboDual to efficiently deploy large VLA models for real-world robotic manipulation tasks by combining them with lightweight and adaptable specialist models. The dual-system approach can potentially improve performance, efficiency, and adaptability in data-constrained environments. Follow-up questions: 1. How does the performance of RoboDual vary across different VLA architectures as the generalist policy? Are there specific VLA characteristics that are more conducive to synergistic integration with a specialist? 2. What are the tradeoffs between using a multi-task versus a single-task trained specialist policy in RoboDual, specifically in terms of performance, data efficiency, and computational cost? 3. Could the current fixed inference ratio between generalist and specialist be replaced with an adaptive mechanism that dynamically adjusts the frequency based on task complexity or environment dynamics? |
| Empirical Study of Mutual Reinforcement Effect and Application in Few-shot Text Classification Tasks via Prompt (Read more on arXiv or HuggingFace) | Tatsunori Mori, Chengguang Gan | a) The research investigated the Mutual Reinforcement Effect (MRE), examining whether word-level and text-level information in text classification tasks mutually enhance performance. b) The authors conducted fine-tuning experiments with a novel input-output format on 21 MRE mixed datasets using LLaMA3-8B, and applied word-level information as a knowledgeable verbalizer in few-shot text classification using T5-base. c) In 16 out of 18 sub-datasets, knowledgeable verbalizers constructed with word-level information outperformed the original method in text classification, with improved F1 scores on sentiment analysis datasets. It’s unclear what “original method” refers to specifically. d) AI practitioners can leverage word-level information, such as entities and sentiment polarity, to improve the performance of text classification models, particularly in sentiment analysis and few-shot learning scenarios. Follow-up questions: 1. What is the precise construction method of the “original KV” used as a baseline in the knowledgeable verbalizer experiments? How were the label-related high-frequency words chosen and utilized? 2. Could the authors provide more details on the pre-processing steps and the specific configurations of OpenPrompt utilized for the knowledgeable verbalizer experiments? This would allow replication of these results. 3. What specific metrics beyond F1-score (e.g., precision, recall) were observed in the knowledgeable verbalizer experiment, and how did they vary across different datasets and languages? |
| Towards Natural Image Matting in the Wild via Real-Scenario Prior (Read more on arXiv or HuggingFace) | Qianru Sun, Hao Zhang, Peng-Tao Jiang, Yu Liang, XiaRho | This research aims to improve interactive image matting, specifically using bounding boxes as input, by addressing limitations of existing methods relying on synthetic data and frozen segmentation models. The authors introduce a new dataset, COCO-Matting, derived from COCO and featuring 38,251 human instance-level alpha mattes in complex natural scenes, and propose the Semantic Enhanced Matting (SEMat) framework. SEMat incorporates a feature-aligned transformer and matte-aligned decoder within a modified SAM architecture and uses regularization and trimap losses during training. On the HIM2K dataset, the HQ-SAM-based SEMat achieved a 9.4% relative improvement in Mean Absolute Difference compared to the previous state-of-the-art, SmartMat. This research provides AI practitioners with a new dataset and model architecture for enhanced interactive matting in real-world scenarios. Follow-up questions: 1. Given the computational cost of training SEMat, are there strategies for efficient fine-tuning or adaptation to specific downstream tasks with limited resources? 2. The paper mentions limitations regarding SAM’s performance on rare objects. How does this limitation specifically translate to SEMat’s performance, and are there mitigation strategies, such as data augmentation or few-shot learning techniques, to address this? 3. How does the performance of SEMat compare to other interactive segmentation models besides SAM when adapted for matting using the proposed COCO-Matting dataset and training framework? |
Papers for 2024-10-15
| Title | Authors | Summary |
|---|---|---|
| MMIE: Massive Multimodal Interleaved Comprehension Benchmark for Large Vision-Language Models (Read more on arXiv or HuggingFace) | WendellZwh, wangzhaoyang, StarThomas1002, Lillianwei, richardxp888 | This research aimed to create a benchmark for evaluating interleaved multimodal comprehension and generation in Large Vision-Language Models (LVLMs). The researchers curated a 20K multimodal dataset, MMIE, from existing sources, spanning diverse fields and including multiple-choice and open-ended questions. They fine-tuned InternVL-2-4B with a human-annotated scoring dataset to create an automated evaluation metric. The best-performing integrated LVM (GPT-40 + SDXL) achieved a score of 65.47% on MMIE, indicating significant room for improvement in the field. This suggests to practitioners that current interleaved LVLMs and integrated LVLMs have substantial limitations in tasks requiring both image and text understanding and generation, even with advanced models. Follow-up Questions: 1. How does the performance of the fine-tuned InternVL-2-4B scoring model compare to human evaluation on a larger, unseen test set, and what are the specific strengths and weaknesses of the automated metric observed in such a comparison? 2. What are the specific error modes of the different LVLMs evaluated across the categories and fields in MMIE, and how can these insights be used to inform the development of more robust and capable models? 3. What is the distribution of question types (e.g., multiple-choice vs. open-ended, complexity of reasoning required) within each of the 12 fields of MMIE, and how does this distribution influence the performance variations observed across different LVLMs? |
| LOKI: A Comprehensive Synthetic Data Detection Benchmark using Large Multimodal Models (Read more on arXiv or HuggingFace) | Junan Zhang, Zilong Huang, beccabai, bczhou, Yejy53 | a) The research aims to evaluate the performance of Large Multimodal Models (LMMs) in detecting synthetic data across various modalities (video, image, 3D, text, and audio). b) A novel benchmark called LOKI, comprising 18K questions across 26 subcategories with multi-level annotations, was created and used to evaluate 22 open-source and 6 closed-source LMMs, alongside expert synthetic detection models and human evaluators. c) GPT-4 achieved the highest accuracy among the evaluated models in synthetic data judgment (63.9% overall, excluding audio), and 73.7% accuracy on multiple-choice questions using paired real data. d) LMMs demonstrate moderate performance in synthetic data detection and offer enhanced explainability compared to expert models. The benchmark revealed model biases, a lack of expert domain knowledge in some LMMs, and unbalanced multimodal capabilities, with superior performance in image and text modalities but weaker performance in 3D and audio. This suggests focusing on improved training and architecture design for LMMs, especially in less common modalities, and further developing methods to mitigate model bias. Follow-up questions: 1. How does the performance of LMMs vary when fine-tuning on specific domain datasets within LOKI, particularly for categories like satellite imagery and medical images where a lack of expert knowledge was observed? 2. What specific architectural changes or training strategies could be employed to address the unbalanced multimodal capabilities observed, particularly the relatively poor performance on 3D and audio data? 3. Does the observed model bias (tendency to favor either synthetic or real data) correlate with any specific training data characteristics or model architectures, and what mitigation strategies could be explored to improve unbiased decision-making? |
| Toward General Instruction-Following Alignment for Retrieval-Augmented Generation (Read more on arXiv or HuggingFace) | Zhicheng Dou, Runqi Qiao, Yutao Zhu, Xiaoshuai Song, Guanting Dong | This research aims to improve instruction-following alignment for Retrieval-Augmented Generation (RAG) systems. The authors developed VIF-RAG, a verifiable automated data synthesis pipeline combining augmented instruction rewriting with multiple validation processes, including code-based verification. VIF-RAG significantly improved performance on the FollowRAG benchmark, achieving an average of 52.2% instruction-following accuracy on the Natural Questions dataset compared to 38.8% for the Mistral-7B-SFT baseline. This suggests that VIF-RAG effectively enhances instruction following capabilities in RAG systems while preserving other fundamental LLM abilities. The paper doesn’t specify if this is using Mistral-7B-SFT-VIF-RAG. Follow-up Questions: 1. How does the performance of VIF-RAG scale with larger models and datasets beyond those used in the experiments? 2. What are the computational costs associated with the VIF-RAG pipeline, particularly the code-based verification component? 3. Could the VIF-RAG framework be adapted for other retrieval-augmented tasks beyond question answering, such as summarization or code generation? |
| MEGA-Bench: Scaling Multimodal Evaluation to over 500 Real-World Tasks (Read more on arXiv or HuggingFace) | wenhu, yuexiang96, DongfuJiang, yuanshengni, shermansiu | a) The research aimed to create a comprehensive benchmark, MEGA-BENCH, for evaluating multimodal foundation models across a diverse range of real-world tasks and output formats. b) A task taxonomy was developed and used to guide the collection of 505 tasks with over 8,000 samples, annotated by experts. A suite of 45 customized metrics, including rule-based and LLM-assisted metrics, was used for evaluation. c) GPT-4 achieved the highest overall score across multimodal tasks, outperforming Claude 3.5 by 3.5%. Among open-source models, Qwen2-VL performed best, exceeding the second-best open-source model by approximately 10%. d) MEGA-BENCH provides AI practitioners with a tool for fine-grained analysis of model capabilities across various dimensions (application, input type, output format, skill), enabling targeted model improvement and optimization for specific downstream applications. The superior performance of GPT-4 highlights the continued advancement of closed-source models in multimodal understanding. Follow-up questions: 1. How does MEGA-BENCH’s task diversity and distribution compare to existing multimodal benchmarks, beyond those listed in Table 1, in terms of covering specific skills like numerical reasoning or code generation? 2. What are the details of the LLM-assisted evaluation prompts and how were they validated to ensure consistent and reliable scoring across different annotators and tasks? 3. What are the specific types of “UI-related” and “Document” formats where LLaVA-OneVision-72B struggled, and what architectural or training limitations might explain this weakness? |
| Animate-X: Universal Character Image Animation with Enhanced Motion Representation (Read more on arXiv or HuggingFace) | Dandan Zheng, Shiwei Zhang, Xiang Wang, Shuai Tan, BiaoGong | a) The research aims to develop a character image animation model that generalizes to diverse character types (called “X”), including anthropomorphic figures, overcoming limitations of existing human-centric methods. b) Animate-X utilizes a Latent Diffusion Model (LDM) conditioned on reference image features and a novel “Pose Indicator” that combines implicit motion features from CLIP image embeddings with explicit pose features generated by simulating misalignments during training. c) On the A²Bench, a new dataset of anthropomorphic characters and dance videos introduced by the authors, Animate-X achieved a Fréchet Inception Distance (FID) score of 26.11, significantly outperforming other methods. d) AI practitioners can leverage Animate-X and the proposed Pose Indicator to animate a wider variety of characters, including those with non-human body structures, which is crucial for applications in gaming, entertainment, and virtual reality. The introduction of A²Bench provides a standardized benchmark for evaluating anthropomorphic character animation. Follow-up Questions: 1. How does the computational cost of Animate-X, particularly the Pose Indicator component, compare to other state-of-the-art methods, and how could this impact real-time animation applications? 2. The paper mentions limitations in hand and face modeling. What specific strategies could be explored to address these limitations and improve the realism of generated animations? 3. How does the choice of the pre-trained CLIP model impact performance, and could finetuning CLIP on a dataset of anthropomorphic characters further improve Animate-X’s generalizability? |
| Omni-MATH: A Universal Olympiad Level Mathematic Benchmark For Large Language Models (Read more on arXiv or HuggingFace) | Zhe Yang, Feifan Song, Bofei Gao, mch0115, tobiaslee | a) The research aimed to create a challenging benchmark, Omni-MATH, to evaluate large language models’ (LLMs) mathematical reasoning capabilities at the Olympiad level and analyze model performance across diverse mathematical disciplines and difficulty levels. b) The researchers collected 4,428 competition-level math problems, categorized them into 33+ sub-domains and 10+ difficulty levels, and evaluated 15 LLMs using GPT-40 for verification and an open-source verifier, Omni-Judge. c) The highest-performing model, OpenAI 01-mini with test-time scaling, achieved 60.54% accuracy on Omni-MATH. d) LLMs struggle significantly with Olympiad-level math problems, highlighting a need for improved mathematical reasoning capabilities. The introduction of Omni-MATH and Omni-Judge provides new tools for evaluating and improving these capabilities. The impactful finding is the low accuracy of even the most advanced LLMs on this benchmark, directly demonstrating the limitations of current models in complex mathematical reasoning and highlighting the need for further research in this area. Follow-up questions: 1. What specific techniques were used in the development of the open-source verifier, Omni-Judge, and how can its accuracy be further improved for evaluating increasingly complex mathematical solutions generated by LLMs? 2. Given the identified weaknesses in discrete mathematics, what specific training data augmentation or model architectural changes might be most effective in improving LLM performance in this domain? 3. How does the performance of LLMs on Omni-MATH correlate with their performance on other reasoning benchmarks, and does this correlation suggest specific generalizable strategies for enhancing reasoning capabilities across different domains? |
| LiveXiv – A Multi-Modal Live Benchmark Based on Arxiv Papers Content (Read more on arXiv or HuggingFace) | M. Jehanzeb Mirza, Sivan Doveh, Felipe Maia Polo, Nimrod Shabtay, wlin21at | LiveXiv introduces a live, multi-modal benchmark for evaluating Large Multi-Modal Models (LMMs) using content from arXiv papers. The methodology involves automatically generating Visual Question Answering (VQA) pairs from figures and tables in scientific manuscripts, followed by filtering to ensure multi-modality and reduce hallucinations. Initial benchmark results on 17 LMMs show Claude achieving the highest performance (75.4% VQA, 83.5% TQA). An efficient evaluation method based on Item Response Theory allows performance estimation with reduced computational cost (70% reduction). The benchmark aims to address test data contamination and provide insights into LMM capabilities on less contaminated data. Follow-up questions: 1. How does the automatic VQA generation process handle complex figures with multiple subplots or intricate relationships between visual elements and captions? 2. What specific filtering techniques are used to mitigate hallucinations and ensure questions truly require multi-modal understanding? 3. How does the IRT-based efficient evaluation method compare to other benchmark efficiency approaches in terms of accuracy and computational savings? |
| Cavia: Camera-controllable Multi-view Video Diffusion with View-Integrated Attention (Read more on arXiv or HuggingFace) | Thorsten Gernoth, Liangchen Song, Chen Huang, Yifan Jiang, ir1d | a) The research aimed to develop a framework for generating multi-view consistent videos with precise camera control, addressing limitations in existing video diffusion models regarding 3D consistency and camera controllability. b) Cavia extends a monocular video diffusion model by incorporating view-integrated attention modules (cross-view and cross-frame 3D attention) and employs a joint training strategy utilizing static, monocular dynamic, and multi-view dynamic video datasets. c) Cavia achieved superior performance in geometric consistency and perceptual quality compared to baseline methods, demonstrating a 29.39% precision and 15.22% matching score in multi-view consistency evaluations on the RealEstate10K dataset using SuperGlue for correspondence matching. d) AI practitioners can leverage Cavia to generate multi-view consistent videos with controlled camera trajectories, potentially enabling applications in virtual reality, augmented reality, and 3D scene reconstruction. The improved geometric consistency directly enhances the realism and usability of generated video content for these applications. Follow-up questions: 1. How does the computational cost of Cavia’s view-integrated attention modules compare to standard attention mechanisms, and how does this impact real-time video generation capabilities? 2. Could the training strategy be further improved by incorporating other data sources or augmentation techniques to enhance generalization to more complex camera intrinsics or dynamic scenes? 3. What are the limitations of using SuperGlue for evaluating multi-view consistency, and are there alternative evaluation metrics that could provide more comprehensive insights into the 3D consistency of generated videos? |
| TemporalBench: Benchmarking Fine-grained Temporal Understanding for Multimodal Video Models (Read more on arXiv or HuggingFace) | Jianrui Zhang, Reuben Tan, Mu Cai, fengyao1909, BochengZou | a) The research aimed to create a benchmark for evaluating fine-grained temporal understanding in multimodal video models, addressing the limitations of existing benchmarks that primarily focus on coarse-grained annotations and exhibit language prior bias. b) Researchers curated TemporalBench, a dataset of approximately 10,000 video question-answer pairs derived from 2,000 human-annotated video captions with detailed descriptions of temporal dynamics, and proposed Multiple Binary Accuracy (MBA) as a metric to mitigate bias in multi-choice QA. c) State-of-the-art models like GPT-40 achieved only 38.5% accuracy on TemporalBench using MBA on short videos, significantly lower than human performance (67.9%). d) AI practitioners should focus on improving models’ ability to understand fine-grained temporal relationships in videos, as current models struggle with this aspect, particularly in long videos and tasks requiring precise temporal reasoning. The proposed MBA metric is a more robust evaluation method for temporal understanding. Follow-up Questions: 1. How can the TemporalBench dataset be integrated into existing training pipelines for multimodal video models to specifically improve temporal reasoning capabilities? 2. Beyond video QA and captioning, how can TemporalBench be leveraged for other downstream tasks like action anticipation or event forecasting that heavily rely on temporal understanding? 3. What are the specific design principles behind the negative caption generation using LLMs in TemporalBench, and how can these be adapted to other video understanding datasets? |
| Semantic Image Inversion and Editing using Rectified Stochastic Differential Equations (Read more on arXiv or HuggingFace) | Sanjay Shakkottai, Constantine Caramanis, Nataniel Ruiz, Yujia Chen, Litu Rout | a) This paper addresses the challenge of inverting Rectified Flow (RF) models like Flux for image editing and faithful reconstruction, aiming to overcome limitations of Diffusion Model (DM) inversion in terms of editability and faithfulness. b) The authors propose a controlled Ordinary Differential Equation (ODE) for RF inversion, which interpolates between an unconditional RF vector field and a conditional vector field derived from an optimal control formulation (Linear Quadratic Regulator). They prove the equivalence of this controlled ODE to a rectified Stochastic Differential Equation (SDE). c) On the LSUN-bedroom dataset, their method achieves 4.7% higher faithfulness and 13.79% higher realism compared to the best optimization-free DM inversion method, SDEdit-SD1.5, for stroke-to-image generation. d) AI practitioners can leverage this efficient RF inversion method for zero-shot image editing and faithful reconstruction without additional training, latent optimization, or complex attention mechanisms, enabling faster and more accurate manipulation of real images. The superior performance of RF inversion over DM inversion in this specific task suggests RFs as a potent alternative for image manipulation tasks. Follow-up questions: 1. How does the proposed controlled ODE/SDE approach for RF inversion compare to other RF inversion techniques beyond those based on DMs, in terms of computational efficiency and memory footprint? 2. Could the theoretical framework of rectified SDEs be extended to other generative models beyond rectified flows, and what potential benefits or challenges might arise? 3. What are the limitations of the proposed method in handling highly complex or detailed images, and how could these limitations be addressed in future work? |
| Tree of Problems: Improving structured problem solving with compositionality (Read more on arXiv or HuggingFace) | Rachel Bawden, Benoît Sagot, Armel Zebaze | a) The research aims to improve large language model (LLM) performance on complex, structured problems, particularly those involving multiple reasoning steps, by introducing a novel prompting strategy called Tree of Problems (ToP). b) ToP decomposes a complex problem into a tree of simpler, analogous subproblems, solves the leaf nodes using Chain-of-Thought (CoT) prompting, and recursively merges solutions in a bottom-up approach. c) On the sorting task from Besta et al. (2024), ToP achieves 68% accuracy with GPT-3.5-turbo, outperforming Tree of Thoughts (ToT) and Graph of Thoughts (GoT) by 40% and 19% respectively. d) AI practitioners can leverage ToP as a simpler, more efficient alternative to ToT and GoT for complex tasks decomposable into similar subtasks, potentially improving performance and reducing inference costs. e) The paper did not clearly define how the merge prompt is generated, stating only that it is “specific”. Follow-up questions: 1. What is the specific structure and content of the merge_prompt used in the ToP framework, and how is it adapted for different tasks? 2. How does ToP performance compare to other compositional prompting methods like Least-to-Most on more complex real-world datasets beyond the toy tasks and BIG-Bench Hard benchmarks? 3. What are the computational cost trade-offs (e.g., number of inference calls, latency) of using ToP versus alternative methods like CoT, ToT, and GoT across various tree breadths and depths? |
| TVBench: Redesigning Video-Language Evaluation (Read more on arXiv or HuggingFace) | Cees G. M. Snoek, Manuel Mucientes, yukimasano, mdorkenw, dcores | a) The paper investigates the shortcomings of existing video-language benchmarks, particularly focusing on their lack of emphasis on temporal understanding and the presence of spatial and textual biases, proposing a new benchmark as a solution. b) The authors analyze existing benchmarks like MVBench by evaluating the performance of text-only, image-only, and video models on original and manipulated (shuffled, reversed) videos. They also assess open-ended question-answering benchmarks and their evaluation using LLMs. They then introduce TVBench, a new multiple-choice question-answering video benchmark designed to require temporal reasoning. c) Image-language model GPT-4o achieves 49% accuracy on the fine-grained action task in MVBench, comparable to state-of-the-art video models and surpassing random chance by 20.5% overall, demonstrating the benchmark’s spatial bias. Most recent state-of-the-art video-language models perform near randomly on TVBench, while Tarsier and Gemini 1.5 Pro clearly outperform this baseline, showcasing TVBench’s ability to identify models with strong temporal understanding. d) AI practitioners developing video-language models should consider the limitations of existing benchmarks and incorporate TVBench into their evaluation pipelines to more accurately assess and improve the temporal understanding capabilities of their models. e) The paper doesn’t quantitatively describe the performance drop of Tarsier and Gemini 1.5 Pro on shuffled/reversed TVBench videos, though it is mentioned qualitatively. It also does not provide details on the method used to generate QA pairs for their proposed dataset outside of stating templates were used, rather than LLMs. Follow-up questions: 1. What specific templates were used for generating the question-answer pairs in TVBench, and how was the avoidance of bias ensured during template creation? 2. What is the precise quantitative performance drop observed for Tarsier and Gemini 1.5 Pro on TVBench when videos are shuffled and reversed, respectively? How does this compare to the other video models evaluated? 3. How does the dataset size and diversity of TVBench compare to existing video question answering benchmarks like MVBench, and what are the potential limitations of using a smaller dataset for comprehensive evaluation? |
| Generalizable Humanoid Manipulation with Improved 3D Diffusion Policies (Read more on arXiv or HuggingFace) | Xialin He, Tianyi Chen, Wenhao Wang, Zixuan Chen, Yanjie Ze | a) This research aims to develop a visuomotor policy that enables generalizable humanoid robot manipulation skills in diverse real-world scenarios, trained with data from a single scene. b) The authors introduce the Improved 3D Diffusion Policy (iDP3), which leverages egocentric 3D visual representations, a pyramid convolutional encoder, scaled vision input, and a longer prediction horizon, eliminating the need for camera calibration and point cloud segmentation. Data was collected using a whole-upper-body teleoperation system mapping human movements to a full-sized humanoid robot. c) iDP3 outperformed baseline methods (Diffusion Policy with ResNet18, frozen R3M, and DP3 encoders) in unseen real-world scenarios and showed view invariance; iDP3 achieved a 99/147 success rate on the Pick&Place task across four different setups in diverse real-world scenes after training on only one scene. d) AI practitioners can utilize iDP3 to train generalizable visuomotor policies for humanoid robots without relying on complex camera calibration and point cloud segmentation, potentially simplifying real-world deployment. The paper strongly indicates the superiority of egocentric 3D representations for view invariance in robot manipulation. Follow-Up Questions: 1. The paper mentions noisy 3D point clouds as a limitation. How much does the quality of the 3D data influence the performance of iDP3, and what strategies could further mitigate the impact of noisy sensor data? 2. What is the computational cost of using scaled-up vision input (4096 points) in iDP3, and how does it affect the real-time performance of the policy on the humanoid robot? 3. While the paper shows results on Pick&Place, Pour, and Wipe, how would iDP3 perform on more complex, long-horizon manipulation tasks, and what modifications might be necessary? |
| LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory (Read more on arXiv or HuggingFace) | Kai-Wei Chang, Yuwei Zhang, Wenhao Yu, Hongwei Wang, xiaowu0162 | a) This paper investigates the long-term memory capabilities of chat assistants in sustained interactions. b) The authors introduce LongMemEval, a benchmark with 500 questions across five memory abilities (information extraction, multi-session reasoning, temporal reasoning, knowledge updates, and abstention) embedded within scalable user-assistant chat histories. Commercial chat assistants and long-context LLMs were evaluated. c) Existing long-term memory systems and long-context LLMs exhibit significant performance degradation (30-60% accuracy drop) on LongMemEval compared to simpler memory tasks. d) AI practitioners should consider memory design choices (indexing, retrieval, and reading strategies) to improve long-term memory capabilities in chat assistants. Specific techniques like session decomposition and fact-augmented key expansion are shown to be effective. Follow-up questions: 1. What are the detailed implementations of the proposed memory design optimizations (session decomposition, fact-augmented key expansion, time-aware indexing) and how can they be integrated into existing chat assistant architectures? 2. How does the performance of the proposed memory designs vary across different LLM sizes and architectures, and what are the trade-offs between memory capacity, retrieval speed, and response quality? 3. What are the limitations of the current LongMemEval benchmark, and what future extensions or modifications are needed to further evaluate the robustness and generalization of long-term memory in chat assistants? |
Papers for 2024-10-14
| Title | Authors | Summary |
|---|---|---|
| Baichuan-Omni Technical Report (Read more on arXiv or HuggingFace) | kenshinn, dbv, dongguosheng, TJU-Tianpengli, lin5547 | This research aimed to develop an open-source, omni-modal large language model (MLLM) capable of processing image, video, audio, and text data concurrently. The authors employed a two-stage training approach: multimodal alignment pre-training across different modalities, followed by multitask supervised fine-tuning using a dataset comprising over 600,000 samples across various modalities and over 200 tasks. Baichuan-Omni achieved 72.2% accuracy on the CMMLU benchmark, significantly outperforming the open-source multimodal baseline VITA (46.6%). This provides AI practitioners with a competitive open-source omni-modal LLM for various applications requiring concurrent processing of different modalities, particularly in Chinese language understanding. The paper does not clearly describe the hardware or training time used. Follow-up questions: 1. What were the specific hardware requirements and training duration for Baichuan-Omni? This information is critical for reproducibility and practical application. 2. Could you elaborate on the “packing technique” employed during the multitask fine-tuning stage and its impact on training efficiency and memory usage? A more in-depth explanation of this optimization would be helpful. 3. How does the real-time interaction capability, specifically the streaming input of audio and video, function in practice? More details about the implementation and performance characteristics of this feature are needed. |
| Meissonic: Revitalizing Masked Generative Transformers for Efficient High-Resolution Text-to-Image Synthesis (Read more on arXiv or HuggingFace) | LXT, Enxin, WeiChow, Owen777, BryanW | a) This research aims to improve masked image modeling (MIM) for text-to-image synthesis to achieve efficiency and quality comparable to diffusion models, particularly in high-resolution image generation. b) Meissonic, a 1B parameter model, is introduced, incorporating a multi-modal and single-modal transformer architecture, rotary positional embeddings, adaptive masking rate as a sampling condition, feature compression layers, micro-conditioning (including human preference scores), and a multi-stage training approach using curated datasets. c) Meissonic achieves a Human Preference Score v2.0 of 28.83, exceeding or matching SDXL and other state-of-the-art models in several benchmarks. d) Meissonic offers AI practitioners an efficient, high-resolution (1024x1024), and aesthetically competitive alternative to diffusion-based models for text-to-image synthesis, potentially reducing computational costs for training and inference. Its capability to generate solid-color backgrounds without modification is also highlighted. Follow-up Questions: 1. What are the specific details of the feature compression and decompression layers, and how much do they contribute to the overall efficiency gains during 1024x1024 image generation? 2. The paper mentions Meissonic’s ability to synthesize letters but not words. What are the limitations preventing full word synthesis, and what future research directions could address this? 3. How does Meissonic’s performance compare to diffusion models in image editing tasks beyond the EMU-Edit dataset, specifically in more complex or less common editing operations? |
| From Generalist to Specialist: Adapting Vision Language Models via Task-Specific Visual Instruction Tuning (Read more on arXiv or HuggingFace) | Daniel Shu Wei Ting, Rick Siow Mong Goh, Jun Zhou, Yang Zhou, yangbai123 | This research explores whether Vision Language Models (VLMs) can match or exceed task-specific models (TSMs) in performance. The authors introduce VITask, a framework that uses exemplar prompting (EP) with TSM features, response distribution alignment (RDA), and contrastive response tuning (CRT) to enhance VLM performance on specific tasks. On the MedMNIST dataset, VITask with EP achieved the highest accuracy and F1 scores on 8 of 12 medical image diagnosis tasks. This suggests that integrating task-specific knowledge from TSMs significantly improves VLM performance on specialized tasks, even outperforming larger, more generally trained models. AI practitioners can leverage VITask to efficiently adapt pre-trained VLMs for domain-specific applications without extensive retraining. Follow-up questions: 1. The paper mentions VITask’s robustness to incomplete instructions, but the magnitude of this robustness isn’t quantified beyond Figure 4. How does performance degrade with varying levels of instruction incompleteness across different tasks? 2. The paper focuses on image classification. How adaptable is the VITask framework to other vision-language tasks, such as visual question answering or image captioning, where defining a single TSM might be more complex? 3. What are the computational resource requirements (e.g., GPU memory, training time) for implementing VITask compared to standard instruction tuning or end-to-end fine-tuning of VLMs? |
| EvolveDirector: Approaching Advanced Text-to-Image Generation with Large Vision-Language Models (Read more on arXiv or HuggingFace) | Yujie Wei, AnalMom, xiangwang1223, JacobYuan, ruizhaocv | This research explores training an open-source text-to-image model with public resources to achieve comparable capabilities to existing advanced models whose parameters and training data are proprietary. The EvolveDirector framework trains a base diffusion transformer model using a dynamically updated dataset of image-text pairs generated by advanced models via their APIs. A large vision-language model (VLM) continuously evaluates the base model and refines the dataset through operations like discrimination, expansion, mutation, and deletion based on comparisons between the base model’s output and the advanced model’s output. Results show the trained model, Edgen, outperforms the advanced models in human evaluation across general image generation and specific domains like human and text generation, achieving a 98.08% preference rate overall. This implies that practitioners can potentially replicate and even surpass the capabilities of closed-source advanced models using publicly available resources and strategic data curation guided by VLMs. Follow-up questions: 1. What specific VLMs were used in the comparison study shown in Figure 4, and were they fine-tuned for this image evaluation task or used zero-shot? More details on VLM prompting and evaluation would be helpful. 2. What are the computational costs and API expenses associated with training Edgen compared to training a model on a large static dataset like LAION? A cost breakdown would clarify the practical advantages of EvolveDirector. 3. The paper mentions instability in training with smaller datasets. What specific techniques, besides layer normalization after Q and K projections, were used to stabilize training and prevent mode collapse during multi-scale training? More details would be helpful to replicate the results. |
| StructRAG: Boosting Knowledge Intensive Reasoning of LLMs via Inference-time Hybrid Information Structurization (Read more on arXiv or HuggingFace) | Haiyang Yu, Xuanang Chen, Robin-Lee, xphan, lzq2021 | StructRAG aims to improve Large Language Model (LLM) performance on knowledge-intensive reasoning tasks by using a hybrid information structuring method. The framework dynamically selects the optimal structure type (table, graph, algorithm, catalogue, or chunk) based on the task. It then converts raw documents into this structured format and uses a structured knowledge utilizer to decompose complex questions and extract precise knowledge for inference. Experiments on the Loong benchmark show state-of-the-art performance, with improvements increasing with task complexity. Follow-up questions: 1. What is the computational overhead of dynamically selecting and constructing different structure types during inference? 2. How does StructRAG scale to even larger document sets or more complex structure types? 3. Can the preference learning approach for structure selection be adapted to incorporate user preferences or specific domain knowledge? |
| PositionID: LLMs can Control Lengths, Copy and Paste with Explicit Positional Awareness (Read more on arXiv or HuggingFace) | Yibo Zhang, Feiyu Duan, Zekun Wang, StephenHuang, Wangchunshu | This research addresses the challenge of Large Language Models (LLMs) adhering to length constraints and performing accurate copy-paste operations. The authors propose PositionID Prompting and PositionID Fine-Tuning, where unique identifiers are assigned to textual units (words, sentences, paragraphs) to enhance positional awareness during text generation. For copy-paste, they introduce PositionID CP Prompting, a three-stage tool-use mechanism involving copy and paste tool calls with explicit positional parameters. On the LenCtrl-Bench dataset, PositionID Prompting achieved a Rouge-L score of 23.2, outperforming other length control baselines. The paper’s principal implication for AI practitioners is that explicit positional awareness can significantly improve LLM performance in length-controlled text generation and accurate copy-paste tasks. Follow-up questions: 1. How does the performance of PositionID Fine-Tuning scale with model size and dataset variability? 2. What are the computational overhead and latency implications of incorporating PositionID techniques, particularly for real-time applications? 3. Could PositionID methods be extended beyond length control and copy-paste to other tasks requiring fine-grained textual manipulation, such as text editing or structured data generation? |
| Semantic Score Distillation Sampling for Compositional Text-to-3D Generation (Read more on arXiv or HuggingFace) | Runjia Li, Bohan Zeng, Junlin Han, Zixiang Zhang, Ling Yang | a) The research aims to improve the expressiveness and precision of compositional text-to-3D generation, particularly for complex scenes with multiple objects and intricate interactions. b) The proposed Semantic Score Distillation Sampling (SEMANTICSDS) method integrates program-aided layout planning, novel semantic embeddings, and a region-wise SDS process guided by a rendered semantic map. This leverages pre-trained 2D diffusion priors within a 3D Gaussian Splatting (3DGS) representation. c) SEMANTICSDS achieves state-of-the-art performance on complex text-to-3D generation tasks, demonstrated by a 91.1% score in Prompt Alignment, exceeding other baseline methods. d) AI practitioners can leverage SEMANTICSDS to generate high-quality 3D assets from textual descriptions with improved accuracy and control over the composition and attributes of multiple objects within a scene. Follow-up questions: 1. How does the computational cost of SEMANTICSDS compare to other state-of-the-art text-to-3D methods, particularly regarding the overhead introduced by the semantic embedding and region-wise SDS process? 2. The paper mentions limitations of existing layout-based methods. Could the authors elaborate on specific failure cases of SEMANTICSDS and discuss potential future improvements to address those limitations? 3. Are there specific types of text prompts or scene complexities where the benefits of SEMANTICSDS are most pronounced, and are there any scenarios where simpler methods might suffice? |
| SuperCorrect: Supervising and Correcting Language Models with Error-Driven Insights (Read more on arXiv or HuggingFace) | Joseph E. Gonzalez, Minkai Xu, Tianjun Zhang, Zhaochen Yu, Ling Yang | a) The research aims to improve the mathematical reasoning and self-correction abilities of smaller language models (LLMs). b) A two-stage framework, SuperCorrect, is proposed: 1) Hierarchical thought template-based supervised fine-tuning (SFT) using insights from a larger teacher LLM, and 2) Cross-model collaborative Direct Preference Optimization (DPO) guided by the teacher LLM’s correction traces. c) SuperCorrect-Qwen-7B achieved 70.2% accuracy on the MATH dataset, outperforming DeepSeekMath-7B by 7.8% and Qwen2.5-Math-7B by 15.1%. d) AI practitioners can leverage SuperCorrect to enhance the performance of smaller LLMs on complex reasoning tasks, reducing the reliance on larger, computationally expensive models. The paper’s strongest contribution is the cross-model collaborative DPO, offering a novel approach to improve self-correction in LLMs, a key factor for reliable AI system development. Follow-up questions: 1. How does the performance of SuperCorrect scale with different sizes of teacher and student LLMs? Specifically, what are the trade-offs between teacher LLM size and the improvement observed in the student LLM? 2. Could the hierarchical thought template generation process be automated or improved, reducing reliance on manually generated solutions or teacher LLM output? 3. How does SuperCorrect perform on other reasoning-intensive tasks beyond mathematics, such as logical deduction or commonsense reasoning? |
| Mechanistic Permutability: Match Features Across Layers (Read more on arXiv or HuggingFace) | Ian Maksimov, kefirski, elephantmipt | a) The paper investigates how interpretable features, extracted using Sparse Autoencoders (SAEs), evolve across the layers of a deep neural network (specifically, the Gemma 2 language model). b) The researchers introduce SAE Match, a data-free method that aligns SAE features from different layers by minimizing the mean squared error (MSE) between the “folded” parameters of the SAEs (incorporating activation thresholds). They also use external LLM evaluations of feature descriptions and metrics like change in cross-entropy loss and explained variance when approximating hidden states with matched features. c) The study found that matching SAE features using folded parameters improves alignment quality compared to not using folded parameters, as evidenced by lower MSE values and more “SAME” labels from LLM evaluations. Specifically, unfolded matching resulted in consistently higher MSE values compared to folded matching across all tested SAE layers. d) For AI practitioners, this research offers a method to track feature evolution and persistence through network layers, potentially improving interpretability and enabling techniques like layer pruning based on feature similarity. The impact of SAE sparsity on feature matching is also explored, potentially guiding practitioners in choosing appropriate SAE configurations for analysis. Follow-up questions: 1. The paper mentions a performance drop in feature matching quality at the 10th layer. What are the potential causes of this drop, and how can it be addressed? Does this layer represent a shift in the type of features being learned by the model? 2. While the paper focuses on the Gemma 2 model, how generalizable is the SAE Match method to other architectures and model types? What modifications or adaptations might be necessary for effective application to different models? 3. Could the method be extended to support other interpretability techniques beyond Sparse Autoencoders? For example, could it be adapted to align features extracted by probing methods or other types of autoencoders? |
| Multi-Agent Collaborative Data Selection for Efficient LLM Pretraining (Read more on arXiv or HuggingFace) | Xinlin Zhuang, Jiahui Peng, Zhen Hao Wong, Ling Yang, beccabai | a) The research aimed to improve the data efficiency of large language model (LLM) pretraining by resolving conflicts between different data selection methods. b) A multi-agent collaborative framework was proposed, where each data selection method (quality, domain, topic) acted as an agent, with an agent console dynamically integrating their scores and adjusting agent weights based on performance on reference tasks. c) The multi-agent approach achieved an average performance gain of up to 10.5% across multiple language model benchmarks compared to baseline methods, including a 7.1% improvement over the influence function-based method MATES. d) LLM practitioners can potentially improve training efficiency and downstream task performance by integrating multiple data selection strategies within a dynamic, collaborative framework rather than relying on individual methods in isolation. Follow-up questions: 1. What is the computational overhead of the multi-agent framework during pretraining, and how does it compare to the overhead of methods like MATES, which require recalculating influence scores? 2. Could the multi-agent framework be adapted to incorporate other data selection heuristics beyond quality, domain, and topic, and what would be the key considerations for such an adaptation? 3. How sensitive are the overall performance gains to the choice of reference tasks and the optimization strategy for updating the agent and collaboration weights during training? |
| KV Prediction for Improved Time to First Token (Read more on arXiv or HuggingFace) | moinnabi, mrastegari, yjin25, qicao-apple, mchorton | a) The paper investigates reducing the Time To First Token (TTFT) of transformer-based language models, particularly on resource-constrained edge devices. b) It introduces “KV Prediction,” using a smaller auxiliary transformer model to predict the Key-Value (KV) cache of a larger base model via learned linear projections. After prediction, inference continues solely with the base model. c) On TriviaQA, KV Prediction achieves 15%-50% better accuracy retention compared to baselines at equal TTFT FLOP counts. d) AI practitioners can use KV Prediction to significantly improve the TTFT of large language models on edge devices, enabling a better user experience in latency-sensitive applications like chatbots without sacrificing much accuracy. The significant improvement in accuracy retention compared to token pruning methods provides a more robust approach to on-device LLM efficiency. Follow-up questions: 1. How does the performance of KV Prediction scale with the size of the base and auxiliary models, and what is the optimal size ratio for different resource constraints? 2. What are the memory implications of storing and utilizing the predicted KV cache, especially for longer sequences, and how can these be mitigated? 3. Could the predictor network be improved beyond linear projections, for example, by using a small transformer, and would this lead to substantial accuracy gains at a manageable increase in computational overhead? |
| Mentor-KD: Making Small Language Models Better Multi-step Reasoners (Read more on arXiv or HuggingFace) | SKyii, monocrat23, nokomon | a) The paper investigates how to improve the multi-step reasoning capabilities of smaller language models (LMs) through knowledge distillation from larger language models (LLMs). b) The proposed Mentor-KD framework uses an intermediate-sized, task-specific “mentor” LM to augment the distillation set from the LLM teacher by generating additional chain-of-thought rationales and soft labels for the student LM. c) On four reasoning datasets (GSM8K, ASDiv, SVAMP, CommonsenseQA), Mentor-KD with a FlanT5-XL student model achieved an average accuracy approximately 2.0% higher than the previous state-of-the-art, MCC-KD. d) AI practitioners can potentially use Mentor-KD to develop more efficient and performant smaller LMs for complex reasoning tasks, reducing the reliance on expensive and resource-intensive LLM inference. The demonstrated improvement in smaller LM performance through data augmentation with a mentor model provides a promising pathway for deploying sophisticated reasoning abilities on resource-constrained devices. Follow-up questions: 1. How does the computational cost of training the mentor model compare to the cost savings from reduced LLM API calls, and what is the break-even point in terms of dataset size or inference volume? 2. How does the performance of Mentor-KD vary across different model architectures beyond encoder-decoder models, particularly decoder-only models like GPT series? 3. How does the choice of mentor model size affect student performance, and are there guidelines for selecting an optimal mentor size based on the student model and task? |
| DA-Code: Agent Data Science Code Generation Benchmark for Large Language Models (Read more on arXiv or HuggingFace) | Yiming Huang, lx865712528, bjEdward, FangyuLei, Jianwen2003 | The paper introduces DA-Code, a benchmark designed to evaluate Large Language Model (LLM) performance on agent-based data science coding tasks. The benchmark features complex tasks requiring grounding and planning, diverse real-world data sources, and solutions utilizing Python, SQL, and Bash. When evaluated using the DA-Agent framework, the best performing LLM, GPT-4, achieved only 30.5% accuracy. This low accuracy underscores the significant challenge LLMs face in autonomously completing real-world data science tasks, highlighting the need for further improvement in LLM agent capabilities. The EEEA (Exploration-Execution-Evaluation-Adjustment) pattern observed in agent trajectories offers valuable insights into LLM problem-solving approaches. Follow-up Questions: 1. How does the performance of open-source LLMs on specific DA-Code task categories (e.g., data wrangling, machine learning) compare to closed-source models, and what factors might contribute to observed performance differences? 2. Given the limited effectiveness of current LLMs in complex data scenarios like those presented in DA-Code, what specific research directions (e.g., enhanced training data, improved agent frameworks) are most promising for improving LLM performance on these types of tasks? 3. Can the DA-Code benchmark be adapted or extended to evaluate other aspects of LLM agents beyond code generation, such as explanation generation or interactive data exploration capabilities? |
Papers for 2024-10-11
| Title | Authors | Summary | |
|---|---|---|---|
| MathCoder2: Better Math Reasoning from Continued Pretraining on Model-translated Mathematical Code (Read more on arXiv or HuggingFace) | juntingpan, shiwk20, Houxing, scikkk, AJZhou | a) This research aimed to improve large language models’ (LLMs) mathematical reasoning abilities through continued pretraining on a dataset enriched with code and associated reasoning steps. b) The researchers curated a 19.2B-token dataset, MathCode-Pile, consisting of math-related web data, code using mathematical packages, textbooks, synthetic data, and importantly, model-generated code with corresponding natural language reasoning steps extracted from mathematical texts. LLMs were then pretrained on MathCode-Pile. c) MathCoder2-Llama-3-8B, trained with MathCode-Pile, achieved 4-shot accuracies of 38.4% on MATH and 69.9% on GSM8K, demonstrating improvements of 17.0% and 15.1% respectively over the baseline Llama-3 model trained without MathCode-Pile’s model-translated code and reasoning steps data. d) AI practitioners can leverage MathCode-Pile and the method for generating code paired with reasoning steps to enhance the mathematical capabilities of LLMs, especially for tasks requiring tool-integrated reasoning. The open-sourcing of the code and data facilitates reproducibility and further research. Follow-up questions: 1. How does the performance of MathCoder2 compare to other state-of-the-art models on more complex mathematical reasoning tasks beyond the five benchmark datasets used in the study? 2. What are the computational resource requirements for pretraining with MathCode-Pile, and how scalable is the proposed method for larger model sizes or datasets? 3. Could the performance improvement seen with the paired code and reasoning steps be further enhanced by different data generation strategies, such as incorporating diverse reasoning paths or error analysis? | |
| PrefixQuant: Static Quantization Beats Dynamic through Prefixed Outliers in LLMs (Read more on arXiv or HuggingFace) | Yi Bin, Jiahao Wang, Yi Liu, wqshao126, ChenMnZ | a) The research aims to improve the efficiency of Large Language Model (LLM) quantization, specifically addressing the challenge of token-wise outliers that hinder per-tensor static quantization. b) PrefixQuant prefixes high-frequency outlier tokens and the [BOS] token in the KV cache, thereby preventing their generation during inference and enabling effective per-tensor static quantization. Block-wise fine-tuning is also used to further refine the quantization parameters. c) On a W4A4KV4 (4-bit weight, activation, and KV cache) quantized Llama-3-8B model, PrefixQuant achieved a 7.43 WikiText2 perplexity and 71.08% average accuracy on five common-sense reasoning tasks, outperforming previous dynamic quantization methods. d) AI practitioners can utilize PrefixQuant to achieve faster and more memory-efficient LLM deployment through its per-tensor static quantization approach, exceeding the performance of existing dynamic quantization techniques without retraining. The paper specifically highlights increased inference speeds compared to previous approaches. Follow-up questions: 1. How does the performance of PrefixQuant scale with different model sizes and architectures beyond those tested in the paper? 2. What are the specific memory savings achieved by PrefixQuant compared to dynamic quantization methods and FP16 models across different hardware platforms? 3. The paper mentions isolating outlier tokens improving training stability. Are there quantitative measures of this increased stability (e.g., variance of loss during training), and how significant is this improvement compared to existing quantization-aware training methods? | |
| MLLM as Retriever: Interactively Learning Multimodal Retrieval for Embodied Agents (Read more on arXiv or HuggingFace) | Zongqing Lu, Xinru Xu, tellarin, yuejunpengpku | a) This research aims to improve embodied agent performance by developing a more effective multimodal trajectory retriever that prioritizes task relevance over surface-level similarity. b) The proposed method, MLLM As ReTriever (MART), uses interactive learning to fine-tune an MLLM retriever with preference pairs based on trajectory effectiveness, incorporating a Trajectory Abstraction mechanism to condense trajectory information. c) In experiments across AI2-THOR and LEGENT environments, MART significantly outperformed baseline methods, achieving a 10% higher success rate on unseen tasks in AI2-THOR. d) AI practitioners can leverage MART to improve embodied agent performance in unseen environments and complex, long-horizon tasks by fine-tuning an MLLM as a task-aware retriever rather than relying solely on similarity-based retrieval. Follow-up questions: 1. How does the computational cost of fine-tuning the MLLM retriever with preference pairs scale with the size of the expert trajectory memory? 2. Could the Trajectory Abstraction mechanism be further improved by incorporating reinforcement learning to dynamically select the most relevant milestones based on the current task and environment? 3. How robust is MART to noisy or incomplete trajectory data, and what strategies could be employed to mitigate the impact of such data on retriever performance? | |
| DICE: Discrete Inversion Enabling Controllable Editing for Multinomial Diffusion and Masked Generative Models (Read more on arXiv or HuggingFace) | akashsri, FelixXu, quandao10, ligongh, AristHe | a) This paper addresses the challenge of controlled content editing in discrete diffusion models, including multinomial diffusion and masked generative models. b) The authors introduce DICE (Discrete Inversion for Controllable Editing), a novel inversion algorithm that records noise sequences and masking patterns during the reverse diffusion process, enabling accurate reconstruction and flexible editing without predefined masks or attention manipulation. c) Experiments on image and text modalities show DICE achieves superior performance; on the PIE-Bench dataset, DICE+Paella achieved a structure distance of 11.34×10⁻³, outperforming masked inpainting and continuous diffusion models. d) DICE provides AI practitioners with a new technique for fine-grained manipulation of discrete data, such as text and image tokens, by enabling precise inversion and controlled editing with discrete diffusion models. The improved structural preservation and editing capabilities demonstrated by DICE on images and text represent a significant advancement for applications like text-guided image editing and sentiment modification in text. Follow-up questions: 1. How does the computational cost of DICE compare to existing methods like DDIM inversion or masked inpainting, particularly for high-resolution images or long text sequences? 2. The paper mentions hyperparameters τ, λ₁, and λ₂. What is the impact of these hyperparameters on editing performance, and are there recommended strategies or guidelines for tuning them for different tasks and datasets? 3. Could DICE be extended or adapted to work with other types of discrete data beyond text and images, such as audio or time series data represented as discrete tokens? | |
| Benchmarking Agentic Workflow Generation (Read more on arXiv or HuggingFace) | Ningyu, xiaoyuehanbin, consultantQ, Runnaning, GoooDte | a) This research introduces WORFBENCH, a benchmark for evaluating Large Language Model (LLM) agents’ ability to generate workflows, addressing limitations in existing frameworks. b) WORFBENCH includes diverse scenarios, complex graph workflow structures, and a rigorous evaluation protocol called WORFEVAL based on subsequence and subgraph matching algorithms. c) Evaluation across various LLMs revealed a significant performance gap between linear and graph planning, with GPT-4 achieving only 52.47% on graph workflow generation. d) For AI practitioners, this highlights the need to improve LLM agents’ graph planning capabilities, potentially through integrating world knowledge or world models, as this significantly impacts their effectiveness in complex, real-world scenarios. The gap between sequence and graph planning capabilities emphasizes that current LLMs struggle with generating more complex, parallel workflows, even with strong language understanding. Follow-up Questions: 1. Could providing LLMs with explicit training data on graph structures, beyond simply relying on implicit learning from sequential data, improve graph workflow generation performance? 2. What specific strategies for integrating world knowledge or world models would be most effective in addressing the observed limitations in graph planning? 3. How can the insights from WORFBENCH be applied to improve the design and development of workflow-based LLM applications in specific domains like robotics or software automation? | |
| Agent S: An Open Agentic Framework that Uses Computers Like a Human (Read more on arXiv or HuggingFace) | Shuyu Gan, Saaket Agashe, xw-eric, jc-y42, Jiuzhouh | a) The research aimed to develop an agentic framework enabling autonomous interaction with computers through a Graphical User Interface (GUI) to automate complex tasks. b) Agent S integrates experience-augmented hierarchical planning, continual memory updates, and an Agent-Computer Interface (ACI) tailored for Multimodal Large Language Models (MLLMs). c) On the OSWorld benchmark, Agent S achieved a 20.58% overall success rate, a substantial improvement over the baseline’s 11.21% and a new state-of-the-art result. d) AI practitioners can leverage Agent S to build GUI agents capable of complex task automation, particularly in “Daily” and “Professional” computer task categories, where significant performance gains were observed. The high success rate improvement directly impacts the feasibility of deploying autonomous GUI agents for practical applications. Follow-up questions: 1. What are the specific primitive actions included in the constrained action space of the ACI, and how are they chosen to balance expressiveness and safety for MLLM-based GUI agents? 2. Given the observed error analysis focusing on planning and grounding, what future work is planned to address these bottlenecks and further improve Agent S’s reliability, specifically in terms of reducing repetitive actions caused by grounding errors? 3. How does the continual learning process adapt to evolving software interfaces or application updates, and what mechanisms ensure the ongoing relevance and effectiveness of the learned experiences stored in the narrative and episodic memories? | |
| Rectified Diffusion: Straightness Is Not Your Need in Rectified Flow (Read more on arXiv or HuggingFace) | Ling Yang, hsli-cuhk, Edify-Kd2024, DrinkingCoder, wangfuyun | a) The paper investigates the core factors contributing to the effectiveness of rectified flow for accelerating diffusion model generation and explores its generalization to broader diffusion model variants. b) The authors propose Rectified Diffusion, which retrains a pre-trained diffusion model using pre-computed noise-sample pairs, eliminating the need for flow-matching and v-prediction used in rectified flow. They also introduce Rectified Diffusion (Phased), which enforces local first-order linearity of the ODE path within segmented time steps, and utilize consistency distillation for low-step generation enhancement. c) Rectified Diffusion achieves a 1-step FID score of 27.26 on the COCO-2017 validation set compared to 47.91 for Rectified Flow, demonstrating faster training and superior performance. d) AI practitioners can leverage Rectified Diffusion to simplify the training process and improve the performance of accelerated diffusion models without model conversion to flow-matching forms, potentially enabling faster and higher quality generation for various applications. The most impactful finding is that paired noise-sample retraining is the crucial element, not ODE path straightness, expanding the applicability of rectified diffusion to wider diffusion model types. Follow-up questions: 1. How does the performance of Rectified Diffusion scale with different model architectures and datasets beyond Stable Diffusion and COCO? 2. What are the practical considerations and limitations when implementing the phased approach for real-world applications with varying computational constraints? 3. How does the choice of consistency distillation technique impact the final performance, and are there alternative distillation methods that could further improve low-step generation quality? | |
| Intriguing Properties of Large Language and Vision Models (Read more on arXiv or HuggingFace) | Ho-Jin Choi, yechan99, mkmiracle, kobiso, passing2961 | This research investigates the perceptual and cognitive properties of Large Language and Vision Models (LLVMs), particularly how they process and interpret visual information. The study evaluates LLaVA-series models on 10 benchmarks, including MMVP, MathVista, and AI2D, using methods such as permutation of visual patch tokens, occlusion of image regions, and use of synthetic images. Results show that LLVMs exhibit permutation invariance with minimal performance drop (e.g., <1% average drop for LLaVA 1.5 across 10 benchmarks after shuffling visual patch tokens) and robustness to occlusion, even solving some math problems with limited visual input. This implies that LLVMs process images globally rather than relying heavily on localized pixel information. For AI practitioners, this suggests that optimization efforts should focus on enhancing global image understanding and cross-modal alignment rather than solely on pixel-level processing. Here are some follow-up questions an AI practitioner might ask: 1. Given the observed permutation invariance, could architectural modifications that explicitly encourage local feature attention improve performance on tasks requiring detailed visual understanding, such as MMVP or fine-grained image classification? 2. How can the observed trade-off between complex cognitive reasoning abilities and basic visual recognition capabilities (catastrophic forgetting) be mitigated during the fine-tuning process of LLVMs? 3. How can we design more complex and interactive evaluation benchmarks to better assess the performance and generalization capabilities of LLVMs in real-world scenarios that necessitate multi-turn interactions and personalized responses? | |
| Towards Self-Improvement of LLMs via MCTS: Leveraging Stepwise Knowledge with Curriculum Preference Learning (Read more on arXiv or HuggingFace) | Ye Tian, haitaominlp, Pluie1503, freesunshine0316, russwang | a) This research aims to improve the reasoning capabilities of Large Language Models (LLMs) by more effectively distilling behaviors learned through Monte Carlo Tree Search (MCTS). b) The proposed ALPHALLM-CPL framework uses stepwise trajectory pair extraction from MCTS and curriculum preference learning (CPL) to train LLMs. CPL dynamically adjusts the training sequence of trajectory pairs, prioritizing those most critical for learning. c) On the GSM8K benchmark, ALPHALLM-CPL improved the performance of LLaMA2-7B from 14.6 to 36.5, a 150% increase. d) AI practitioners can leverage ALPHALLM-CPL to significantly enhance the mathematical reasoning abilities of LLMs using MCTS without needing extensive external data or stronger models, offering a path toward more autonomous LLM improvement. Follow-up questions: 1. What is the computational cost of generating the stepwise trajectory pairs and implementing the curriculum preference learning compared to existing MCTS distillation methods? 2. How does the performance of ALPHALLM-CPL vary with different values of the margin ‘τ’ and balance rate ‘α’ used in trajectory pair extraction and curriculum preference learning, respectively? What guidelines are there for tuning these hyperparameters? | |
| Preserving Multi-Modal Capabilities of Pre-trained VLMs for Improving Vision-Linguistic Compositionality (Read more on arXiv or HuggingFace) | Junmo Kim, In So Kweon, Dong-Jin Kim, Jae Won Cho, ytaek-oh | This research aimed to improve the compositional reasoning of Vision-Language Models (VLMs) while maintaining their performance on standard multi-modal tasks. The researchers developed Fine-grained Selective Calibrated CLIP (FSC-CLIP), which incorporates local hard negative loss based on patch-token alignments and selective calibrated regularization to mitigate the negative impact of hard negative training. FSC-CLIP, when fine-tuned on a 100K subset of LAION-COCO, achieved a compositionality score of 53.5 and a zero-shot classification score of 55.9, nearly matching the pre-trained CLIP’s zero-shot performance. This suggests that FSC-CLIP allows for significant improvements in compositional reasoning without sacrificing performance on other crucial VLM tasks, offering a more balanced and robust model for AI practitioners. It is unclear if this method extends beyond fine-tuning to pre-training, or whether it is directly applicable to other similar architectures or models besides CLIP. Follow-up questions: 1. How does the computational cost of FSC-CLIP during training and inference compare to existing fine-tuning methods like DAC-LLM or NegCLIP, especially with larger datasets and models? 2. Could the authors elaborate on the limitations of using short captions, and provide concrete examples of the complex contextual nuances and longer-range dependencies in detailed descriptions that current VLMs struggle with? What future research directions are suggested for addressing these challenges? | |
| SFTMix: Elevating Language Model Instruction Tuning with Mixup Recipe (Read more on arXiv or HuggingFace) | Sanqiang Zhao, Marzyeh Ghassemi, wzhouad, szhang42, YuxinXiao | This paper investigates improving large language model (LLM) instruction-tuning performance without relying on curated datasets. The authors propose SFTMix, which leverages training dynamics to split a dataset into confident and unconfident subsets and applies a Mixup-based regularization during instruction tuning. Results on MT-Bench and AlpacaEval-2 show that SFTMix outperforms the next-token prediction (NTP) baseline, with Llama-3.1-8B achieving a 4.5825 overall score on MT-Bench with SFTMix versus 4.3625 with NTP. This implies that AI practitioners can potentially improve LLM instruction-tuning performance and generalization on downstream tasks by incorporating the SFTMix recipe without requiring costly dataset curation. The paper does not specify the precise algorithm for assigning data points to confident/unconfident splits based on the perplexity calculations. Follow-up questions: 1. What is the specific algorithm used to assign data points to the “confident” and “unconfident” subsets based on the calculated Conf(Vᵢ | Xᵢ) values? Is it a simple threshold, or a more complex clustering approach? 2. How does the computational cost of calculating the training dynamics and performing the Mixup regularization compare to the computational savings from using less curated data? Is there a net benefit in terms of resource usage? 3. How does SFTMix perform with very large LLMs and datasets where calculating perplexity over the entire training set for multiple checkpoints becomes significantly more expensive? Are there strategies for efficient approximation or scaling in such scenarios? |
| Progressive Autoregressive Video Diffusion Models (Read more on arXiv or HuggingFace) | Hao Tan, Zhan Xu, smebliu, YicongHong, desaix | a) The research aims to extend the temporal capacity of video diffusion models, which are currently limited to short video generation due to computational constraints during training. b) The authors propose progressive autoregressive video diffusion models, assigning progressively increasing noise levels to latent frames within the attention window during denoising, enabling autoregressive generation of extended video sequences. This method involves finetuning existing video diffusion models on a modified noise schedule and applying a specific autoregressive sampling procedure. c) On a long video generation task (60 seconds, 1440 frames), their best performing model (PA-M) achieved an average dynamic degree score of 0.8, substantially outperforming other baselines while maintaining competitive scores on other metrics like aesthetic and imaging quality. It is unclear how the number of training steps differed between PA-M and other models. d) AI practitioners can leverage this progressive denoising technique to generate significantly longer, high-quality videos using existing video diffusion model architectures, potentially reducing the need for computationally expensive training of entirely new long-video models. The paper implies this progressive denoising method can be applied to different video diffusion architectures, but only demonstrates it on transformer-based architectures. Follow-up questions: 1. Could the performance gains of progressive autoregressive denoising be further enhanced by exploring alternative noise scheduling strategies beyond the linear schedule used in this research? 2. How does the computational cost of finetuning a pre-trained video diffusion model with progressive noise levels compare to the computational cost of training a new model specifically designed for long-video generation? 3. The paper mentions chunk-by-chunk processing as being crucial. How does chunk size impact long-video generation quality and computational cost, and is there an optimal chunk size for different model architectures? | |
| GLOV: Guided Large Language Models as Implicit Optimizers for Vision Language Models (Read more on arXiv or HuggingFace) | aquila147, mdorkenw, paulgavrikov, sivand, kevinmzy | This research explores using Large Language Models (LLMs) to optimize prompts for Vision-Language Models (VLMs), aiming to improve VLM performance on downstream vision tasks like image classification. The key methodology, GLOV, involves a meta-prompting LLM with task descriptions and ranked in-context examples, coupled with embedding space guidance to steer prompt generation. Results show GLOV improves zero-shot CLIP accuracy on ImageNet by up to 15.0% and LLaVa accuracy by up to 57.5%. This implies AI practitioners can leverage LLMs to automatically discover highly effective prompts for VLMs, significantly boosting performance without gradient-based training or fine-tuning. Follow-up questions: 1. What are the computational resource requirements (e.g., GPU memory, runtime) for running GLOV, especially with larger datasets and VLMs? 2. How sensitive is GLOV’s performance to the choice of LLM and its hyperparameters (e.g., number of optimization steps, guidance scaling factor)? 3. How does the performance of GLOV-generated prompts compare to fine-tuning VLMs on downstream tasks in few-shot settings? | |
| Optima: Optimizing Effectiveness and Efficiency for LLM-Based Multi-Agent System (Read more on arXiv or HuggingFace) | Cheng Yang, Chen Qian, Jiarui Yuan, zibuyu9, weizechen | a) The research aimed to develop a training framework for Large Language Model (LLM)-based Multi-Agent Systems (MAS) that enhances communication efficiency and task effectiveness. b) OPTIMA, the proposed framework, uses an iterative generate, rank, select, and train paradigm with a reward function balancing task performance, token efficiency, and communication readability, incorporating techniques like Supervised Fine-Tuning (SFT), Direct Preference Optimization (DPO), and Monte Carlo Tree Search (MCTS). c) OPTIMA achieved up to a 2.8x performance gain with less than 10% of the tokens compared to Multi-Agent Debate (MAD) on tasks requiring heavy information exchange. d) OPTIMA enables more efficient use of inference compute, potentially leading to better inference-time scaling laws, which AI practitioners can leverage for performance gains without additional model training. OPTIMA’s demonstrated ability to significantly reduce token usage while improving performance is directly applicable to improving the computational efficiency of deployed LLM-based MAS. Follow-up questions: 1. How does OPTIMA’s MCTS-inspired DPO data generation compare to alternative data generation methods for multi-agent DPO in terms of computational cost and resulting data quality? 2. Could the observed improvements in inference scaling laws be further amplified by combining OPTIMA with more advanced answer aggregation techniques like weighted voting? 3. What are the limitations of OPTIMA’s current implementation, and what future research directions could address these limitations (e.g., scaling to larger models, more complex multi-agent scenarios)? | |
| Emergent properties with repeated examples (Read more on arXiv or HuggingFace) | François Charton, Knykny | a) The research investigates the impact of training example repetition on transformer performance in mathematical tasks, challenging the prevailing assumption that maximizing distinct training examples is always optimal. b) The study uses algorithmically generated datasets for greatest common divisor (GCD), modular multiplication, and matrix eigenvalue calculation, controlling repetition frequency and employing two-set training (repeating a random subset more frequently). c) For GCD, with a training budget of 600 million examples and a data budget of 100 million, two-set training with a repeated subset of 50,000 examples (repeated 3000 times) achieved 69 correctly predicted GCDs, outperforming single-set training which achieved 27. d) AI practitioners should consider training set size (distinct examples) as a hyperparameter and explore the potential of two-set training, where repeating a small random subset more frequently can improve performance and learning speed. The paper lacks information on the computational costs of two-set training compared to standard practices. Follow-up questions: 1. How does the computational cost of two-set training, including storage and processing overhead from increased repetition, compare to standard single-epoch training with a larger dataset? 2. How does two-set training perform in comparison to curriculum learning approaches using specifically curated example subsets for repetition? 3. What is the relationship between the optimal repetition frequency and dataset characteristics like size and task complexity in a two-set training paradigm? | |
| Scaling Up Your Kernels: Large Kernel Design in ConvNets towards Universal Representations (Read more on arXiv or HuggingFace) | xyyue, DingXiaoH, Yiyuan | This paper investigates whether large-kernel ConvNets can offer universal modeling capabilities similar to Vision Transformers (ViTs) with reduced complexity. The authors propose UniRepLKNet, a novel ConvNet architecture based on a set of design principles for large kernels, emphasizing depth-wise convolutions, identity shortcuts, and dilated small kernel re-parameterization. UniRepLKNet achieves 88.0% ImageNet top-1 accuracy and demonstrates strong performance across modalities like audio (98.5% accuracy on Speech Commands V2), video, and time-series forecasting. This suggests that large-kernel ConvNets provide a viable, efficient alternative to transformers for diverse AI tasks. Follow-up questions: 1. The paper mentions modality-specific preprocessing to transform data into 3D embedding maps. Could the authors elaborate on the specific preprocessing steps used for each modality beyond the brief descriptions provided? This information would be crucial for replicating the results and applying the architecture to new modalities. 2. What are the memory and computational requirements of UniRepLKNet compared to ViTs and other state-of-the-art models on downstream tasks beyond ImageNet classification? More detailed comparisons would help assess the practical advantages of UniRepLKNet for resource-constrained applications. 3. How does the performance of UniRepLKNet change with varying kernel sizes in different stages, and what guidelines can be derived for selecting optimal kernel sizes based on specific task characteristics? Deeper analysis of kernel size influence could lead to more fine-grained architectural optimization. | |
| MotionGS: Exploring Explicit Motion Guidance for Deformable 3D Gaussian Splatting (Read more on arXiv or HuggingFace) | ztz1989, jiahao97, Free1unch, Rosetta-Leong, RuijieZhu | a) The paper aims to improve dynamic scene reconstruction quality and robustness by incorporating explicit motion priors into deformable 3D Gaussian Splatting (3DGS). b) MotionGS, the proposed framework, decouples optical flow into camera and motion flow, using the latter to guide 3D Gaussian deformation. It also incorporates a camera pose refinement module that alternately optimizes 3D Gaussians and camera poses. c) On the NeRF-DS dataset, MotionGS achieves a mean PSNR of 24.54, outperforming the baseline method (Deformable 3DGS) which achieved 23.61. d) AI practitioners can use MotionGS to reconstruct dynamic scenes from monocular video with improved quality and robustness compared to existing deformable 3DGS methods, especially in scenarios involving complex or rapid motion. The CUDA-based implementation of the Gaussian flow and camera pose optimization allows for efficient training and rendering. Follow-up questions: 1. Could the optical flow decoupling module be adapted or improved for scenes where segmentation masks for dynamic objects are not readily available or easily obtained? 2. How does the computational cost of the motion flow extraction and camera pose refinement impact real-time rendering performance, and what are the potential optimization strategies to mitigate this? 3. How sensitive is MotionGS to the accuracy of the initial camera poses provided by COLMAP, and are there alternative initialization strategies that could further improve robustness in challenging scenarios? |
Papers for 2024-10-10
| Title | Authors | Summary |
|---|---|---|
| GLEE: A Unified Framework and Benchmark for Language-based Economic Environments (Read more on arXiv or HuggingFace) | Roi Reichart, Samuel Joseph Amouyal, Omer Madmon, ireinman, EilamSha | a) This research aimed to create a standardized framework for evaluating large language model (LLM) agents in language-based economic games and comparing their behavior to humans. b) The researchers developed GLEE, a framework parameterizing bargaining, negotiation, and persuasion games, controlling for game horizon, information structure, and communication form. They collected a dataset of LLM vs. LLM interactions (7.15M decisions in 954K games across four LLMs) and human vs. LLM interactions (3.4K games across 195 configurations, played on a custom-built interface). Regression models were used to predict metric values for uncollected configurations, enabling cross-model comparison. c) Humans outperformed LLMs in bargaining as the proposer (Alice) but performed worse as the responder (Bob), while in negotiation, LLMs generally achieved positive self-gain compared to humans’ negative average self-gain. d) AI practitioners can use GLEE and its accompanying dataset to benchmark and compare LLM performance across various economic game scenarios, potentially leading to the development of more effective and human-like agents for applications requiring strategic decision-making in natural language. The paper highlights the sensitivity of average metric values to configuration distributions, suggesting practitioners consider specific application contexts when designing LLM agents for economic interactions. Follow-up questions: 1. How does the choice of LLM architecture (e.g., transformer size, decoder-only vs. encoder-decoder) affect agent performance within the GLEE framework, and are there specific architectures better suited for certain economic games? 2. Can the regression models used to predict metrics be improved by incorporating more sophisticated techniques (e.g., neural networks) or features derived from the text of the LLM-generated messages? 3. What specific prompt engineering strategies can be employed to mitigate the observed discrepancies between human and LLM performance in different roles within negotiation and bargaining games? |
| Personalized Visual Instruction Tuning (Read more on arXiv or HuggingFace) | Jipeng Zhang, Tianyang Han, research4pan, Sterzhang, renjiepi | a) This research aims to enhance Multimodal Large Language Models (MLLMs) to conduct personalized conversations, addressing their current limitation in recognizing specific individuals within images and generating corresponding information. b) The key methodology is Personalized Visual Instruction Tuning (PVIT), involving a data curation framework that synthesizes personalized training data using visual expert models, image generation models, and LLMs, and then fine-tunes the MLLM using this data. Personalized wrapper tokens are also introduced to prevent ambiguity when multiple individuals are present. c) On the P-Bench benchmark designed to evaluate personalized conversation abilities, PVIT-trained P-LLaVA achieves 96.69% average accuracy on answerable multiple-choice questions, significantly outperforming other SOTA MLLMs. d) AI practitioners can use PVIT to fine-tune MLLMs for enhanced personalization, enabling development of applications like personalized visual assistants or domestic robots capable of recognizing family members. The automatic data generation aspect of PVIT reduces the burden of manual data curation for personalized training. Follow-up questions: 1. Could the PVIT framework be adapted to personalize other aspects of MLLM responses beyond individual recognition, such as preferred conversational style or specific knowledge domains? 2. How does the computational cost of fine-tuning with PVIT compare to other personalization methods that introduce new parameters or model heads? 3. What are the limitations of the automatically generated personalized training data, and how can these be addressed to further improve the performance of personalized MLLMs? |
| Towards World Simulator: Crafting Physical Commonsense-Based Benchmark for Video Generation (Read more on arXiv or HuggingFace) | kpzhang, hflqf88888, wqshao126, ljq940913, FanqingM | a) This research investigates the ability of text-to-video (T2V) models to generate videos adhering to basic physical laws, a key step towards building world simulators. b) The authors introduce PhyGenBench, a benchmark with 160 prompts related to 27 physical laws, and PhyGenEval, a hierarchical evaluation framework utilizing vision-language models and large language models. c) Even the best-performing T2V model (Gen-3) achieved a low physical commonsense accuracy score of 0.51 on PhyGenBench. d) This highlights a significant limitation of current T2V models in accurately representing physical world dynamics, requiring AI practitioners to prioritize incorporating physical commonsense into model training beyond simply improving general video quality metrics. e) The paper mentions exploring scaling laws, prompt engineering, and video enhancement techniques as potential solutions but does not definitively quantify their impact on improving physical commonsense in generated videos. Follow-up questions: 1. Could providing T2V models with access to physics simulators or synthetic datasets during training improve their performance on PhyGenBench? 2. What specific architectural changes in T2V models might be most effective in enhancing their understanding of dynamic physical phenomena? 3. How can PhyGenEval be adapted or extended to evaluate more complex physical interactions and nuanced physical laws beyond those represented in the current PhyGenBench? |
| Deciphering Cross-Modal Alignment in Large Vision-Language Models with Modality Integration Rate (Read more on arXiv or HuggingFace) | Pan Zhang, Xiaoyi Dong, lindahua, yuhangzang, shikiw | a) This paper aims to develop a metric for evaluating the pre-training quality of Large Vision-Language Models (LVLMs) without requiring computationally expensive supervised fine-tuning. b) The researchers propose Modality Integration Rate (MIR), calculated by measuring the layer-wise Fréchet Inception Distance (FID) between vision and text token representations after text-centric normalization. c) MIR correlates strongly with post-supervised fine-tuning benchmark performance; for example, when pre-training LLaVA-1.5 7B with varying amounts of data, MIR effectively identified performance saturation at 800K-1M samples, while loss and perplexity continued to decrease beyond this point. d) AI practitioners can use MIR to optimize LVLM pre-training by efficiently identifying optimal data scales, detailedness, training strategies, and module designs without relying solely on costly downstream evaluation. This directly impacts model development efficiency. e) The paper does not provide a precise definition of “text-centric normalization”, though it mentions l2-normalization and a scaling factor. Follow-up questions: 1. Could the authors provide more detail on the implementation of “text-centric normalization,” including the outlier removal function and how the scaling factor αk is specifically computed for each layer k? 2. How computationally efficient is MIR to calculate compared to traditional metrics, and does its computational cost scale linearly with the number of samples used? 3. While MIR correlates with downstream performance, does minimizing MIR during pre-training guarantee optimal downstream performance, or are there other factors to consider? |
| IterComp: Iterative Composition-Aware Feedback Learning from Model Gallery for Text-to-Image Generation (Read more on arXiv or HuggingFace) | Ling Yang, Thu-redrobot, kelisiya, yaqicc, comin | a) The research aims to improve compositional text-to-image generation by leveraging the strengths of multiple diffusion models. b) IterComp aggregates composition-aware model preferences from a “gallery” of six diffusion models and uses iterative feedback learning with trained reward models to refine a base diffusion model (SDXL). c) IterComp outperforms other models on the T2I-CompBench in complex composition generation, achieving a score of 0.4873 compared to the second-best score of 0.4312. d) AI practitioners can use IterComp to fine-tune existing text-to-image models for improved performance in complex compositional scenarios, leveraging the framework’s ability to integrate preferences from multiple models. Follow-up Questions: 1. The paper mentions progressively expanding the model gallery. What criteria are used for selecting new models to add, and how does this expansion affect the computational cost of training and inference? 2. What are the specific architectural details of the composition-aware reward models, and how are the image and text features combined within them? The paper mentions BLIP and cross-attention, but more detail would be beneficial for replication. 3. How robust is IterComp to variations in the initial base diffusion model? Would similar improvements be observed if a different base model was used, and does the choice of initial model influence the optimal model gallery composition? |
| Aria: An Open Multimodal Native Mixture-of-Experts Model (Read more on arXiv or HuggingFace) | JunnanLi, guoyinwang, sirius-ctrl, teowu, dxli1 | This research aims to develop an open-source, multimodal native Mixture-of-Experts (MoE) model with strong capabilities across diverse modalities. The authors pre-trained ARIA, a fine-grained MoE decoder with a lightweight visual encoder, from scratch using a 4-stage pipeline focused on language, multimodal understanding, long context, and instruction following, with 6.4T language and 400B multimodal tokens. ARIA achieved 65.3% accuracy on the LongVideoBench (test set), outperforming Pixtral-12B and Llama3.2-11B. This provides AI practitioners with an accessible and high-performing open-source model for multimodal applications, particularly those involving long sequences and diverse data types. The paper does not explicitly detail the specific architectures of competing models, or the hardware used in the various experiments. Follow-up questions: 1. Could the authors provide more details on the specific architecture of the visual encoder and how it handles different image resolutions and video input? This would be helpful for understanding how the model processes and integrates visual information. 2. The paper mentions a 4-stage training pipeline. Could the authors provide more quantitative details on the data and compute resources allocated to each stage? This would clarify the resource requirements for replicating or adapting the training process. 3. How does ARIA’s performance compare to proprietary models on tasks that specifically test fine-grained multimodal reasoning capabilities, such as detailed image captioning or visual question answering with complex reasoning steps? This is crucial for understanding the model’s strengths and weaknesses in real-world scenarios. |
| Pixtral 12B (Read more on arXiv or HuggingFace) | saurabhgarg, devendrachaplot, EmmaBH, Simontwice, pragra | a) This research introduces Pixtral 12B, a 12-billion parameter multimodal language model designed to understand both images and text, aiming to achieve strong performance on multimodal benchmarks without compromising text-only reasoning capabilities. b) Pixtral 12B utilizes a novel vision encoder trained from scratch to handle variable image sizes and aspect ratios, combined with a Mistral Nemo 12B decoder, and incorporates ROPE-2D for relative position encoding. Evaluation was performed on existing and newly created benchmarks, including a novel multimodal benchmark, MM-MT-Bench, designed for practical multi-turn scenarios. c) Pixtral 12B outperforms all open-source models of similar size on the MM-MT-Bench benchmark, achieving a score of 6.05, and exhibits competitive performance compared to larger models on established multimodal and text-only benchmarks. d) Pixtral 12B offers AI practitioners a powerful, open-source, multimodal model with strong performance on a range of tasks, potentially serving as a drop-in replacement for existing text-only or less capable multimodal deployments. The introduction of MM-MT-Bench provides a new benchmark for evaluating practical multimodal use cases. Follow-up questions: 1. What are the specific architectural details of the Pixtral-ViT vision encoder, including the number of layers, attention heads, and hidden dimension? 2. How does the performance of Pixtral 12B compare to closed-source models like GPT-4 on more complex, real-world image understanding tasks? 3. What are the limitations of Pixtral 12B in terms of image resolution, complexity, or specific modalities (e.g., video, audio)? |
| Unveiling the Backbone-Optimizer Coupling Bias in Visual Representation Learning (Read more on arXiv or HuggingFace) | szli-0000, sunbaigui, SOTA-Owner, ZCLiu35, ZedongWangAI | This paper investigates the interplay between vision backbones and optimizers, questioning their assumed independent applicability. Researchers benchmarked 20 backbones (CNNs, ViTs, etc.) against 20 optimizers (SGD, AdamW, etc.) on CIFAR-100, ImageNet, and COCO, evaluating accuracy, hyperparameter robustness, and learned parameter patterns. Results revealed a backbone-optimizer coupling bias (BOCB), where classical CNNs perform better with SGD families, while modern architectures like ViTs favor adaptive learning rate optimizers; for example, ConvNeXt-T achieved 86.19% top-1 accuracy with AdamW but only 33.26% with LARS on CIFAR-100. This implies that AI practitioners should carefully consider the backbone-optimizer pairing, as BOCB can significantly impact performance and generalization. The paper mentions analyzing learned parameter patterns, but specifics of the analysis methods and quantitative results are unclear within the abstract and first page. Follow-up questions: 1. Could the authors elaborate on the specific metrics used to analyze learned parameter patterns (e.g., PL exponent alpha, entropy, L2-norm, PCA energy ratio) and provide quantitative results or visualizations showcasing these patterns for different backbone-optimizer combinations? 2. How does the severity of BOCB vary across different downstream tasks and datasets beyond image classification (e.g., object detection, segmentation)? Are there specific tasks or datasets where BOCB is more or less pronounced? 3. The paper mentions “insights on more robust vision backbone design” - can the authors provide specific examples of design modifications or principles that could mitigate BOCB and improve overall robustness to optimizer choice? |
| Pyramidal Flow Matching for Efficient Video Generative Modeling (Read more on arXiv or HuggingFace) | quzhe, Payne53, Ninggggy, feifeiobama, rain1011 | a) The research aims to develop a more computationally efficient video generation model than existing cascaded approaches. b) The authors propose “pyramidal flow matching,” reinterpreting the denoising trajectory as a series of pyramid stages operating on compressed representations, combined with a temporal pyramid for autoregressive history conditioning, and implemented within a single Diffusion Transformer. c) The method enables generation of 5-second 768p videos at 24 FPS with 20.7k A100 GPU training hours and achieves a quality score of 84.74 on VBench, outperforming other open-source models. d) AI practitioners can utilize this approach to train high-quality video generation models with significantly reduced computational costs and training time compared to full-sequence diffusion models. The impactful finding is the substantial reduction in training compute, enabling faster iteration and experimentation with large video models. Follow-up questions: 1. What is the detailed architecture of the 3D VAE used for spatiotemporal compression, and how does its performance compare to other video compression techniques in terms of reconstruction quality and compression ratio? 2. How does the proposed pyramidal flow matching method scale with increasing video length and resolution, and what are the practical limitations in terms of maximum video duration and resolution that can be achieved with reasonable computational resources? 3. Could the authors elaborate on the specific implementation details of the “corrective Gaussian noise” and its impact on the continuity of the generated video across different pyramid stages? |
| MM-Ego: Towards Building Egocentric Multimodal LLMs (Read more on arXiv or HuggingFace) | HaoxuanYou, FrozzZen, edaxberger, haotiz, leoye | This research aims to build a multimodal foundation model for understanding egocentric videos. The authors developed a “narration to egocentric QA” data engine to generate 7M QA samples from Ego4D narrations, a Memory Pointer Prompting mechanism within a multimodal LLM architecture, and a new benchmark called EgoMemoria containing 7,026 multiple-choice questions across 629 egocentric videos. MM-Ego, the resulting model, achieves a Mean Debiased Accuracy (MDA) of 61.27% on EgoMemoria, outperforming other models. This provides AI practitioners with a new model and benchmark for developing and evaluating egocentric video understanding systems, advancing the field of egocentric AI. Follow-up Questions: 1. How does the Memory Pointer Prompting mechanism’s computational cost scale with increasing video length compared to existing long-context transformer approaches? 2. What specific types of egocentric video understanding tasks, beyond episodic memory, could benefit from the MM-Ego model and EgoMemoria benchmark, and how might the dataset and model need to be adapted? 3. How robust is the “narration to egocentric QA” data engine to variations in narration quality and style, and what measures are taken to mitigate potential biases introduced during data generation? |
| One Initialization to Rule them All: Fine-tuning via Explained Variance Adaptation (Read more on arXiv or HuggingFace) | Marc Peter Deisenroth, Benedikt Alkin, thomasschmied, sirluk, paischer101 | a) The paper investigates how to improve the initialization of Low-Rank Adaptation (LoRA) for fine-tuning foundation models to enhance convergence and downstream task performance. b) Explained Variance Adaptation (EVA) initializes LoRA’s new weights using a data-driven approach: performing Singular Value Decomposition (SVD) on minibatches of activation vectors from the downstream task data, sorting right-singular vectors by explained variance, and using the top-k components for initialization. Ranks are re-distributed among weight matrices to maximize explained variance. c) EVA combined with DORA achieved 73.5% accuracy on BoolQ, outperforming standard LoRA (67.2%) and other baselines on a suite of language generation tasks when fine-tuning Llama-2-7B. d) AI practitioners can leverage EVA to potentially accelerate fine-tuning and improve the performance of foundation models on downstream tasks by using a more informed initialization strategy for LoRA, focusing compute resources on rank adaptation, rather than uniform rank distribution across layers. Follow-up Questions: 1. The paper mentions computational overhead for the initial SVD computation, but doesn’t quantify it relative to the subsequent fine-tuning process. What is the time and memory cost of the EVA initialization compared to the overall fine-tuning time and memory usage for various model sizes? 2. How does the choice of the rank redistribution hyperparameter p affect the trade-off between performance and computational cost during initialization and fine-tuning, and are there any heuristics for choosing an appropriate p for a new dataset or task? 3. The paper focuses on vision, language, and reinforcement learning tasks. How well does EVA generalize to other modalities or model architectures beyond transformers? |
| Story-Adapter: A Training-free Iterative Framework for Long Story Visualization (Read more on arXiv or HuggingFace) | Yunfei Xie, RitaCoding, MudeHui, xk-huang, JohnWeck | a) The paper addresses the challenge of maintaining semantic consistency and generating fine-grained interactions in long story visualization (up to 100 frames) using text-to-image diffusion models. b) The proposed Story-Adapter framework uses an iterative paradigm, refining generated images based on text prompts and all previously generated images from the prior iteration, utilizing a training-free global reference cross-attention (GRCA) mechanism. c) Story-Adapter achieves a 9.4% improvement in average Character-Character Similarity (aCCS) compared to the StoryGen baseline on the StorySalon dataset for regular-length story visualization. d) AI practitioners can leverage Story-Adapter to generate more coherent and higher-quality visualizations of long stories without requiring additional training of the underlying diffusion model, simplifying integration and deployment. The impactful finding is the iterative refinement with GRCA, which allows for the integration of global story context without the computational expense of methods like Consistent Self-Attention. Follow-up questions: 1. How does the linear weighting strategy for fusing text and image modalities in Story-Adapter impact the trade-off between text adherence and visual consistency across different story genres or artistic styles? 2. Could the GRCA module be adapted to other generative tasks beyond story visualization, such as video generation or 3D scene synthesis, and what modifications might be necessary for optimal performance? 3. What are the practical memory and latency considerations for deploying Story-Adapter for real-time or interactive story visualization applications? |
| Self-Boosting Large Language Models with Synthetic Preference Data (Read more on arXiv or HuggingFace) | Zhifang Sui, Li Dong, thegenerality, THU-CHUNXIA, Rsy24 | a) The research aimed to develop a method for continually improving Large Language Models (LLMs) without the resource-intensive collection of human preference data. b) The proposed method, SynPO, uses a self-boosting paradigm with synthetic preference data, involving a self-prompt generator, a response improver, and iterative preference optimization. c) After four SynPO iterations, Llama3-8B and Mistral-7B achieved over 22.1% win rate improvements on AlpacaEval 2.0 and ArenaHard. d) SynPO offers AI practitioners a more efficient and cost-effective way to align LLMs, reducing the need for extensive human annotation in preference learning. e) The paper focuses specifically on SimPO for the preference optimization stage but mentions compatibility with other methods like DPO and KTO without providing comparative results. Follow-up questions: 1. How does the performance of SynPO compare to other preference optimization methods like DPO and KTO when used within the SynPO framework, and what are the trade-offs in terms of computational cost and alignment effectiveness? 2. What specific strategies were used to mitigate potential biases introduced by the synthetic data generation process, and how was the quality and diversity of the synthetic data evaluated beyond inter-prompt similarity and GPT-4 topic classification? 3. Could the authors elaborate on the limitations of using the initial model outputs as a proxy for gold-standard responses in the early stages of SynPO, especially concerning the potential for reinforcing existing model biases and limitations? |
| Falcon Mamba: The First Competitive Attention-free 7B Language Model (Read more on arXiv or HuggingFace) | Ilyas Chahed, Dhia Eddine Rhaiem, ybelkada, yellowvm, JingweiZuo | a) This research investigated whether a purely attention-free State Space Language Model (SSLM) could achieve competitive performance compared to Transformer-based models at a 7B scale. b) The researchers developed Falcon Mamba 7B, a 7B parameter language model based on the Mamba architecture, trained on 5.8 trillion tokens. c) Falcon Mamba 7B achieved an average score of 64.09 across six benchmarks in Hugging Face Leaderboard v1 (ARC-25, HellaSwag-10, MMLU-5, Winogrande-5, TruthfulQA-0, GSM8K-5), outperforming similarly sized models, including Llama3.1 8B and Mistral 7B. d) AI practitioners can consider using pure Mamba-based architectures for tasks requiring long sequence generation, as Falcon Mamba 7B demonstrates competitive performance with lower memory and computational costs compared to transformers, especially with long sequences. It also offers an alternative for scaling LLMs. Follow-up Questions: 1. While Falcon Mamba 7B shows strong performance in few-shot learning, the paper briefly mentions limitations in in-context learning. What specific experiments were conducted to evaluate in-context learning, and what were the quantitative results compared to transformers? 2. The paper highlights the advantage of constant memory usage during generation with Mamba architecture. Was the impact of sequence length during training also explored and if so what are the observed trade-offs on the resultant model’s performance on downstream tasks? 3. What specific techniques or strategies were used for model initialization and learning rate adjustment during training to address the reported loss spikes and divergence issues with the Mamba architecture? |
| TweedieMix: Improving Multi-Concept Fusion for Diffusion-based Image/Video Generation (Read more on arXiv or HuggingFace) | Jong Chul Ye, gkwon | a) The research aims to improve the generation of images and videos containing multiple user-specified concepts using diffusion models, addressing limitations in existing methods regarding concept blending and scalability. b) TweedieMix divides the reverse diffusion sampling process into two stages: initial multi-object-aware sampling using a base model and a novel resampling strategy, followed by integrating concept-specific fine-tuned models through region-wise guidance and mixing in the Tweedie’s denoised image space. For video generation, a training-free approach injects features from a keyframe generated with the multi-concept image generation method into subsequent frames of a pre-trained image-to-video diffusion model. c) TweedieMix achieves a higher CLIP score (Text-sim: 0.3872, Image-sim: 0.8202) compared to baseline multi-concept generation methods, indicating improved text-alignment and image-alignment. d) AI practitioners can leverage TweedieMix to develop applications generating high-fidelity images and videos with multiple user-defined concepts without extensive model fine-tuning or complex weight merging procedures, facilitating easier customization of generative models. Follow-up questions: 1. The paper mentions limitations with highly complex text prompts. What specific metrics quantify this limitation, and how might these limitations be addressed in future work, beyond upgrading the diffusion backbone? 2. Could the feature injection technique used for video generation be adapted or optimized for other video diffusion models beyond I2VGen-XL? How sensitive is the video generation quality to the selection of frames for feature injection? |
| Temporal Reasoning Transfer from Text to Video (Read more on arXiv or HuggingFace) | Chancy, PY007, yaolily, lyx97, tobiaslee | a) This research investigates the bottleneck in Video Large Language Models’ (LLMs) ability to perform temporal reasoning tasks. b) The researchers conducted probing experiments on synthesized videos and corresponding text descriptions, comparing the performance of full Video LLMs, LLM decoders, and visual feature encoders. They then introduced Textual Temporal reasoning Transfer (T3), which synthesizes textual temporal reasoning tasks from image-text datasets and fine-tunes LongVA-7B on this data. c) Results indicate that the LLM decoder is the primary bottleneck in video temporal reasoning, as visual encoders achieved high accuracy on probing tasks while LLMs struggled even with textual temporal questions. T3 improved LongVA-7B’s temporal understanding, leading to a 5.3 absolute accuracy improvement on the TempCompass benchmark. d) AI practitioners developing Video LLMs should focus on enhancing the temporal reasoning capabilities of the underlying LLM rather than solely focusing on visual feature encoding. Textual temporal reasoning datasets synthesized from existing image-text data offer a scalable and efficient method for improving Video LLM performance in this area. Follow-up questions: 1. What specific architectural modifications or training strategies could further enhance the LLM’s ability to handle temporal information beyond the T3 approach? 2. How does the performance of T3 scale with larger LLMs and more complex temporal reasoning tasks beyond those explored in the paper? 3. Could the synthesized textual temporal datasets be beneficial for training other temporal reasoning tasks beyond video understanding, such as natural language understanding of event sequences or time series data? |
| TRACE: Temporal Grounding Video LLM via Causal Event Modeling (Read more on arXiv or HuggingFace) | Xiaoying Tang, Mingda Li, Jingyu Liu, qingbinliu, Yongxin-Guo | a) The research aimed to address the mismatch between the inherent structure of videos and the language modeling approach of current Video Large Language Models (LLMs) for Video Temporal Grounding (VTG) tasks. b) The authors proposed a causal event modeling framework, representing videos as sequences of events with timestamps, salient scores, and captions, and developed TRACE, a task-interleaved video LLM, to implement this framework. TRACE processes visual frames, timestamps, salient scores, and text as separate tasks with dedicated encoders and decoding heads, sequencing these tasks according to the causal framework. c) TRACE demonstrated superior zero-shot performance on various VTG tasks, improving CIDEr score by 3.1% and F1 score by 4.9% on YouCook2 compared to existing video LLMs. d) For AI practitioners, TRACE offers a more effective architecture for developing video LLMs for VTG tasks, potentially enabling improvements in downstream applications like moment retrieval, dense video captioning, and highlight detection. The improved zero-shot performance reduces the reliance on resource-intensive fine-tuning for numerous tasks. Follow-up questions: 1. How does the adaptive head-switching mechanism in TRACE specifically contribute to the improved generation performance, and what are its limitations in handling complex event transitions within videos? 2. The paper mentions filtering and re-annotation of some datasets. What specific criteria were used for these processes, and how might these modifications affect the generalizability of TRACE to other VTG datasets with different annotation styles? 3. What is the computational overhead of the separated multi-task processing approach compared to existing video LLMs, and how can this be optimized for real-world deployment in resource-constrained environments? |
| Data Selection via Optimal Control for Language Models (Read more on arXiv or HuggingFace) | Li Dong, thegenerality, Rsy24, howang, t1101675 | a) The research investigates selecting high-quality pre-training data from large corpora to improve language model (LM) performance and training efficiency. b) The authors formulate data selection as an Optimal Control problem, leveraging Pontryagin’s Maximum Principle (PMP) to derive necessary conditions for optimal data selection and develop a framework called PMP-based Data Selection (PDS). PDS assigns quality scores to instances based on their impact on downstream tasks using a proxy dataset and trains a data scorer to predict these scores for the entire corpus. c) Experiments show that pre-training a 1.7B parameter LM on a PDS-selected corpus achieves a 2.0x speedup compared to conventional pre-training on a uniformly sampled corpus. d) PDS offers a principled method for data selection that can significantly accelerate LM training and improve downstream task performance, mitigating the increasing computational demands of pre-training large language models. Follow-up Questions: 1. How does the performance of PDS compare to online data selection methods in terms of both computational cost and downstream task performance for models of various scales? 2. What are the limitations of using a proxy dataset and data scorer, and how can these limitations be addressed to further improve the quality of selected data, especially for domain-specific applications? 3. How robust is PDS to the choice of downstream task used for calculating the data quality scores, and how can this choice be optimized for specific downstream applications or when multiple downstream tasks are of interest? |
| CursorCore: Assist Programming through Aligning Anything (Read more on arXiv or HuggingFace) | Shijin Wang, Rui Li, Qi Liu, Eviloder, TechxGenus | This research aims to improve AI-assisted programming by aligning models with diverse information sources during the coding process. The authors introduce a novel conversational framework, Assistant-Conversation, and a data synthesis pipeline, Programming-Instruct, to generate a 219K sample dataset used to train the CursorCore LLM series. On the Assist Programming Eval (APEval) benchmark, CursorCore-1.3B achieves a 10.4% higher Pass@1 score than the best comparable model. This suggests that training specialized LLMs on comprehensive coding process data significantly enhances programming assistance performance. Follow-up questions: 1. How does the performance of CursorCore vary across different programming languages beyond Python, and what adaptations are necessary for broader language support? 2. What specific techniques are used in the Programming-Instruct pipeline to handle complex code changes and ensure the generated data reflects realistic coding scenarios? 3. How robust is CursorCore to noisy or incomplete coding history information, and how does the model handle such situations in practice? |
| ViBiDSampler: Enhancing Video Interpolation Using Bidirectional Diffusion Sampler (Read more on arXiv or HuggingFace) | Jong Chul Ye, Taesung Kwon, sr2851766 | a) The paper aims to enhance video keyframe interpolation quality by addressing off-manifold issues encountered by existing time-reversal fusion methods in image-to-video diffusion models. b) The proposed ViBiDSampler employs a bidirectional sampling strategy, sequentially denoising along forward and backward temporal paths conditioned on start and end frames, respectively, combined with Classifier-Free Guidance++ (CFG++) and Diffusion Denoising Score (DDS) for on-manifold guidance. c) On the DAVIS dataset, ViBiDSampler achieved an LPIPS score of 0.2355, outperforming baseline methods such as FILM (0.2697), TRF (0.3102), DynamiCrafter (0.3274), and Generative Inbetweening (0.2823). d) AI practitioners can utilize ViBiDSampler as a more efficient and effective method for video keyframe interpolation, potentially reducing artifacts and improving perceptual quality without the need for model fine-tuning or multiple re-noising steps as required by some existing methods. Follow-up questions: 1. How does the computational cost of ViBiDSampler’s bidirectional sampling compare to TRF and Generative Inbetweening, considering both the number of function evaluations and wall-clock time, specifically for higher-resolution video generation beyond 1024×576? 2. How robust is ViBiDSampler to variations in the temporal distance between keyframes? Does performance degrade significantly with larger gaps, and are there strategies within the bidirectional sampling framework to mitigate this? 3. What are the limitations of using CLIP image embeddings as conditioning, and could alternative or complementary conditioning methods further improve the coherence and fidelity of the interpolated frames, particularly for videos containing complex semantic content? |
| Response Tuning: Aligning Large Language Models without Instruction (Read more on arXiv or HuggingFace) | Hyounghun Kim, seokhyun | a) This research investigates whether establishing a response space alone, without instruction-response mappings, can align pre-trained Large Language Models (LLMs) for instruction following and safety. b) The authors propose Response Tuning (RT), which omits the instruction-conditioning step in conventional instruction tuning and trains LLMs solely on responses. They compare RT models to instruction-tuned models on various benchmarks. c) RT models achieved comparable performance to instruction-tuned counterparts on several evaluations, achieving a 91% acceptability rating for Llama-3.1-8B trained with Alpaca responses. d) The study suggests that instruction-following capabilities may be largely acquired during pre-training and that establishing an appropriate response space alone can effectively surface these capabilities, simplifying alignment procedures for AI practitioners. e) The paper claims that the structural attributes of training responses impact user preference, but it’s not fully clear how these attributes are quantitatively measured or controlled, despite mentioning the use of a refinement prompt with a stronger LLM. Follow-up questions: 1. Can the authors provide more details on the refinement prompt used to control structural attributes, including specific examples and how effectiveness was measured beyond GPT-4 pairwise comparisons? 2. How does the performance of RT scale with significantly larger models and datasets, and are there any observed limitations in terms of complexity or generalization of instructions? 3. What are the computational resource (time, memory, compute) implications of RT compared to traditional instruction tuning, specifically regarding training and inference? |
| ING-VP: MLLMs cannot Play Easy Vision-based Games Yet (Read more on arXiv or HuggingFace) | Haoran Zhang, zhangysk, CheeryLJH, EZ-hwh, Rosiness | This research investigates the spatial imagination and multi-step reasoning abilities of Multimodal Large Language Models (MLLMs) in vision-based planning. The authors introduce ING-VP, a benchmark comprising six games with varying levels, evaluated across six inference settings (image/text input, single/multi-step reasoning, with/without history). Evaluation of 15 MLLMs showed even the top-performing model, Claude-3.5 Sonnet, achieved an average accuracy of only 3.37%. This suggests current MLLMs have significant limitations in spatial reasoning and planning, particularly in accurately processing the relative positions of visual elements. AI practitioners should consider these perceptual limitations and lack of robust planning capabilities when developing or applying MLLMs for tasks requiring spatial understanding and interaction. Follow-up questions: 1. How does the performance of MLLMs in ING-VP compare to specifically designed spatial reasoning models that are not LLMs? 2. What specific architectural changes or training strategies could be explored to improve MLLMs’ performance on tasks requiring precise location understanding within images? 3. The paper mentions subtle prompt variations impacting model outputs; could further investigation reveal specific prompt engineering techniques to mitigate some of these inconsistencies? |
| Mixed-Session Conversation with Egocentric Memory (Read more on arXiv or HuggingFace) | Taeyoung Kim, khh3323, jihyoung | a) The research aimed to develop a dialogue system capable of managing multi-session conversations with varying partners while maintaining contextual coherence. b) A new dataset, MISC, containing 8.5K episodes of six-session dialogues with four speakers (one main, three partners) and a novel dialogue model, EMMA (Egocentric Memory Enhanced Mixed-session Conversation Agent), using egocentric memory management were introduced. c) Human evaluation of MISC showed high consistency (4.83-4.9 across three annotator groups) and coherence (4.78-4.85) scores. d) AI practitioners can utilize the MISC dataset and the EMMA model’s egocentric memory approach to build more coherent and consistent multi-session, multi-partner conversational AI systems. The high consistency score suggests this approach is effective in maintaining continuity across sessions with different partners. Follow-up questions: 1. How does EMMA’s retrieval module specifically prioritize relevant memories from previous sessions, given that it has access to all past interactions? More details on the retrieval module’s architecture and training process would be beneficial. 2. What are the limitations of using GPT-3.5 for dialogue generation after using GPT-4 for scenario generation, and how might this impact the overall quality and consistency of the MISC dataset? 3. Could the authors provide further details on the computational resources required to train EMMA, particularly the dialogue and retrieval modules? This information would be crucial for practitioners considering replicating or adapting the model. |
| Retrieval-Augmented Decision Transformer: External Memory for In-context RL (Read more on arXiv or HuggingFace) | Markus Hofmarcher, razp, vihangp, paischer101, thomasschmied | a) The research aimed to improve in-context reinforcement learning (ICL) in environments with long episodes and sparse rewards, which pose challenges for existing ICL methods that rely on full episode contexts. b) The authors introduced Retrieval-Augmented Decision Transformer (RA-DT), which integrates an external memory mechanism with a Decision Transformer (DT). RA-DT retrieves relevant sub-trajectories from the memory using a pre-trained embedding model and incorporates them into the DT via cross-attention. c) RA-DT outperformed baseline ICL methods on grid-world environments, achieving near-optimal performance on Dark-Room 10x10 while using a context length of 50 transitions compared to baselines using a context length of 2400. While RA-DT showed improved average performance on more complex environments like Meta-World, DMControl and Procgen, no in-context improvement was observed on hold-out tasks in these environments. d) AI practitioners can leverage RA-DT to potentially reduce the computational cost and improve the effectiveness of ICL in certain RL environments, particularly those with long episodes that are computationally prohibitive for traditional ICL methods. The lack of ICL improvement on hold-out tasks for more complex environments suggests that further research is needed to improve retrieval techniques or conditioning strategies, highlighting a current limitation of offline, next-action prediction based ICL methods. Follow-up questions: 1. How does the performance of RA-DT vary with the size and diversity of the external memory, and what strategies can be used to optimize memory construction for specific domains? 2. What modifications to the retrieval mechanism or the DT architecture could enable more effective meta-learning in complex environments, leading to stronger ICL performance on hold-out tasks? 3. Could incorporating online learning or value function estimation into the RA-DT framework address the limitations observed in next-action prediction ICL and improve performance in complex, fully-observable environments? |
| FürElise: Capturing and Physically Synthesizing Hand Motions of Piano Performance (Read more on arXiv or HuggingFace) | C. Karen Liu, Elizabeth Schumann, Haochen Shi, Pei Xu, rcwang | a) The research aims to capture and synthesize physically plausible 3D hand motions of piano performances for novel musical pieces. b) A large-scale dataset (“FürElise”) of 10 hours of hand motion data from 15 pianists was collected using multi-view video and refined with inverse kinematics informed by MIDI data. A control policy was trained using reinforcement learning with imitation and goal-based rewards, leveraging diffusion-generated motions and music-based motion retrieval from the dataset. c) The trained policy, evaluated on 14 unseen musical pieces, achieved an average F1-score of over 0.8, significantly outperforming diffusion-generated motions alone. d) AI practitioners can utilize the FürElise dataset and the proposed pipeline combining diffusion models, motion retrieval, and reinforcement learning to synthesize realistic and dexterous hand motions for complex tasks, particularly in domains requiring precise physical interaction, such as character animation and robotics. Follow-up Questions: 1. How does the proposed method address the limitations of diffusion models in generating physically plausible motions, specifically regarding the penetration and floating artifacts often observed in hand-object interactions? What specific techniques are employed in the inverse kinematics refinement stage to address artifacts and ensure synchronized hand motion with MIDI key press events? 2. Could details be provided on the architecture and training process of the discriminator network used for imitation learning? What loss function is employed, and how is the balance between imitation and goal-based rewards managed during training? |
| AutoDAN-Turbo: A Lifelong Agent for Strategy Self-Exploration to Jailbreak LLMs (Read more on arXiv or HuggingFace) | Edward Suh, huansun, someshjha, peiranli0930, ShletonLiu-N | AutoDAN-Turbo aims to automatically discover and combine jailbreak strategies for large language models (LLMs). The method utilizes a lifelong learning agent with three modules: attack generation and exploration, strategy library construction, and jailbreak strategy retrieval. AutoDAN-Turbo achieved an 88.5% attack success rate on GPT-4-1106-turbo, a 74.3% improvement over the runner-up on the HarmBench dataset. This implies that AutoDAN-Turbo can effectively bypass the safety alignment of even highly robust LLMs. Follow-up questions: 1. How does the strategy library construction module address the potential for redundant or similar strategies being discovered? 2. What specific metrics were used to evaluate the “maliciousness” of the LLM responses, and how was the scorer LLM trained to apply these metrics? 3. What are the limitations of using only textual output for black-box attacks, and what potential avenues exist for incorporating other modalities (e.g., image generation) into the framework? |
| Multimodal Situational Safety (Read more on arXiv or HuggingFace) | xw-eric, dawnsong, acompalas, Xuandong, LCZZZZ | a) This research investigates how effectively Multimodal Large Language Models (MLLMs) assess the safety of user queries or instructions based on the visual context, a problem termed “Multimodal Situational Safety.” b) Researchers created a new benchmark, MSSBench, comprising 1820 image-query pairs across “chat” and “embodied” scenarios, and evaluated eight MLLMs using an accuracy-based metric. They also introduced multi-agent pipelines to improve situational safety reasoning. c) Current MLLMs struggle with this task; the highest-performing model, Claude 3.5 Sonnet, achieved only 62.2% average accuracy. d) AI practitioners developing multimodal assistants should prioritize improving situational safety awareness in MLLMs, as current models exhibit significant limitations in integrating visual context for safe responses, especially in embodied scenarios. This highlights a critical area for further research and development to prevent unsafe actions or advice in real-world applications. Follow-up questions: 1. How does the performance of multi-agent pipelines vary across different MLLM architectures and sizes, and what architectural modifications could further enhance their effectiveness in situational safety assessment? 2. What specific safety training strategies could be employed to address the over-sensitivity observed in some MLLMs while simultaneously improving their ability to recognize genuinely unsafe situations in embodied scenarios? 3. What are the practical considerations (e.g., latency, computational cost) for deploying the proposed multi-agent pipelines in real-world multimodal assistant applications, and how can these be optimized for efficient and safe operation? |
| T2V-Turbo-v2: Enhancing Video Generation Model Post-Training through Data, Reward, and Conditional Guidance Design (Read more on arXiv or HuggingFace) | wangwilliamyang, wenhu, rpiramuthu, xfgao, jiachenli-ucsb | a) The research aimed to enhance a pre-trained text-to-video (T2V) model during post-training by incorporating supervision signals from high-quality data, reward models, and conditional guidance. b) The core methodology involved consistency distillation (CD) augmented with classifier-free guidance (CFG) and motion guidance derived from temporal attention, along with reward optimization from a mixture of image-text and video-text reward models (RMs). A preprocessing step pre-calculates the computationally expensive motion guidance term. c) T2V-Turbo-v2 achieved a state-of-the-art Total Score of 85.13 on VBench, surpassing proprietary systems like Gen-3 and Kling. d) The research demonstrates the critical importance of dataset selection and RM diversity for effective T2V model post-training, offering AI practitioners valuable insights into improving video generation quality and text alignment. The preprocessing approach to incorporating motion guidance presents a practical solution for managing computational cost. Follow-up questions: 1. How does the performance of T2V-Turbo-v2 vary across different pre-trained T2V models, and are there specific architectural features that make some models more amenable to this post-training approach? 2. What is the computational cost and memory footprint of the preprocessing step, and how does it scale with the size of the training dataset? 3. How robust is the motion guidance to variations in video quality within the training dataset, and are there techniques to mitigate potential negative impacts from lower-quality videos? |
| Multimodal Large Language Models for Inverse Molecular Design with Retrosynthetic Planning (Read more on arXiv or HuggingFace) | Jie Chen, Wojciech Matusik, Michael Sun, Gang Liu, mjiang89 | a) This research investigates the limitations of large language models (LLMs) in controllable and synthesizable molecular design, proposing a multimodal LLM (MLLM) called Llamole to address these challenges. b) Llamole integrates a base LLM with a Graph Diffusion Transformer (Graph DiT) for molecule generation, a Graph Neural Network (GNN) for reaction prediction, and A* search for retrosynthetic planning, utilizing a trigger-query-prediction approach to control the interleaved generation of text and graphs. c) Llamole significantly outperforms 14 adapted LLMs across 12 metrics for controllable molecular design and increases retrosynthetic planning success rate from 5.5% to 35%. d) AI practitioners can leverage Llamole’s multimodal architecture for enhanced controllability and synthesizability in molecular design, potentially leading to more efficient and effective drug and material discovery. e) The enhanced performance of Llamole highlights the value of integrating LLMs with domain-specific graph modules for complex scientific applications. Follow-up questions: 1. What are the specific architectural details of the Graph DiT and GNN modules used in Llamole, and how were they pre-trained for molecular design tasks? 2. How does Llamole handle the trade-off between efficiency and effectiveness in multi-step retrosynthetic planning, particularly concerning the computational cost of A* search and the LLM-based cost function? 3. Could the trigger-query-prediction approach used in Llamole be generalized to other scientific domains involving graph-structured data, such as protein design or materials discovery? |
| BroadWay: Boost Your Text-to-Video Generation Model in a Training-free Way (Read more on arXiv or HuggingFace) | Pan Zhang, Pengyang Ling, Jiazi Bu, lindahua, yuhangzang | a) The paper investigates improving the quality of text-to-video (T2V) generation by addressing temporal inconsistency and limited motion magnitude, without requiring model retraining. b) BroadWay, a training-free method, is proposed, consisting of Temporal Self-Guidance (TSG), which reduces disparity between temporal attention maps across decoder blocks, and Fourier-based Motion Enhancement (FME), which amplifies high-frequency components of the temporal attention map. c) Experiments show that BroadWay improves video quality, with user studies demonstrating a preference for BroadWay-enhanced videos over vanilla T2V generated videos in 74.58% of cases for AnimateDiff and 69.46% of cases for VideoCrafter2. d) AI practitioners working on T2V generation can utilize BroadWay as a plug-and-play method to enhance the structural plausibility, temporal consistency, and motion magnitude of generated videos without requiring additional training or significant computational overhead. The significant improvement in user-perceived video quality highlights the potential for a better user experience in T2V applications. Follow-up questions: 1. How does the performance of BroadWay vary across different T2V architectures beyond AnimateDiff and VideoCrafter2, particularly those with diverse motion modules or training strategies? 2. What are the computational costs (e.g., latency) associated with applying BroadWay during inference, and how do these scale with video resolution and length? 3. Could the insights about the link between temporal attention maps and motion quality be leveraged to develop new, trainable modules for motion enhancement during the training phase of T2V models? |
| Collective Critics for Creative Story Generation (Read more on arXiv or HuggingFace) | Hyounghun Kim, minwook | a) This research aims to develop a framework for generating creative long-form stories with narrative coherence using Large Language Models (LLMs). b) The proposed Collective Critics for Creative Story Generation (CRITICS) framework integrates a collaborative critique mechanism into a plan-then-story generation process, using multiple LLM critics and a leader to iteratively refine story plans (CRPLAN) and enhance story expressiveness (CRTEXT). c) Human evaluation of 300 pairwise story plan comparisons showed CRITICS significantly outperformed the baseline DOC pipeline in interestingness (67.33% vs. 57.56%), coherence (95.11% vs. 57.33%), and creativity (85.00% vs. 84.33%). d) CRITICS offers AI practitioners a method for refining LLM-generated stories for improved creativity and engagement while maintaining coherence, potentially leading to the development of more sophisticated and engaging narrative generation systems. The paper notes CRITICS’ effectiveness depends on the underlying LLM capabilities and current implementation is optimized for English. Follow-up questions: 1. Could CRITICS be adapted for non-English languages, and what modifications would be required to prompts and criteria for effective cross-lingual transfer? 2. How does the computational cost of the iterative critique process in CRITICS scale with story length and the number of critic LLMs used, and what optimization strategies could be explored to improve efficiency? 3. Can the criteria used by the critics be dynamically adjusted during the refinement process based on user feedback or other real-time signals to personalize the level and style of story creativity? |
| Diversity-Rewarded CFG Distillation (Read more on arXiv or HuggingFace) | alexrame, Sper42, bachem, ferretj, aagostinelli86 | This research aims to improve the quality-diversity trade-off in generative models, specifically for text-to-music generation. The authors introduce a novel finetuning strategy called diversity-rewarded CFG distillation, combining Classifier-Free Guidance (CFG) distillation with reinforcement learning using a diversity reward based on embedding similarity. Results on MusicLM show that model merging via linear interpolation of weights from a quality-focused model (β=0) and a diversity-focused model (β=15) creates a Pareto front outperforming individual models and baselines. Human evaluation confirms that the merged model (LERP(0,15)) exhibits higher diversity than CFG-augmented base model while maintaining comparable quality. This implies that AI practitioners can leverage this technique to control the quality-diversity balance at deployment time without CFG’s inference overhead by interpolating pre-trained model weights. Follow-up questions: 1. The paper mentions potential “reward hacking” with the diversity metric; could the authors elaborate on specific instances observed and suggest mitigation strategies beyond those mentioned (e.g., human/AI feedback embedding)? 2. How does the computational cost of training the embedding model (E) compare to the cost of finetuning the generative model, and how does the embedding model’s architecture and training impact the overall performance and efficiency of the proposed method? 3. Could the authors provide more details on the variance reduction baseline used in their RL implementation, and its effect on the stability and convergence of the diversity optimization? |
| Jointly Generating Multi-view Consistent PBR Textures using Collaborative Control (Read more on arXiv or HuggingFace) | Dante De Nigris, SlavaElizarov, CiaraRowles, bostadynamics, esx2ve | a) The research aims to generate multi-view consistent Physically Based Rendering (PBR) textures from a text prompt and mesh, addressing the challenge of view inconsistency in existing text-to-texture methods. b) The proposed method extends the Collaborative Control paradigm to a multi-view context, leveraging a pre-trained RGB diffusion model and jointly diffusing multi-view PBR images in view space conditioned on a reference view, its DINOv2 features, and per-pixel correspondences between views. A simple fusion technique then merges the diffused images into a final texture map. c) Ablation studies demonstrate the importance of pixel-wise correspondence attention and occlusion awareness for multi-view consistency, with the removal of correspondence attention noticeably worsening fusion fitting loss. No specific quantitative improvement compared to baseline methods is provided for overall texture quality or realism. d) AI practitioners working with 3D models can leverage this method to generate PBR texture maps directly from text prompts and meshes, potentially bypassing traditional, more laborious texturing workflows. However, the paper does not offer comparisons against other multi-view text-to-texture methods in terms of realism or efficiency. Follow-up questions: 1. How does the computational cost of this multi-view Collaborative Control approach compare to alternative multi-view texture generation methods, such as those using SDS or iterative inpainting? 2. What is the quantitative impact of the multi-view approach on metrics such as texture resolution, realism, and consistency compared to the original single-view Collaborative Control method or other state-of-the-art methods? How do these metrics relate to visual quality as perceived by humans? 3. The paper mentions challenges with unobserved areas during fusion. What specific strategies for addressing these unobserved areas are being considered for future work, and how might these impact performance and texture quality? |
| TinyEmo: Scaling down Emotional Reasoning via Metric Projection (Read more on arXiv or HuggingFace) | ggcristian | a) The research aimed to develop smaller, more efficient multimodal large language models (MM-LLMs) for improved emotional reasoning and classification in visual sentiment analysis. b) A novel architecture was introduced, featuring a metric-learned cross-modal projector to handle emotion classification separately from the LLM, which focused solely on reasoning, trained using a new synthetic Emotional Visual Instruct dataset. c) TinyEmo-700M (with only 700M parameters) achieved 57.62% zero-shot accuracy on a combination of emotion datasets, outperforming a larger state-of-the-art model (EmoVIT with 7.91B parameters) which achieved 55.57% in the same task. d) AI practitioners can leverage the TinyEmo architecture and training strategy to develop smaller, more efficient, and better-performing MM-LLMs for emotion-related tasks, reducing computational overhead and improving performance by decoupling classification from reasoning. The impactful finding is that data quality and diversity appear more crucial than model size for emotion classification in MM-LLMs. Follow-up Questions: 1. How does the performance of TinyEmo’s conditional reasoning approach compare to other conditional text generation methods on emotion reasoning tasks using established NLP evaluation metrics beyond CLIPScore and Ref-CLIPScore? 2. What are the specific implementation details of the semi-automated bias detection framework, and how can it be adapted for other potential biases beyond the watermark example? 3. What are the limitations of using synthetic data for emotional reasoning, and how can these limitations be addressed in future research, especially with regards to evaluating the quality of generated emotional text? |
| F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matching (Read more on arXiv or HuggingFace) | Zhikang Niu, kaiyu-hf, ChunHuiWangFN, D-Keqi, SWivid | a) This research aimed to develop a robust, non-autoregressive text-to-speech (TTS) model with faster training and inference than current diffusion-based models, while maintaining high quality and zero-shot capabilities. b) F5-TTS leverages Flow Matching with a Diffusion Transformer (DiT) architecture, using ConvNeXt for text preprocessing and a novel Sway Sampling strategy for flow steps during inference. The model is trained on a text-guided speech infilling task using the Emilia dataset. c) F5-TTS achieved a Word Error Rate (WER) of 2.42 on the LibriSpeech-PC test-clean dataset with 32 NFE and Sway Sampling, and a real-time factor (RTF) of 0.15 with 16 NFE and Sway Sampling. d) AI practitioners can utilize F5-TTS as a faster, more robust alternative to existing non-autoregressive TTS models, particularly for zero-shot and multilingual applications. The Sway Sampling strategy can be readily integrated into other Flow Matching based models. Follow-up questions: 1. How does the performance of Sway Sampling with different coefficient s values compare across various datasets beyond those mentioned in the paper (e.g., datasets with different language families or acoustic characteristics)? 2. What are the specific implementation details and computational cost of integrating the Sway Sampling strategy into other Flow Matching based TTS models? Does this integration require retraining the existing models? 3. While the paper mentions robustness improvements over E2 TTS, what specific metrics or analyses were used to quantify these robustness gains, especially regarding alignment failures? More detailed comparison and analysis would be helpful. |
| MentalArena: Self-play Training of Language Models for Diagnosis and Treatment of Mental Health Disorders (Read more on arXiv or HuggingFace) | Chi Han, Qingyun Wang, May Fung, jindongwang, Cheng228 | a) The research aimed to develop a framework for training language models to improve performance on tasks related to the diagnosis and treatment of mental health disorders. b) The study employed a self-play training methodology called MentalArena, involving a language model acting as both patient and therapist, coupled with modules for symptom encoding and decoding to generate training data and mitigate intent bias. c) The fine-tuned model based on GPT-3.5-turbo achieved an average 20.74% improvement over the baseline GPT-3.5-turbo across six benchmark datasets related to biomedical question answering and mental health detection. d) AI practitioners can utilize the MentalArena framework and the generated dataset to develop more effective language models for healthcare applications, specifically for mental health diagnosis and treatment. The significant performance improvement achieved through self-play highlights its potential for enhancing LLM capabilities in specialized domains. Follow-up questions: 1. How does the Symptom Decoder module specifically address and quantify the reduction in intent bias during the self-play interactions? 2. Could the MentalArena framework be adapted for other medical specialties beyond mental health, and what modifications might be necessary? 3. What are the computational resource requirements for training with the MentalArena framework, particularly for larger language models like Llama-3? |
| TextToon: Real-Time Text Toonify Head Avatar from Single Video (Read more on arXiv or HuggingFace) | Chenliang Xu, Lele Chen, Luchuan Song, pliu23, goddice | a) The research aims to develop a real-time system for generating and animating toonified head avatars from single monocular videos using text-based style descriptions. b) The proposed method, TextToon, utilizes a conditional Tri-plane Gaussian Deformation Field to learn stylized facial representations and a patch-aware contrastive learning approach for fine-tuning style adaptation. It integrates 3DMM tracking for head pose and expression estimation and employs a “lazy factor” to handle non-rigid shoulder movements. c) TextToon achieves real-time performance, operating at 48 FPS on a GPU and 15-18 FPS on a mobile device (without 3DMM tracking), and allows for rapid style adaptation in minutes. In a user study, TextToon achieved an average score of 4.1 out of 5 for Video Quality. d) AI practitioners can leverage this approach for real-time avatar creation and animation in applications like video conferencing, gaming, and virtual reality, benefiting from its user-friendly text-driven stylization and efficient performance. The speed of style fine-tuning enables quick adaptation to diverse artistic styles. Follow-up questions: 1. What are the limitations of the Text2Image module used in TextToon regarding complex editing instructions and handling of occlusions or extreme expressions not present in the training data? 2. How does the proposed method address the potential for “identity drift” often observed in stylization methods based on StyleGAN inversion, and are there any quantitative evaluations measuring identity preservation throughout the stylization process? 3. Can the conditional Tri-plane Gaussian Deformation Field be extended to incorporate other modalities, like audio, for controlling the avatar’s expressions and lip movements in real-time? |
| Holistic Unlearning Benchmark: A Multi-Faceted Evaluation for Text-to-Image Diffusion Model Unlearning (Read more on arXiv or HuggingFace) | Dongwoo Kim, Sangdon Park, Minjong, hi-sammy | a) This research aims to comprehensively evaluate the effectiveness and side effects of text-to-image diffusion model unlearning methods. b) The authors develop a benchmark called HUB, evaluating six unlearning methods (ESD, UCE, AC, SA, SalUn, Receler) across five aspects: effectiveness on target concepts, image faithfulness, prompt compliance, robustness to side effects, and consistency in downstream tasks. c) No single method performed optimally across all evaluation aspects; for example, while Receler and SalUn showed robustness in removing the target concept under diverse prompts, they also exhibited a decrease in generated image quality. SalUn generated images with the lowest FID score of 21.4 compared to the original model’s score of 20.8. d) AI practitioners should consider the trade-offs between effectiveness, image quality, and potential side effects (e.g. over-erasing) when selecting an unlearning method for a specific application. The benchmark provides a tool for making informed decisions about which unlearning method is most suitable, based on specific project requirements. e) The paper briefly states the reasoning behind the choice of the four concepts as “covering diverse and exhaustive scenarios”, however more explanation as to why these particular scenarios are “exhaustive” would be helpful. Follow-up questions: 1. Given the over-erasing effect observed with some methods, what strategies can be explored to mitigate the unintended removal of related concepts while still effectively suppressing the target concept? 2. How does the computational cost of each unlearning method compare, and how might this influence method selection in resource-constrained settings? 3. The paper analyzes the over-erasing effect using prompts of closely-related concepts, but doesn’t explore how it influences the generation of loosely-related or even unrelated concepts which may potentially share some latent feature with the target concept. How does over-erasing affect the overall generative ability of the unlearned models? |
| Hallucinating AI Hijacking Attack: Large Language Models and Malicious Code Recommenders (Read more on arXiv or HuggingFace) | fgmckee, dnoever | a) The research investigates the risk of large language models (LLMs) recommending malicious code within software supply chains, particularly due to context-shifting within programming scenarios. b) The study empirically tested several prominent foundational LLMs by providing prompts related to code generation, then examining the responses for recommendations of compromised API endpoints, RSS feeds, GitHub repositories, and npm packages. c) The research demonstrates that LLMs, despite safety guardrails, can be manipulated into suggesting malicious code by framing risky suggestions within seemingly benign programming challenges; one specific finding is that GPT-40, while refusing to design a fake login page directly, generated code mimicking the PayPal website when framed as an HTML programming problem. d) The main implication for AI practitioners is the need to develop stronger context-aware safeguards within LLMs and to critically evaluate AI-generated code recommendations, as the current vulnerability to context-shifting exposes security risks for software supply chains. Follow-up questions: 1. What specific mitigation techniques could be implemented to prevent context-shifting attacks, such as enhanced input sanitization or context-aware filtering of LLM outputs? 2. How can code-review processes be augmented to effectively detect potentially malicious code introduced through LLM hallucinations or compromised recommendations? 3. Could this type of vulnerability be utilized for “red teaming” exercises to proactively identify and address potential security weaknesses in LLMs before they are exploited by malicious actors? |
| Seeker: Enhancing Exception Handling in Code with LLM-based Multi-Agent Approach (Read more on arXiv or HuggingFace) | Minlie Huang, Yuan Yuan, Yuxuan Chen, XUANMINGZHANG | This research explores whether Large Language Models (LLMs) can improve the standardization, interpretability, and generalizability of exception handling in code. The researchers developed Seeker, a multi-agent framework employing five agents (Planner, Detector, Predator, Ranker, and Handler) that integrate external exception documentation (CEE) with Deep Retrieval-Augmented Generation (Deep-RAG). Compared to baseline methods, Seeker achieved a 92% Code Review Score (CRS), indicating that 92% of generated exception handling implementations were deemed “good” by a GPT-40 evaluator. This suggests that incorporating domain-specific knowledge and structured handling strategies into LLMs can significantly enhance the robustness of generated code, particularly in exception handling. Follow-up questions: 1. How does Seeker’s performance vary across different programming languages, given the language-specific nature of exception handling mechanisms? 2. What are the computational resource requirements and scalability limitations of Seeker when applied to very large codebases? 3. Could the multi-agent architecture and Deep-RAG approach be generalized to other code reliability issues beyond exception handling, such as memory leaks or security vulnerabilities? |
| Do great minds think alike? Investigating Human-AI Complementarity in Question Answering with CAIMIRA (Read more on arXiv or HuggingFace) | Jordan Boyd-Graber, Hal Daumé III, zhoutianyi, mgor | This research investigates the differences in question-answering abilities between humans and AI systems. The study uses CAIMIRA, a novel framework based on Item Response Theory (IRT), to analyze over 300,000 responses from ~70 AI systems and 155 humans on QuizBowl questions. Results show that humans outperform AI on knowledge-grounded abductive and conceptual reasoning, while LLMs like GPT-4-TURBO and LLAMA-3-70B excel at targeted information retrieval and fact-based reasoning. On questions requiring abductive recall (defined in the paper), human performance significantly exceeded GPT-4-TURBO’s, highlighting humans’ superior ability to connect abstract clues to specific entities. AI practitioners should focus on developing QA systems that address the current weaknesses of LLMs in higher-order reasoning and nuanced linguistic interpretation, particularly in tasks with less direct information mapping. Follow-up questions: 1. How does CAIMIRA handle the potential bias introduced by using QuizBowl data, which might favor certain knowledge domains or reasoning skills? 2. Could the study’s findings be replicated with other question-answering datasets beyond QuizBowl, and if so, would we expect similar patterns of human-AI complementarity? 3. What specific architectural or training modifications to LLMs could be investigated to improve performance on questions requiring abductive recall, based on the insights gained from human responses? |
| MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering (Read more on arXiv or HuggingFace) | lilianweng, tejalp, thesofakillers, evanmays, nch0w | a) This research aims to evaluate the ability of AI agents to perform real-world machine learning engineering (MLE) tasks. b) Researchers created MLE-bench, a benchmark of 75 diverse Kaggle competitions, and evaluated several frontier language models using open-source agent scaffolds, comparing agent performance against human leaderboards. c) The best-performing setup, OpenAI’s ol-preview model with AIDE scaffolding, achieved at least the level of a Kaggle bronze medal in 16.9% of competitions (pass@1), increasing to 34.1% with 8 attempts (pass@8). d) AI practitioners should note that while current leading language models can achieve meaningful scores on MLE tasks with appropriate scaffolding, they still struggle with aspects like debugging and recovering from errors, particularly in more complex competitions. The significant improvement observed with increased attempts (pass@k) suggests further research on agent iteration and refinement strategies could be beneficial. e) The paper does not clarify whether all 75 competitions used are medal-granting on Kaggle or whether some were adapted by the researchers. Follow-up questions: 1. What specific modifications were made to the AIDE, MLAB, and OpenHands scaffolds to improve their performance on MLE-bench, and what was the rationale behind these modifications? 2. How do the types and complexities of the MLE tasks included in the benchmark compare to typical real-world ML engineering work beyond Kaggle competitions? 3. What are the computational costs (e.g., GPU hours, tokens) associated with running the benchmark, and what are the practical implications of this for researchers with limited resources? |
| Does Spatial Cognition Emerge in Frontier Models? (Read more on arXiv or HuggingFace) | vkoltun, philkra, erikwijmans, sramakrishnan | a) The research investigates whether spatial cognition emerges in contemporary frontier models, including large language models (LLMs) and large multimodal models (VLMs). b) A new benchmark called SPACE was created, evaluating large-scale mapping, small-scale object reasoning, and cognitive infrastructure like spatial attention and memory, using text and image-based tasks derived from cognitive science literature. c) Frontier models performed near chance level on key large-scale tasks, like those involving egocentric views; however, on the small-scale selective attention task, some models like GPT-40 achieved over 95% accuracy. d) AI practitioners should consider the limitations of current frontier models in spatial cognition, particularly when applied to embodied AI or tasks requiring robust spatial understanding. The discrepancy between high performance on some small-scale tasks and near-chance performance on large-scale, embodied tasks suggests uneven development of spatial reasoning abilities. e) The paper does not provide detailed implementation specifics for the text array encoding for textual presentations of small-scale tasks, other than to mention they encode spatial information with 2D character arrays. Follow-up questions: 1. What specific architectural changes could be explored to improve frontier model performance on large-scale, egocentric spatial tasks, given the current limitations? 2. How does the performance of models on SPACE correlate with performance on other established reasoning benchmarks, and what does this reveal about the relationship between spatial cognition and other cognitive abilities in these models? 3. Can the textual encodings of spatial information used in SPACE be open-sourced to facilitate further research and development of improved spatial reasoning capabilities in LLMs? |
Papers for 2024-10-09
| Title | Authors | Summary |
|---|---|---|
| LongGenBench: Long-context Generation Benchmark (Read more on arXiv or HuggingFace) | Peijie Dong, wenxinsiju, xuminghui, Dominic789654 | This research addresses the lack of benchmarks for evaluating long-context generation capabilities of LLMs, focusing on consistency in logical flow. The authors introduce a synthetic benchmark, LongGenBench, which redesigns input formats from existing benchmarks (MMLU, GSM8K, CSQA) to necessitate cohesive, multi-answer responses, thus evaluating generation in addition to retrieval skills. Results show that both API-accessed and open-source models exhibit performance degradation in these long-context generation scenarios, ranging from 1.2% to 47.1%. The Gemini-1.5-Flash model showed the least degradation (1.2% on GSM8K) among API-accessed models. This research implies that AI practitioners should consider model limitations in long-context generation and prioritize models exhibiting greater resilience in such tasks. Here are some follow-up questions an AI practitioner might ask: 1. How does the performance degradation observed in LongGenBench correlate with different long-context techniques, such as efficient attention mechanisms or state-space models? 2. What are the specific architectural differences between Gemini-1.5-Flash and other API-accessed models that contribute to its superior performance in long-context generation as measured by LongGenBench? 3. Could fine-tuning strategies specifically targeting long-context generation consistency mitigate the performance degradation observed across different model architectures? |
| $\textbf{Only-IF}$:Revealing the Decisive Effect of Instruction Diversity on Generalization (Read more on arXiv or HuggingFace) | Francois Charton, Justin Wang, shizhuo2 | a) This research investigated the impact of instruction diversity on the generalization ability of large language models (LLMs) for instruction following. b) Controlled experiments using symbolic string rewriting tasks inspired by the Turing-complete Markov algorithm, along with real-world code generation and general reasoning tasks, were conducted. c) Models trained on fewer than 300 unique string rewriting instructions consistently failed to generalize, while models trained on over 1000 distinct instructions generalized effectively. In code generation, a model fine-tuned with 20,000 diverse instructions (OSS-Instruct, Alpaca, CoT) outperformed models trained on 75,000 code-specific instructions on the DeepSeek-Coder-6.7B-Base model. d) AI practitioners should prioritize diversifying instruction data across different semantic domains rather than simply increasing the volume of data from a specific domain when fine-tuning LLMs for improved generalization. The impactful finding that a smaller, diverse dataset can outperform a larger, domain-specific dataset highlights the critical role of strategic data diversification in LLM development. Follow-up questions: 1. How does the proposed methodology for evaluating instruction following, using symbolic string rewriting, translate to more complex real-world tasks beyond code generation, such as those involving multi-modal inputs or outputs? 2. While the research demonstrates the benefits of cross-domain diversification, it also mentions a trade-off between generalization and specialization. What specific metrics or methods can be used to determine the optimal balance between diverse and specialized instructions in a dataset for a given task and LLM architecture? 3. Could the findings related to the number of unique instructions required for generalization (e.g., >1000 for the string rewriting task) be further analyzed to determine how this threshold scales with the complexity of the target tasks and the size of the LLM? |
| RevisEval: Improving LLM-as-a-Judge via Response-Adapted References (Read more on arXiv or HuggingFace) | lifengshang, YuxinJiang, Tiezheng, yufeiwang201217a, DonJoey | a) This research explores whether generating response-adapted references using LLMs can improve the reliability of LLM-based evaluation of text generation, especially in open-ended tasks. b) REVISEVAL, the proposed method, revises the model-generated response using the task instruction and evaluation rubric to create a response-adapted reference, which then guides subsequent evaluation by LLM-as-a-Judge or classic text metrics. c) REVISEVAL improved the accuracy of Llama 3.1-8B as a judge on the LLMBar benchmark by approximately 6% compared to reference-free evaluation, highlighting its ability to mitigate biases like verbosity. d) AI practitioners can use REVISEVAL to improve the accuracy and reduce bias in automated evaluation of open-ended text generation tasks, potentially reducing the need for expensive and time-consuming human evaluation. The paper suggests that leveraging the generative capabilities of LLMs for revision, rather than just discrimination, can lead to more effective automated evaluation, especially with weaker LLMs. Follow-up questions: 1. How does the performance of REVISEVAL with different reviser LLMs (other than GPT-4 and Llama 3.1-8B) compare across various NLG and instruction-following tasks? 2. What are the computational costs of using REVISEVAL compared to other evaluation methods, and how can these costs be optimized for practical applications? 3. Could the revision process in REVISEVAL be further improved by incorporating techniques like reinforcement learning from human feedback (RLHF) to directly optimize the quality of the generated references? |
| A Spark of Vision-Language Intelligence: 2-Dimensional Autoregressive Transformer for Efficient Finegrained Image Generation (Read more on arXiv or HuggingFace) | Sinan Tan, Jinze, JustinLin610, ZefanCai, leonardPKU | a) The research aims to address the information loss and computational limitations of vector-quantization (VQ) in autoregressive (AR) image generation. b) A novel architecture, the 2-Dimensional Autoregression (DnD) Transformer, is introduced, which predicts multiple codes for an image by incorporating a depth dimension in addition to spatial dimensions, thereby increasing the Information Compression Ratio. c) On ImageNet256×256, DnD-Transformer achieves a Fréchet Inception Distance (FID) of 1.54 and an Inception Score (IS) improvement of 82.6 over the baseline LlamaGen XXL model with the same parameter count (1.4B) and using classifier-free guidance scale (cfg) of 2. d) AI practitioners can use DnD-Transformer to generate higher-quality images, particularly those containing fine-grained detail and rich text, more efficiently than previous AR models relying solely on 1D autoregression. The emergent vision-language capabilities also open possibilities for text-rich image generation in an unconditional setting. Follow-up questions: 1. How does the performance of DnD-Transformer scale with different codebook sizes (N) and downscaling factors (f), and what is the trade-off between image quality and computational cost in these scenarios? 2. What are the specific implementation details for integrating DnD-Transformer with existing LLMs for end-to-end training, and what are the observed benefits and challenges in such a setup? 3. How robust is the “spark” of vision-language intelligence observed in DnD-Transformer, and can this capability be explicitly controlled or directed for specific text-image generation tasks, rather than relying solely on emergent behavior? |
| ControlAR: Controllable Image Generation with Autoregressive Models (Read more on arXiv or HuggingFace) | Haocheng Shen, Peize Sun, Shoufa Chen, Tianheng Cheng, Zongming Li | a) The paper investigates controllable image generation using autoregressive (AR) models, aiming to achieve similar control as diffusion models like ControlNet. b) ControlAR encodes spatial control images (e.g., edges, depth maps) into tokens using a Vision Transformer (ViT) and incorporates these tokens into the AR image generation process via conditional decoding, where the next image token prediction is conditioned on both previous image tokens and the current control token. c) ControlAR achieves an FID of 10.53 on lineart edge control with the MultiGen-20M dataset, outperforming ControlNet++. d) This work offers AI practitioners a more memory-efficient alternative to diffusion models for controllable image generation, allowing for arbitrary resolution outputs with competitive quality and controllability. The introduction of conditional decoding, more efficient than prefilling, is particularly relevant for developing and deploying large AR models for image generation tasks. Follow-up questions: 1. How does the performance of different ViT architectures and pretraining schemes for the control encoder affect the final image generation quality and controllability across diverse datasets and control types? 2. What are the computational and memory trade-offs of using ControlAR with larger AR models like LlamaGen-L compared to smaller models like LlamaGen-B for different resolution outputs, and how does this impact practical deployment scenarios? 3. What strategies can be explored to extend ControlAR to handle multiple simultaneous control inputs, and how can the control fusion mechanism be optimized for more complex multi-control scenarios? |
| MA-RLHF: Reinforcement Learning from Human Feedback with Macro Actions (Read more on arXiv or HuggingFace) | Yu Sun, Shuohuan Wang, Huang Fang, Haoran Sun, Yekun Chai | This paper addresses the inefficiency of token-level Reinforcement Learning from Human Feedback (RLHF) in Large Language Models (LLMs) due to the credit assignment problem. The authors propose MA-RLHF, which incorporates macro actions (sequences of tokens) into the RLHF framework using a modified Proximal Policy Optimization (PPO) algorithm called MA-PPO. Experiments on text summarization using the TL;DR dataset show that MA-RLHF achieves parity with standard RLHF 1.7x to 2x faster and ultimately improves reward model scores by up to 30%. This implies that utilizing MA-RLHF can significantly improve training efficiency and performance of LLMs aligned with human preferences, allowing practitioners to train more effectively and produce higher-quality models. Follow-up questions: 1. How does the choice of macro action termination strategy (n-gram, parsing-based, etc.) affect the performance and training efficiency of MA-RLHF on different downstream tasks? 2. Are there specific types of tasks or datasets where the benefits of MA-RLHF are most pronounced, and are there any where it performs worse than standard RLHF? 3. What are the computational and memory implications of implementing MA-RLHF compared to standard RLHF, especially for large-scale models and datasets? |
| Grounded-VideoLLM: Sharpening Fine-grained Temporal Grounding in Video Large Language Models (Read more on arXiv or HuggingFace) | Yufan Zhou, Shizhe Diao, Yu Cheng, Zhiyang Xu, WHB139426 | a) This research addresses the challenge of fine-grained temporal grounding in Video Large Language Models (Video-LLMs), aiming to improve their ability to perceive and reason over specific video moments. b) The authors introduce Grounded-VideoLLM, featuring a two-stream architecture (spatial and temporal) for encoding video segments and incorporating discrete temporal tokens into the LLM’s vocabulary for timestamp representation. A three-stage training strategy progresses from video-caption alignment to temporal token alignment and finally multi-task instruction tuning, supplemented by a curated grounded VideoQA dataset. c) On the NEXT-GQA dataset, Grounded-VideoLLM achieves an Acc@GQA score of 26.7%, a 2.4% improvement over the previous state-of-the-art. d) AI practitioners can leverage Grounded-VideoLLM to develop more accurate and robust video understanding applications, specifically for tasks requiring fine-grained temporal reasoning such as video question answering and dense video captioning. Follow-up questions: 1. What is the computational cost of the two-stream encoding approach, and how does it scale with video length and resolution? 2. How does the choice of the video encoder (InternVideo2 in this case) impact the overall performance of Grounded-VideoLLM, and are there alternative video encoders that could be more efficient or effective? 3. Could you elaborate on the automatic annotation pipeline used to create the grounded VideoQA dataset, including details about prompt engineering and quality control measures to ensure data reliability? |
| Hyper-multi-step: The Truth Behind Difficult Long-context Tasks (Read more on arXiv or HuggingFace) | yuyijiong | This research investigates why long-context language models (LCLMs) struggle with complex tasks despite large context windows. The study uses synthetic key-value and student resume retrieval datasets to evaluate LCLM performance on multi-matching retrieval (retrieving multiple items simultaneously) and logic-based retrieval (retrieval requiring logical judgment). Results show accuracy decreases significantly for multi-matching retrieval as the number of matches increases, with some models approaching 0% accuracy with 5 or more matches in the Student Resume Retrieval task. The paper proposes that these tasks are “hyper-multi-step,” requiring numerous independent steps exceeding LCLM simultaneous processing capacity. This implies that simply increasing context window size may not improve LCLM performance on such tasks. Follow-up questions: 1. What specific architectural limitations within current LCLMs prevent efficient handling of hyper-multi-step problems? 2. Beyond prompting LCLMs to write and execute programs, what alternative approaches might enable LCLMs to handle hyper-multi-step tasks more effectively? 3. How could the insights on the limitations of vector retrieval for logic-based tasks inform the development of more robust retrieval-augmented generation (RAG) systems? |
| EBES: Easy Benchmarking for Event Sequences (Read more on arXiv or HuggingFace) | Evgeny Burnaev, Viktor Moskvoretskii, Igor Udovichenko, Dmitry Osin, dalime | a) The paper introduces EBES, a benchmark for evaluating machine learning models on event sequences (EvS), aiming to standardize evaluation and facilitate comparison of model performance on this type of data. b) EBES uses a standardized evaluation protocol with Monte Carlo cross-validation and hyperparameter optimization (HPO), incorporating diverse real-world and synthetic datasets and multiple established and novel EvS models. c) Results show that GRU-based models generally perform best, and MLP performance is often within 5% of the top model; on the Age dataset, using mean hidden state aggregation with a GRU achieves an accuracy of 0.629 ± 0.005. d) AI practitioners should consider EBES for rigorous evaluation of EvS models and be aware that model performance can be highly dataset-dependent and sensitive to data characteristics like sequence order and timestamps. Furthermore, the paper notes that results on the PhysioNet2012 dataset were statistically indistinguishable between methods, suggesting limitations for its use in evaluating EvS models. Follow-up questions: 1. The paper identifies the learning rate as a crucial hyperparameter. Could more detail be provided on the HPO search space for the learning rate and other hyperparameters, including ranges and distributions used? 2. The paper suggests limitations with the PhysioNet2012 dataset. What specific characteristics of this dataset are believed to contribute to this limitation, and what alternative datasets might be more suitable for benchmarking EvS models in healthcare applications? 3. How easily can EBES be extended to evaluate models for other event sequence tasks beyond sequence-level classification and regression, such as forecasting or imputation? |
Papers for 2024-10-08
| Title | Authors | Summary |
|---|---|---|
| Differential Transformer (Read more on arXiv or HuggingFace) | Li Dong, thegenerality, sunyt32, yuqxia, ytz20 | This research addresses the problem of Transformers over-attending to irrelevant context in attention mechanisms. The authors propose a Differential Transformer (DIFF Transformer) using a differential attention mechanism that calculates attention scores as the difference between two softmax attention maps. Results on language modeling tasks show DIFF Transformer outperforms standard Transformer models, requiring only 65% of the model size or training tokens to achieve comparable performance. For in-context learning on the TREC dataset, DIFF Transformer improved average accuracy by 5.2% to 21.6% compared to the standard Transformer. This architecture allows AI practitioners to train more efficient and performant large language models. Here are some follow-up questions an AI practitioner might have: 1. What is the computational overhead of the differential attention mechanism compared to standard softmax attention, particularly with different FlashAttention implementations? 2. How does the performance of DIFF Transformer compare to other attention-mechanism modifications designed to address similar issues of focusing on irrelevant context, and what are the tradeoffs? 3. Beyond language modeling, how does the differential attention mechanism perform on other downstream tasks that heavily rely on attention, such as machine translation or image captioning? |
| LLMs Know More Than They Show: On the Intrinsic Representation of LLM Hallucinations (Read more on arXiv or HuggingFace) | Roi Reichart, Zorik Gekhman, belinkov, tokeron, hadasor | This research investigated how large language models (LLMs) encode and represent errors, termed “hallucinations,” within their internal activations. The study employed probing classifiers trained on intermediate LLM representations to predict error presence and type, alongside an analysis of repeated sampling of LLM-generated answers. Probing classifiers trained on the activations of exact answer tokens achieved significantly higher error detection performance (AUC of 0.85 on TriviaQA with Mistral-7b-instruct) compared to methods using other tokens. However, these probing classifiers did not generalize well across datasets representing different tasks, suggesting skill-specific truthfulness encoding. The study highlights a potential disconnect between LLMs’ internal representations and external behavior, where the model may internally encode the correct answer but consistently generate an incorrect one. A clear quantitative finding comparing probe-based answer selection accuracy vs. greedy decoding across different error types is not presented in a consolidated manner, making direct comparison difficult. Follow-up questions from an AI practitioner: 1. Could the “skill-specific” nature of truthfulness encoding be mitigated by multi-task training of the probing classifier, and if so, how would performance compare to single-task training on diverse datasets? 2. Given the observed discrepancy between internal encoding and external behavior, what specific modifications to the decoding process or model architecture could potentially improve the alignment and reduce erroneous outputs? 3. How does the performance of exact answer token probing compare to other state-of-the-art error detection methods across a broader range of LLM architectures and sizes, including larger models not tested in this study? |
| VideoGuide: Improving Video Diffusion Models without Training Through a Teacher’s Guide (Read more on arXiv or HuggingFace) | Jong Chul Ye, geonyoung-park, bryanswkim, DHCAI | a) The research aims to improve the temporal consistency of pre-trained text-to-video (T2V) diffusion models without requiring additional training or fine-tuning. b) VideoGuide interpolates denoised samples from a “guiding” pre-trained VDM (which can be the same as the sampling VDM or a different one) into the denoising process of the main “sampling” VDM during the initial sampling steps. c) When applied to AnimateDiff, VideoGuide achieved the best performance across all evaluated metrics, including a subject consistency score of 0.9614, exceeding the base AnimateDiff score of 0.9183. d) VideoGuide offers AI practitioners a computationally efficient method to enhance the temporal quality of existing T2V diffusion models by leveraging other pre-trained models, potentially combining the strengths of different models without requiring retraining. The paper implies, but does not explicitly state, whether this technique preserves unique features of the sampling VDM, such as controllability. Follow-up Questions: 1. How does the choice of the guiding VDM affect the specific aspects of the generated video, such as style, motion, and text coherence, and what strategies can be used for selecting the most effective guiding model for a given task? 2. The paper focuses on 16-frame videos. How does VideoGuide scale with longer video generation and what modifications, if any, are required to maintain performance and computational efficiency? |
| FAN: Fourier Analysis Networks (Read more on arXiv or HuggingFace) | Yongding Tao, Ge Li, Jingjingxu, zkcpku, dongyh | This research investigates how to enable neural networks to effectively model periodicity. The authors propose Fourier Analysis Networks (FAN), which integrate Fourier Series into the network architecture to explicitly encode periodic patterns. On symbolic formula representation tasks, FAN consistently outperforms baselines like MLP, KAN, and Transformer as the number of parameters increases. For example, on the task of representing f(x) = J₀(20x), FAN achieves significantly lower test RMSE than other baselines across various parameter sizes. This suggests that AI practitioners can leverage FAN to improve model performance, particularly in domains involving periodic or quasi-periodic data, such as time series analysis and symbolic computation, by replacing standard MLP layers with FAN layers. It is unclear how the comparative parameter and FLOP counts in Table 1 are calculated. Follow-up questions: 1. How does the performance of FAN scale with the complexity of the periodic functions being modeled, and what are the practical limitations in terms of computational cost? 2. Are there specific types of periodic or quasi-periodic data where FAN offers the most significant advantages over other architectures, and are there any scenarios where it might be less suitable? 3. How robust is FAN to noise in periodic data, and what techniques could be used to further enhance its robustness? |
| Presto! Distilling Steps and Layers for Accelerating Music Generation (Read more on arXiv or HuggingFace) | Jonah Casebeer, Ge Zhu, Njb, tberg12, ZacharyNovack | a) The research aims to accelerate inference in diffusion-based text-to-music (TTM) models by reducing sampling steps and computational cost per step. b) The authors develop Presto, a dual-faceted distillation approach comprising: Presto-S (step distillation using GAN-based distribution matching), Presto-L (layer distillation with variance preservation and budget awareness), and Presto-LS (combined layer-step distillation). c) Presto-LS achieves a 10-18x speedup compared to the base model, resulting in a latency of 230/435ms for generating 32-second mono/stereo audio at 44.1kHz on an A100 40GB GPU, while also improving diversity (higher recall) compared to Presto-S. d) AI practitioners working on real-time or interactive music generation applications can leverage Presto-LS to significantly reduce inference latency without substantial quality loss, potentially enabling new interactive experiences. The paper focuses exclusively on offline generation, and its applicability to real-time or streaming generation remains unclear. Follow-up questions: 1. How does Presto-LS perform on longer music pieces (e.g., > 1 minute), and how does the latency scale with duration? 2. Could the variance preservation technique used in Presto-L be generalized to other diffusion-based generative models beyond music, such as text-to-image or text-to-video? 3. What are the memory and compute requirements for training and deploying the different Presto models (S, L, LS)? |
| Named Clinical Entity Recognition Benchmark (Read more on arXiv or HuggingFace) | Clément Christophe, Tathagata Raha, Muhammad Umar Salman, Marco AF Pimentel, Wadood M Abdul | a) The research aims to establish a standardized benchmark for evaluating Named Clinical Entity Recognition (NER) models in the clinical domain. b) The benchmark employs a curated collection of publicly available clinical datasets with entities standardized using the OMOP Common Data Model, along with token-based and span-based evaluation metrics (precision, recall, and F1-score) in different averaging modes (Micro and Macro). Both exact and partial matching strategies are also incorporated. c) GLiNER-based architectures achieve higher F1-scores (78.25% for condition entities using span-based macro-averaged scores) compared to decoder-only (LLM) models on the clinical NER task. d) AI practitioners developing clinical NER systems should consider using GLiNER-based models for superior performance compared to decoder-only architectures, particularly for token-level classification tasks where accurate extraction of span information is critical. Follow-up questions: 1. Given the performance advantage of GLiNER models over traditional LLMs, what specific adaptations or fine-tuning strategies were used for the GLiNER models included in this benchmark to optimize their performance on the clinical NER task? 2. The paper mentions the issue of label imbalance in clinical datasets. How does this label imbalance affect the evaluation metrics reported, and were any techniques used to mitigate the impact of this imbalance on model training or evaluation? |
| OmniBooth: Learning Latent Control for Image Synthesis with Multi-modal Instruction (Read more on arXiv or HuggingFace) | Xu Yan, Weichao Qiu, bingbl, Evenc, lilelife | a) The research aims to achieve spatial control with instance-level customization in image generation using multi-modal instructions (text and image references) associated with user-defined masks. b) OmniBooth introduces a “latent control signal” (lc), a high-dimensional spatial feature integrating spatial, textual, and image conditions. Text embeddings are “painted” into lc, while image embeddings undergo “spatial warping” before integration. A modified ControlNet framework aligns lc with latent image features. c) On the MS COCO val2017 dataset, OmniBooth achieved a FID score of 17.8, outperforming InstanceDiffusion (FID 23.9) and ControlNet (FID 20.3). The paper doesn’t clarify how the “synthetic COCO val-set” used for evaluation was generated. d) AI practitioners can leverage OmniBooth to develop image generation models offering users fine-grained control over instance placement and attributes via multi-modal instructions, surpassing the limitations of global prompts or single-modality control. The improved FID score suggests potential for higher quality and more controllable image synthesis. Follow-up questions: 1. Could you elaborate on the creation of the “synthetic COCO val-set” used for evaluation? Specifically, how were instance masks and captions generated, and how does this synthetic set relate to the original COCO val2017 set? 2. What are the computational costs (e.g., training time, inference speed) associated with OmniBooth compared to baseline models like ControlNet and InstanceDiffusion? 3. How does the proposed “spatial warping” method handle instances whose reference images significantly differ in aspect ratio or pose from the target mask region? Does this lead to distortions or artifacts in the generated images? |
| TLDR: Token-Level Detective Reward Model for Large Vision Language Models (Read more on arXiv or HuggingFace) | Rui Wang, Tong Xiao, tbpangolin, pzzhang, deqing | a) The research aimed to develop a token-level reward model (TLDR) for multimodal large language models (VLMs) to improve interpretability and granularity compared to traditional binary reward models. b) TLDR uses a perturbation-based method to generate synthetic hard negatives and token-level labels to train the model, leveraging a pretrained VLM (PaliGemma-3B-Mix-448) and a linear reward model head applied to each token. c) TLDR achieves 98.6% token-level accuracy and can speed up human annotation by 3 times when correcting synthetic captions. A correlation of 0.892 (p=0.006) was found between the log of the hallucination rate and MMMU score. d) TLDR provides AI practitioners with a tool for enhanced self-correction in VLMs, more effective hallucination detection, and faster data annotation for vision-language tasks. Follow-up questions: 1. How does the performance of TLDR scale with larger VLMs and datasets, particularly with more complex and nuanced visual scenes? 2. Can TLDR be adapted for other multimodal tasks beyond image captioning and VQA, such as visual question generation or image retrieval? 3. What are the computational resource requirements for training and deploying TLDR, and how might these impact practical application in resource-constrained settings? |
| UniMuMo: Unified Text, Music and Motion Generation (Read more on arXiv or HuggingFace) | Yutong Zhang, Kun Su, Han Yang, auspicious3000, Jiaben | a) This research aimed to create a unified model, UniMuMo, capable of generating music, motion, and text in arbitrary combinations conditioned on inputs from any of these modalities. b) The key methodology involved aligning unpaired music and motion data based on rhythmic patterns, encoding music and motion into a joint token space using a shared codebook, and training a transformer decoder with a novel music-motion parallel generation scheme. A T5 decoder is then fine-tuned for captioning. c) UniMuMo achieved competitive results on unidirectional generation benchmarks, for example, achieving a CLAP similarity score of 0.29 on text-to-music generation when trained on data containing vocals. The paper does not provide clear comparisons on combined generation tasks (e.g., text and music to motion). d) This work provides AI practitioners with a unified framework for multimodal content generation involving music, motion, and text, potentially streamlining development and deployment compared to using separate models for each task. The impact on real-world combined generation tasks is unclear due to the lack of reported results on such scenarios. Follow-up questions: 1. What are the quantitative results of UniMuMo on multi-conditional generation tasks like text-and-music-to-motion or music-and-text-to-motion, as shown in Figure 1, since these seem to be the major contribution differentiating it from other methods? 2. Could the authors provide further insights into the limitations of the rhythmic pattern alignment technique and its potential impact on generating motions for music with complex and varying rhythms? 3. Can the proposed framework be extended to other modalities beyond music, motion, and text, such as image or video? |
| LLaMA-Berry: Pairwise Optimization for O1-like Olympiad-Level Mathematical Reasoning (Read more on arXiv or HuggingFace) | Tong Che, Jingdi Lei, schrodingers-tiger, jwu323, qq8933 | This research aims to improve large language model (LLM) performance on complex mathematical reasoning, particularly at the Olympiad level. The LLaMA-Berry framework utilizes Self-Refine applied to Monte Carlo Tree Search (SR-MCTS) for solution path optimization and a Pairwise Preference Reward Model (PPRM) with Enhanced Borda Count (EBC) for solution evaluation. On the AIME2024 benchmark, the success rate increased from 2/30 (baseline LLaMA-3.1-8B-Instruct) to 8/30 using LLaMA-Berry. This suggests that LLaMA-Berry can enhance LLM reasoning ability on difficult benchmarks without additional training, potentially reducing the need for extensive labeled data in complex mathematical problem-solving. Follow-up questions: 1. How does the computational cost of SR-MCTS and PPRM with EBC scale with increasing model size and problem complexity, and what are the practical implications for deployment? 2. What is the performance of LLaMA-Berry with different LLMs other than the ones mentioned in the ablation study, especially with larger parameter models and close-source ones? 3. Could the pairwise comparison approach of PPRM be adapted to other domains beyond mathematical reasoning, such as code generation or theorem proving, and what modifications would be required? |
| MathHay: An Automated Benchmark for Long-Context Mathematical Reasoning in LLMs (Read more on arXiv or HuggingFace) | cxiong, lunshi, hendrydong, yuhuixu, demolei | This research aims to evaluate the long-context mathematical reasoning abilities of LLMs. The authors developed MATHHAY, an automated benchmark containing 673 mathematical reasoning questions across various topics and difficulty levels, paired with relevant and irrelevant documents forming “haystacks” of 32K-128K tokens. Evaluation involved both exact match and LLM (GPT-40) judging. Gemini-1.5-Pro-002 achieved the highest overall performance, reaching only 51.26% accuracy at 128K tokens. This result highlights the significant need for improvement in LLMs’ long-context mathematical reasoning capabilities, which is crucial for real-world applications involving complex numerical analysis. Follow-up questions: 1. How does the performance of the LLM judge (GPT-40) compare across different question difficulty levels (single-step vs. multi-step) and document placements (First, Middle, Last)? 2. What specific error analysis was performed to understand the types of mistakes LLMs made on MATHHAY, beyond overall accuracy? 3. What are the specific criteria used by the GPT-40 LLM judge to determine the correctness of an answer when an exact match is not found? |
| TurtleBench: Evaluating Top Language Models via Real-World Yes/No Puzzles (Read more on arXiv or HuggingFace) | siminniu, fan2goa1, WinfredShi, Ki-Seki, Duguce | This research aimed to evaluate the reasoning abilities of Large Language Models (LLMs) in dynamic contexts. The researchers created TurtleBench, a dataset of 1,532 yes/no questions derived from user interactions with an online “Turtle Soup Puzzle” game, and evaluated nine LLMs using 0-shot and 2-shot prompting. Claude-3.5-Sonnet and GPT-40 achieved the highest overall accuracy, exceeding 87%, in the zero-shot setting. OpenAI’s o1 series models performed significantly worse than expected. The paper suggests that relying solely on latent Chain-of-Thought, as observed in the o1 models, may not be sufficient for complex reasoning tasks and that excessive CoT length can introduce noise. Follow-up questions: 1. Given the observed performance disparity between OpenAI’s o1 models and other leading LLMs like Claude-3.5-Sonnet and GPT-40 on TurtleBench, what specific architectural or training differences might contribute to this discrepancy? 2. How does the dynamic nature of the TurtleBench dataset, with its real-time collection of user guesses, prevent data contamination and model cheating compared to static benchmarks, and how can this methodology be applied to other reasoning tasks beyond yes/no puzzles? 3. The paper mentions a cost analysis for different LLMs, but what are the trade-offs in terms of cost and performance when choosing between commercially available LLMs (like Claude and GPT) versus open-source models (like Llama) for reasoning tasks, considering the findings of this research on TurtleBench? |
| MonST3R: A Simple Approach for Estimating Geometry in the Presence of Motion (Read more on arXiv or HuggingFace) | fcole, trevordarrell, hurjunhwa, irwinherrmann, Junyi42 | a) The research aims to directly estimate dynamic scene geometry from monocular video, addressing challenges in traditional multi-stage approaches. b) The approach, Motion DUSt3R (MonST3R), adapts the DUSt3R pointmap representation for dynamic scenes by estimating per-timestep pointmaps and aligning them based on static scene elements. It leverages fine-tuning on a combination of synthetic and real-world datasets with depth and pose annotations and introduces optimizations for video-specific tasks like global point cloud alignment and confident static region identification. c) On the Sintel dataset for video depth estimation, MonST3R achieves an absolute relative error of 0.335 and a percentage of inlier points (δ < 1.25) of 58.5%. It demonstrates competitive performance on camera pose estimation and promising qualitative results for feed-forward 4D reconstruction. The paper doesn’t clearly define metrics used for 4D reconstruction. d) MonST3R offers AI practitioners a faster, potentially more robust alternative to traditional optimization-based methods for estimating geometry from dynamic scenes. This is particularly relevant for applications like robotics, augmented reality, and 3D scene understanding. Follow-up questions: 1. The paper mentions challenges with handling dynamic camera intrinsics in practice despite the theoretical capability. Could the authors elaborate on the specific nature of these challenges and the manual constraints required? 2. What are the specific quantitative metrics used to evaluate the 4D reconstruction results, and how does MonST3R compare against other state-of-the-art methods on these metrics? 3. What are the computational requirements (memory and runtime) for applying MonST3R to longer videos and higher resolutions compared to the reported experiments? |
| Autonomous Character-Scene Interaction Synthesis from Text Instruction (Read more on arXiv or HuggingFace) | thuhsy, YixinChen, awfuact, milleret, jnnan | This research investigates synthesizing multi-stage human-scene interactions (HSIs) directly from text instructions and goal locations. The authors propose a framework using an autoregressive diffusion model to generate motion segments, incorporating scene representations and a scheduler for autonomous stage transitions. Quantitative results demonstrate improved motion synthesis over existing methods, achieving a 0.907 F1 score for interactive motion synthesis. The introduced LINGO dataset (16 hours of motion capture data in various indoor scenes) facilitates training models for complex, language-guided HSI generation. This work provides a unified approach to HSI synthesis, enabling more realistic and autonomous character animation in 3D environments. However, the paper does not fully describe the architecture of the autonomous scheduler, limiting a full understanding of its functionality. Follow-up questions: 1. Can you provide more details on the architecture and training process of the autonomous scheduler? 2. How does the model handle ambiguous or poorly written text instructions? What error handling mechanisms are in place? 3. What are the limitations of the LINGO dataset, particularly regarding the diversity and realism of the interactions? |
| Grounding Language in Multi-Perspective Referential Communication (Read more on arXiv or HuggingFace) | alsuhr, mao1207, ZinengTang | This research investigates how differing visual perspectives affect the success of referential communication between embodied agents. The authors created a dataset of human-written referring expressions in a 3D environment and evaluated various vision-language models as speakers and listeners, including GPT-40, LLaVA-1.5, Ferret, and Groma. Fine-grained model Ferret achieved the highest accuracy in comprehending human-written referring expressions at 69.2%, but all models significantly underperformed compared to human-human communication (87.6% success rate). Fine-tuning LLaVA-1.5 with a preference-based learning approach using data from interactions improved its performance to 69.3% communicative success with human listeners, surpassing GPT-40. This implies that learning from interaction data holds significant potential for enhancing referential communication models, even outperforming stronger pre-trained models. Follow-up questions: 1. Could the preference-based learning approach be extended to incorporate multi-turn dialogue where clarification requests are allowed, and how would that impact performance? 2. How do the different referential strategies observed in human vs. model-generated expressions affect listener comprehension, and could explicitly training models on these strategies further improve performance? 3. How robust is the fine-tuned LLaVA-1.5 model to different 3D environments and object types not present in the ScanNet++ dataset used for training and evaluation? |
Papers for 2024-10-07
| Title | Authors | Summary |
|---|---|---|
| Addition is All You Need for Energy-efficient Language Models (Read more on arXiv or HuggingFace) | Wei Sun, luohy | a) The research investigates whether floating-point multiplication in large neural networks, a computationally expensive operation, can be approximated by integer addition for energy efficiency while maintaining accuracy. b) The authors propose a Linear-complexity Multiplication (L-Mul) algorithm that approximates floating-point multiplication with integer addition and evaluate its numerical precision and performance on language, vision, and mathematics tasks using various transformer-based language models (LLMs). The algorithm was compared to different floating-point precisions (bfloat16, float8_e4m3, float8_e5m2) and integrated into attention mechanisms and full model fine-tuning scenarios. c) L-Mul using a 3-bit mantissa outperforms float8_e5m2 multiplication in accuracy across various LLMs. Specifically, on the GSM8k benchmark, using L-Mul in the attention mechanism of Mistral-7b-Instruct-v0.3 increased accuracy to 52.92% compared to 50.19% with float8_e5m2. d) AI practitioners can potentially reduce the energy consumption of LLM inference and training by replacing floating-point multiplications with the L-Mul algorithm, especially within attention mechanisms, without significant performance degradation. Follow-up questions: 1. What is the specific hardware implementation of the L-Mul algorithm, and how does it integrate with existing deep learning frameworks and hardware accelerators? The paper mentions optimal implementation being at the hardware level and limitations with GPU implementation but lacks specific details. 2. How does the performance of L-Mul scale with increasing model size and complexity beyond the models tested in the paper? Further investigation is needed to understand its generalizability. 3. Are there numerical stability implications when using L-Mul for training, particularly regarding vanishing or exploding gradients, which haven’t been discussed in the paper? |
| NL-Eye: Abductive NLI for Images (Read more on arXiv or HuggingFace) | Zorik Gekhman, yonatanbitton, nitay, tokeron, MorVentura | a) The paper investigates the visual abductive reasoning capabilities of Visual Language Models (VLMs), aiming to determine their ability to infer plausible outcomes or causes from visual scenes. b) Researchers created NL-EYE, a benchmark consisting of 350 image triplets designed to evaluate visual abductive reasoning through plausibility prediction and explanation tasks, using both vision-based and text-based reasoning approaches. c) VLMs struggled on NL-EYE, with most failing to exceed random baseline performance in plausibility prediction, while humans achieved 83-85% accuracy. d) This highlights a critical weakness in current VLMs’ ability to perform visual abductive reasoning, necessitating further research into improving their ability to reason over visual data, rather than solely relying on text-based information. Follow-up Questions: 1. Given the VLMs’ success with text-based reasoning but failure with image-based reasoning, what specific architectural changes to the visual encoding components might improve performance on NL-EYE? 2. The paper mentions VLM sensitivity to hypothesis order. What further investigation can be done to isolate whether this is due to limitations in the models’ understanding of spatial relationships within the combined images or an inherent bias in the models’ sequential processing? 3. Could providing pre-training data that emphasizes correlational or causal reasoning relationships between images improve VLMs’ performance on the various reasoning categories in NL-EYE? |
| Selective Attention Improves Transformer (Read more on arXiv or HuggingFace) | Yossi Matias, Matan Kalman, yanivle | a) The paper investigates whether reducing attention to unneeded elements in a transformer’s context can improve performance and efficiency. b) The researchers introduce “Selective Attention,” a parameter-free modification to the standard attention mechanism that allows tokens to mask the attention paid to them by future tokens. Context pruning is also employed, where sufficiently masked tokens are removed from the context buffer. c) Transformers with selective attention and context pruning achieved equivalent validation perplexity on the C4 dataset with up to 47X less memory for their attention module compared to standard transformers, depending on context length and use of an auxiliary loss term. d) AI practitioners can potentially significantly reduce the memory and computational costs of transformer inference, particularly for long sequences, by implementing selective attention and context pruning without sacrificing performance. The paper focuses specifically on decoder-only transformers and primarily evaluates on language modeling, leaving applicability to encoders and other tasks unclear. Follow-up questions: 1. How does Selective Attention compare to other context pruning methods like Dynamic Context Pruning (DCP) in terms of performance trade-offs and implementation complexity on realistic hardware? 2. How robust are the perplexity gains and memory savings of Selective Attention across different datasets and downstream tasks beyond language modeling? 3. Does the choice of head used for the selection function significantly impact the results, and is there a principled way to choose the optimal head? |
| Tutor CoPilot: A Human-AI Approach for Scaling Real-Time Expertise (Read more on arXiv or HuggingFace) | Susanna Loeb, ddemszky, carlycodes, Analu, rose-e-wang | a) The study investigated whether a human-LM system, Tutor CoPilot, could improve tutoring quality and student learning in K-12 mathematics. b) A randomized controlled trial was conducted with 900 tutors and 1,800 K-12 students, comparing a treatment group with access to Tutor CoPilot to a control group without access. NLP classifiers were trained and used to analyze pedagogical strategies employed by tutors. c) Students whose tutors had access to Tutor CoPilot were 4 percentage points more likely to master lesson topics, based on an intent-to-treat analysis. d) For AI practitioners, this study highlights the potential of integrating human expertise with LMs to enhance performance in complex, real-time interaction domains like education. The results suggest focusing on Human-AI collaborative systems that provide real-time, context-specific guidance to augment human expertise rather than replace it. Follow-up questions: 1. What were the specific model architectures and training data used for the Bridge method (mentioned in Figure 1 and throughout) and the NLP classifiers used for identifying pedagogical strategies? More details on the model training and hyperparameter tuning would be helpful for replication or application to other domains. 2. The paper mentions adapting the system to in-person tutoring through speech and visual inputs but doesn’t detail how this would be implemented. What specific technical challenges are anticipated in adapting Tutor CoPilot to process and respond to multimodal input in real-time? 3. The paper mentions limitations regarding the generalizability of the findings beyond the specific tutoring context studied. What steps could be taken to evaluate the robustness and adaptability of the Tutor CoPilot approach across diverse student populations, subject matters, and educational settings? |
| RoCoTex: A Robust Method for Consistent Texture Synthesis with Diffusion Models (Read more on arXiv or HuggingFace) | Jeonga Wi, Junyoung Choi, Jiun, DK9, longshiine | a) The paper aims to develop a robust text-to-texture generation method for 3D meshes that addresses view inconsistencies, seams, and misalignment issues common in existing diffusion-based approaches. b) RoCoTex leverages Stable Diffusion XL with multiple ControlNets (depth, normal, edge) for geometric awareness, a symmetrical view synthesis strategy with regional prompts for view consistency, and novel confidence-based texture blending and soft-inpainting techniques using Differential Diffusion for seam reduction. c) RoCoTex achieved a Kernel Inception Distance (KID) score of 4.03, lower than baseline methods like TEXTure (10.34), Text2Tex (8.15), and Paint3D (6.98), indicating higher quality and diversity of generated textures. d) AI practitioners can utilize RoCoTex for efficient and robust generation of high-quality, consistent textures for 3D models, improving the realism and visual appeal of 3D assets in applications like gaming and virtual/augmented reality. Follow-up questions: 1. How does the performance of RoCoTex scale with increasing mesh complexity and texture resolution, in terms of both quality and computational cost? 2. The paper mentions limitations regarding occlusion and lighting; what specific strategies are planned for future work to address these limitations, and are there any preliminary results or insights available? 3. Could the confidence-based blending and soft-inpainting techniques be adapted and applied to other image synthesis tasks beyond text-to-texture generation? |
| Erasing Conceptual Knowledge from Language Models (Read more on arXiv or HuggingFace) | David Bau, Samuel Marks, sfeucht, RohitGandikota | This research aims to develop a method for erasing specific concepts from large language models (LLMs) while preserving general capabilities and fluency. The proposed method, Erasure of Language Memory (ELM), employs targeted low-rank updates (LoRA) and a multi-objective loss function incorporating erasure, retention, and conditional fluency objectives. On the Weapons of Mass Destruction Proxy (WMDP) biosecurity multiple-choice questions, ELM reduced model accuracy from 64.4% to near-random performance (29.7%). The key implication for AI practitioners is that ELM offers a technique for mitigating risks associated with LLMs generating undesirable content while retaining performance on unrelated tasks. Follow-up questions: 1. How does the computational cost of ELM’s fine-tuning compare to full retraining or other unlearning methods like RMU and RepNoise, particularly for larger models and datasets? 2. Does the paper provide any analysis of the long-term stability of the erasure, for example, does the erased knowledge reappear after further fine-tuning or general use? 3. While the paper states that ELM maintains fluency, are there qualitative examples demonstrating the nature of generated text when prompted with the erased concept, beyond the provided multiple-choice question performance? |
| A Comprehensive Survey of Mamba Architectures for Medical Image Analysis: Classification, Segmentation, Restoration and Beyond (Read more on arXiv or HuggingFace) | gduggal, Man1kandan, Madddy, HARI45SH, shubhii0712 | This paper surveys Mamba architectures and their applications in medical image analysis. The objective is to provide a comprehensive overview of Mamba, a State Space Model (SSM)-based architecture for sequence modeling, covering its evolution, architectures, optimizations, and applications. The survey details various Mamba architectures, including pure Mamba, U-Net variants, and hybrid models, alongside scanning mechanisms and techniques like weakly supervised learning. On 1248x1248 images, Vision Mamba (ViM) uses 73.2% less memory and is 2.8x faster than DeiT. The survey suggests Mamba’s efficiency and linear time complexity makes it a potent alternative to Transformers for medical image analysis tasks, enabling practitioners to handle long-range dependencies and high-complexity data more effectively. Follow-up questions: 1. Given the reported efficiency gains of Mamba over Transformers, what are the practical considerations (e.g., existing library support, ease of implementation, debugging tools) for transitioning existing medical image analysis pipelines from Transformer-based to Mamba-based models? 2. The paper mentions Mamba’s limitations in handling spatial information and non-causal visual data. Are there specific research directions or modifications to Mamba architectures that could mitigate these limitations and broaden its applicability within medical image analysis? 3. The survey highlights several Mamba-based U-Net variants. What are the trade-offs in performance and computational cost among these variants, and how can these trade-offs inform the selection of an appropriate architecture for a specific medical image segmentation task? |
| CANVAS: Commonsense-Aware Navigation System for Intuitive Human-Robot Interaction (Read more on arXiv or HuggingFace) | wpiioos, Unmanned-YuBeen, lastdefiance20, PurpleSand, MilkClouds | This research aimed to develop a robot navigation system capable of interpreting abstract human instructions using commonsense reasoning. The researchers employed imitation learning, training a vision-language model (CANVAS) on a new dataset (COMMAND) containing 48 hours of human-demonstrated navigation in simulated environments. In the challenging “orchard” simulated environment, CANVAS achieved a 67% total success rate, compared to a 0% success rate for the rule-based ROS NavStack. This indicates that training with human demonstrations in simulation can enable robust navigation even with noisy or incomplete instructions. AI practitioners can leverage this approach to develop more user-friendly and adaptable robot navigation systems. Follow-up questions: 1. How does CANVAS handle conflicting information between the sketch trajectory and the language instruction, and what strategies are employed to resolve such conflicts during inference? 2. What specific architectural modifications were made to Idefics2 8B in creating CANVAS-S, beyond simply swapping the vision and text encoders, and what impact did these changes have on performance and efficiency? 3. The paper mentions “randomized starting orientations” for evaluation. What is the distribution of these orientations, and how does robustness to initial orientation affect practical deployment scenarios? |
| MIGA: Mixture-of-Experts with Group Aggregation for Stock Market Prediction (Read more on arXiv or HuggingFace) | Heming Weng, Genesis Wang, yh1567, zjy2001 | a) The research aimed to improve stock market prediction by addressing the limitations of single end-to-end models in capturing the diverse features of different stock styles. b) The authors proposed MIGA (Mixture of Expert with Group Aggregation), a two-stage framework employing an expert router to dynamically allocate stocks to specialized experts and an inner group attention mechanism to facilitate information sharing among experts. c) MIGA-Conv achieved a 24% excess annual return on the CSI300 benchmark, surpassing the previous state-of-the-art model by 8%. It also demonstrated improved performance on ranking metrics like IC and RankIC across CSI300, CSI500, and CSI1000 benchmarks. d) AI practitioners can leverage MIGA to develop more robust and adaptable financial forecasting models by incorporating the Mixture of Experts framework with specialized experts and group aggregation mechanisms. The improved performance on unseen data highlights its potential for real-world applications. Follow-up questions: 1. The paper mentions an ablation study on scaling the number of experts but doesn’t detail the computational cost implications. How does the performance improvement scale with the number of experts, and what are the trade-offs in terms of training time and inference latency? 2. The paper uses a linear layer for the experts. Would more complex expert models (e.g., small transformers) further improve prediction accuracy, and what are the potential drawbacks of such an approach? 3. While the paper focuses on Chinese stock markets, how adaptable is MIGA to other financial markets with different characteristics, and what adjustments might be needed for optimal performance in those markets? |
| NRGBoost: Energy-Based Generative Boosted Trees (Read more on arXiv or HuggingFace) | joaobravo | a) The paper explores generative extensions of tree-based methods for tabular data, focusing on explicit density modeling. b) The authors propose NRGBoost, an energy-based generative boosting algorithm analogous to second-order boosting, trained by maximizing a local second-order approximation to the likelihood. c) NRGBoost achieves comparable discriminative performance to XGBoost on smaller datasets, with an R-squared of 0.547 on the Abalone dataset versus 0.552 for XGBoost, and remains competitive with specialized generative models for sampling. d) AI practitioners working with tabular data can use NRGBoost as a generative model for tasks like single-variable inference and synthetic data generation, potentially offering advantages over existing tree-based and some deep learning alternatives for these applications. Follow-up questions: 1. What are the computational trade-offs between NRGBoost’s improved performance on density estimation and its use of MCMC sampling compared to faster, non-density-based tree models like RFDE? 2. How does the amortization approach for sampling affect the quality of generated samples and training time for varying dataset sizes and complexities? 3. The paper mentions limitations of tree-based models compared to deep learning approaches regarding memory requirements; what strategies could be explored to mitigate this issue for applying NRGBoost to very large datasets? |
Papers for 2024-10-04
| Title | Authors | Summary |
|---|---|---|
| Revisit Large-Scale Image-Caption Data in Pre-training Multimodal Foundation Models (Read more on arXiv or HuggingFace) | Chen Chen, Vasileios Saveris, haotiz, Hong-You, jefflai | a) This research investigates the optimal image-caption data composition for pre-training multimodal foundation models, specifically examining the interplay between synthetic captions and original AltText. b) The authors develop a controllable captioning pipeline to generate diverse caption formats (Short Synthetic Captions (SSC), Descriptive Synthetic Captions (DSC), Dense Synthetic Captions (DSC+), and AltText Fusion Captions (AFC)) and evaluate their impact on CLIP, multimodal LLMs (MM1), and diffusion models. c) Combining SSC and AltText during CLIP pre-training yielded the best performance in retrieval tasks, with over a 10% improvement on COCO retrieval compared to using AltText alone. d) AI practitioners should consider a hybrid approach combining both synthetic captions and AltText when pre-training CLIP, as AltText provides data diversity and synthetic captions enhance image-text alignment. The specific ratio of this combination should be explored depending on the desired trade-off. The paper’s findings on the format of captions show DSC+ is preferred by MLLMs while shorter captions are preferred by CLIP, indicating that caption format should be customized to the specific model. Follow-up questions: 1. What are the computational costs and infrastructure requirements associated with implementing the proposed controllable captioning pipeline, especially for generating captions at the scale of datasets like VeCap-300M? 2. Could the performance gains observed by combining synthetic captions and AltText be replicated using alternative filtering methods besides DFN-2B, and what challenges might arise when combining different filtering or captioning approaches? 3. How does the optimal mixture ratio of synthetic captions and AltText change when scaling up CLIP’s vision encoder, and what are the implications for training larger multimodal foundation models? |
| Video Instruction Tuning With Synthetic Data (Read more on arXiv or HuggingFace) | Wei Li, Chunyuan24, liuziwei7, kimingng, ZhangYuanhan | a) The research aimed to create a high-quality synthetic video instruction-tuning dataset and a corresponding video LMM to improve video understanding beyond simple captioning. b) Researchers developed LLaVA-Video-178K, a synthetic dataset with 178,510 videos and 1.3M instruction samples (captions, open-ended and multiple-choice QA), using GPT-40 and human annotation; they then trained LLaVA-Video, a video LMM, using this dataset and existing visual instruction tuning data, exploring video representation techniques like LLaVA-Video slowFast to maximize frame inclusion. c) LLaVA-Video-7B outperformed LLaVA-OV-7B (a previous top model) in seven out of ten evaluated datasets. On NEXT-QA, adding the LLaVA-Video-178K dataset during training led to a 31.9-point increase in scores. d) This provides AI practitioners with a new high-quality synthetic video instruction tuning dataset and a corresponding LMM, enabling improved development of video understanding models beyond simple captioning. The strong performance increases demonstrate the value of both high-quality, dense annotations and increased frame inclusion within video LMM training. Follow-up Questions: 1. What are the specific details of the LLaVA-Video slowFast implementation, including the algorithms used for slow and fast frame selection and pooling? Appendix B is referenced but not provided, making full evaluation challenging. 2. The paper mentions filtering question-answer pairs generated by GPT-40, but doesn’t provide specifics on the acceptance criteria beyond removing duplicates and unhelpful phrases. What were the precise filtering rules used to ensure quality? 3. What were the specific hyperparameters used for training LLaVA-Video, including learning rate, batch size, and optimization strategy? This information is crucial for replicating and building upon the research. |
| Loong: Generating Minute-level Long Videos with Autoregressive Language Models (Read more on arXiv or HuggingFace) | Tianwei Xiong, XihuiLiu, bykang, Ikuinen, Epiphqny | a) The research aims to generate minute-long, content-rich videos using autoregressive large language models (LLMs). b) Loong, an autoregressive LLM-based model, is trained on a unified sequence of text and video tokens using a progressive short-to-long training strategy with loss re-weighting and inference techniques like video token re-encoding. c) Loong generates minute-long videos and achieves a Fréchet Video Distance (FVD) score of 432 on a custom benchmark of 27-second videos derived from WebVid, using a 7B parameter model. The paper does not provide quantitative comparisons on publicly available long video generation benchmarks. d) AI practitioners can leverage the proposed progressive training and inference strategies to adapt and extend existing LLM-based video generation methods for creating longer, coherent videos, potentially opening new possibilities in content creation and video understanding. Follow-up questions: 1. What is the impact of different video tokenizer architectures on the overall performance of Loong, and how does the compression ratio affect the quality and fidelity of generated long videos? 2. While the paper mentions a super-resolution and refinement module, it lacks specifics. What specific models and techniques were used for post-processing, and what is their contribution to the final video quality (quantitatively)? 3. How does Loong perform on established long video generation benchmarks, enabling a more direct comparison with state-of-the-art methods like StreamingT2V, FreeNoise, and Gen-L? |
| LLaVA-Critic: Learning to Evaluate Multimodal Models (Read more on arXiv or HuggingFace) | Chunyuan24, henghuang, thughost, russwang, txiong23 | a) The research aimed to develop an open-source large multimodal model (LMM) capable of evaluating the performance of other multimodal models across diverse tasks. b) LLaVA-Critic was trained by fine-tuning a pre-trained LLaVA-OneVision model on a 113k sample dataset of critic instruction-following data, incorporating pointwise scoring and pairwise ranking. c) As a judge model, LLaVA-Critic-72B achieved an average Pearson correlation of 0.754 with GPT-40 scores across seven multimodal benchmarks, outperforming the LLaVA-OV-72B baseline (0.634). d) LLaVA-Critic provides a cost-effective, open-source alternative to proprietary models like GPT-4V for evaluating multimodal models, enabling wider access to robust evaluation resources. This is particularly impactful as it reduces reliance on expensive, closed-source APIs for evaluating multimodal models, enabling developers with limited resources to perform rigorous testing and alignment. Follow-Up Questions: 1. Could the authors elaborate on the specific computational resources required for training LLaVA-Critic and its inference latency, to better understand its feasibility for practitioners with varying resource constraints? 2. The paper mentions utilizing LLaVA-Critic for preference learning with DPO. Were other preference learning algorithms like RLHF explored, and if so, how did their performance compare? 3. The paper mentions a v0.5 version of LLaVA-Critic trained on a smaller subset of data. What were the specific limitations or constraints that motivated the creation of this reduced version, and what are the expected performance tradeoffs compared to the full version? |
| Contrastive Localized Language-Image Pre-Training (Read more on arXiv or HuggingFace) | Marcin Eichner, Xinze Wang, haotiz, jefflai, Hong-You | a) This research aims to enhance the localization capability of Contrastive Language-Image Pre-training (CLIP) for fine-grained visual understanding, particularly in multimodal large language models (MLLMs). b) The authors introduce Contrastive Localized Language-Image Pre-training (CLOC), incorporating region-text contrastive loss and a “Prompter” module to extract region embeddings from image embeddings given spatial hints. A visually-enriched and spatially-localized captioning pipeline (VESL) generates pseudo-labeled region-text pairs at scale for training. c) CLOC with 2 billion region labels and a ViT-L/14 architecture achieves 71.1% recall@10 on GRIT region retrieval and improves Ferret MLLM performance on referring description VQA by 6.2% compared to baseline CLIP. d) AI practitioners can utilize CLOC as a drop-in replacement for CLIP in MLLMs to improve performance on referring and grounding tasks that require fine-grained visual understanding. Follow-up questions: 1. The paper mentions working on releasing pre-trained checkpoints and the constructed region-text annotations. Have these resources been released, and if so, where can they be accessed? How does the performance of CLOC compare with other more recent, post-CLIP, image-text models that also incorporate regional information? 2. Could the “Prompter” module be adapted or extended to incorporate other spatial hints beyond bounding boxes and text captions, such as segmentation masks or depth information? What would the implications of such an extension be, and what are the expected challenges? |
| Depth Pro: Sharp Monocular Metric Depth in Less Than a Second (Read more on arXiv or HuggingFace) | Hugo Germain, Aleksei Bochkovskii, srrichter, msantoso98, amael-apple | a) The research aimed to develop a foundation model for zero-shot metric monocular depth estimation that is fast, accurate, and produces high-resolution depth maps with sharp boundaries. b) Depth Pro uses a multi-scale vision transformer architecture, applying plain ViT encoders at multiple scales and fusing the predictions. The training protocol combines real and synthetic datasets with a two-stage curriculum focusing first on robust feature learning and then on boundary sharpening. c) Depth Pro achieves state-of-the-art zero-shot metric depth accuracy with a δ₁ score of 89.0 on the Sun-RGBD dataset and generates a 2.25-megapixel depth map in 0.3 seconds on a V100 GPU. d) AI practitioners can utilize Depth Pro for applications requiring fast and accurate metric depth estimation, particularly in scenarios like novel view synthesis where sharp boundaries are crucial, without needing camera intrinsics or per-domain fine-tuning. The paper’s proposed boundary accuracy metrics based on matting/segmentation data offer a valuable new evaluation tool. Follow-up questions: 1. How does the proposed multi-scale ViT architecture compare in terms of memory consumption to other high-resolution ViT adaptations, especially when dealing with even larger images or videos? 2. The paper mentions limitations with translucent surfaces and volumetric scattering; what specific failure modes are observed in these cases, and are there potential mitigation strategies within the existing architecture or training framework? 3. Could the focal length estimation head be further improved by incorporating self-supervised learning techniques or exploring alternative network architectures specifically designed for focal length prediction? |
| Large Language Models as Markov Chains (Read more on arXiv or HuggingFace) | Abdelhakim Benechehab, Oussama Zekri, ievred, NBoulle, ambroiseodt | a) The paper investigates the theoretical underpinnings of large language model (LLM) inference capabilities, specifically characterizing their behavior and generalization ability. b) The authors establish an equivalence between autoregressive LLMs with a vocabulary size T and context window K and Markov chains defined on a finite state space of size O(TK), analyzing the transition matrix and deriving generalization bounds for both pre-training and in-context learning scenarios. c) For a toy model with vocabulary size T=2 and context window K=3, trained on a binary sequence, the transition matrix has size 14x14, and the model approaches its stationary distribution within approximately 300 steps at temperature 1. d) The analysis provides AI practitioners with a framework to understand the generalization capabilities of LLMs in terms of learning Markov chain transition probabilities. The drawn equivalence to Markov chains offers a theoretical basis for interpreting and predicting the behavior of LLMs, especially in in-context learning scenarios. e) The paper lacks details on the architecture and specific training methodology of the “small GPT-like” toy model used in experiments. It also lacks details on how the prompts are tokenized in the in-context learning experiments. Follow-up Questions: 1. How robust is the equivalence between LLMs and Markov Chains to different tokenization methods, especially for numerical data, given the observed sensitivities highlighted in the paper? 2. Can the Markov Chain framework be leveraged to develop more efficient fine-tuning strategies or prompt engineering techniques for specific downstream tasks involving sequential data? 3. How does the sparsity of the transition matrix, quantified in the paper, influence the computational complexity of estimating the stationary distribution and mixing time of LLMs represented as Markov chains? |
| CLIP-MoE: Towards Building Mixture of Experts for CLIP with Diversified Multiplet Upcycling (Read more on arXiv or HuggingFace) | Yu Cheng, Jihai Zhang, Spico, Xiaoye08 | This research aims to improve Contrastive Language-Image Pre-training (CLIP) performance by addressing its coarse-grained encoding and information loss. The authors propose Diversified Multiplet Upcycling (DMU), fine-tuning multiple CLIP models with shared parameters (except for Feed-Forward Network layers) using Multistage Contrastive Learning (MCL), then integrating these models as experts into a Mixture of Experts (MoE) architecture. On zero-shot image-text retrieval using the ShareGPT4V dataset, CLIP-MoE achieves a top-1 image-to-text retrieval accuracy of 60.5% on Flickr30k, exceeding the OpenAI CLIP baseline by approximately 22%. This offers AI practitioners a model-agnostic method to enhance CLIP performance without extensive retraining from scratch, which is particularly relevant for resource-constrained settings. Follow-up questions: 1. Could the performance gains observed with CLIP-MoE be replicated with different base CLIP architectures (e.g., larger or smaller ViT variants, ResNet-based CLIP)? 2. How does the choice of the number of experts and the top-k routing strategy affect the performance-efficiency trade-off of CLIP-MoE in different downstream tasks and hardware settings? 3. What are the practical considerations for deploying CLIP-MoE in real-world applications, particularly concerning latency and memory footprint compared to standard CLIP models? |
| Eliminating Oversaturation and Artifacts of High Guidance Scales in Diffusion Models (Read more on arXiv or HuggingFace) | Otmar Hilliges, RMW, msadat97 | a) This paper investigates the oversaturation and artifact generation caused by high classifier-free guidance (CFG) scales in diffusion models, aiming to improve generation quality. b) The authors introduce Adaptive Projected Guidance (APG), which decomposes the CFG update into parallel and orthogonal components, down-weighting the parallel component responsible for oversaturation. APG also incorporates rescaling and reverse momentum inspired by gradient ascent optimization. c) APG improved FID scores compared to CFG across multiple models; for example, EDM2-S showed a reduction from 10.42 to 6.49 with a guidance scale of 4. d) APG provides AI practitioners a plug-and-play alternative to CFG that mitigates oversaturation and artifacts at high guidance scales, enabling the use of higher guidance values for enhanced generation quality and alignment with conditional inputs. The most impactful finding is the decomposition of CFG’s update and the subsequent suppression of the parallel component, directly impacting how practitioners can control saturation levels in generated images. Follow-up questions: 1. How does the performance of APG compare to CFG when using different text embedding methods or prompt engineering techniques in text-to-image generation? 2. Could the insights from APG’s decomposition of CFG updates be applied to other guidance methods or even other generative model architectures beyond diffusion models? 3. Are there specific types of conditional inputs (e.g., complex text prompts) where APG’s advantages are more pronounced compared to CFG? |
| SageAttention: Accurate 8-Bit Attention for Plug-and-play Inference Acceleration (Read more on arXiv or HuggingFace) | Jun Zhu, Pengle Zhang, Jia wei, Jintao Zhang, surfingtomchen | a) The research aimed to develop a quantized attention mechanism for transformers that accelerates inference without significant accuracy degradation. b) SageAttention quantizes Q and K tensors to INT8 after smoothing K by subtracting the mean across tokens, utilizes FP16 accumulators for the PV matrix multiplication, and employs an adaptive quantization strategy to select the fastest kernel per layer while maintaining accuracy. c) SageAttention achieves a 2.1x speedup over FlashAttention2 and an average real speedup of 2.83x compared to original attention implementations across various models including Llama2, CogVideoX, Unidiffuser, UltraPixel, and TIMM. d) AI practitioners can use SageAttention as a plug-and-play replacement for existing attention mechanisms to achieve substantial inference speedups in transformer models with negligible performance loss, particularly beneficial for resource-constrained environments or latency-sensitive applications. e) The paper does not explicitly detail the memory usage reductions achieved by SageAttention. Follow-up questions: 1. What is the memory footprint reduction achieved by SageAttention compared to FP16 attention and other efficient attention methods like FlashAttention2 and xformers? 2. How does the adaptive kernel selection strategy perform in terms of overhead and stability across different hardware and batch sizes? 3. Could the smoothing technique for the K matrix be generalized to other quantization schemes or transformer architectures beyond those tested in the paper? |
| MVGS: Multi-view-regulated Gaussian Splatting for Novel View Synthesis (Read more on arXiv or HuggingFace) | Xin Yu, Yida Wang, xiaobiaodu | a) This paper addresses the problem of overfitting to specific views and imprecise 3D geometry in novel view synthesis using Gaussian-based explicit representations like 3D Gaussian Splatting (3DGS). b) The authors introduce Multi-View Gaussian Splatting (MVGS), incorporating multi-view regulated learning, cross-intrinsic guidance, cross-ray densification, and multi-view augmented densification to improve optimization and prevent overfitting. c) MVGS improves NVS performance across various tasks, including a demonstrated improvement of over 1dB PSNR on the Tanks & Temples dataset when integrated with 3DGS and Scaffold-GS compared to their single-view counterparts. d) AI practitioners working with Gaussian-based explicit representations for novel view synthesis can leverage MVGS as a general optimization solution to enhance reconstruction accuracy and view generalization, particularly in challenging scenarios like reflections or dynamic scenes. Follow-up questions: 1. What is the computational overhead of incorporating multi-view training and the proposed densification strategies compared to standard single-view optimization in 3DGS? How does this impact real-time rendering capabilities? 2. The paper mentions performance degradation with excessive multi-view training. What is the optimal number of views (M) in relation to scene complexity and how can this be determined dynamically or automatically? |
| L-CiteEval: Do Long-Context Models Truly Leverage Context for Responding? (Read more on arXiv or HuggingFace) | Jianye Hou, Baibei Ji, Juntao Li, Keyan Zhou, ZetangForward | a) This research investigates whether Long-Context Models (LCMs) genuinely utilize provided context for generating responses or rely on inherent knowledge. b) A multi-task benchmark, L-CiteEval, was created, requiring LCMs to generate statements and supporting citations from long contexts (8K-48K tokens) across 11 tasks. Automatic evaluation metrics for both generation quality (e.g., precision, recall, Rouge-L) and citation quality (citation recall, precision, and F1) were used. c) Open-source LCMs lagged significantly behind closed-source models in citation accuracy, with a performance gap of nearly 20 F1 points observed in some synthetic tasks, despite citing a similar number of segments. d) AI practitioners should be aware that current open-source LCMs are prone to generating responses from internal knowledge rather than the provided context, posing risks for faithfulness in applications. The benchmark and its automatic evaluation suite provide a tool for evaluating and improving context utilization in LCM development. e) The paper notes a correlation between LCM attention mechanisms and the citation generation process but doesn’t provide details on the strength or nature of this correlation. Follow-up questions: 1. What specific architectural differences between the tested open-source and closed-source LCMs could be contributing to the disparity in citation accuracy? 2. How does the choice of retrieval method in the RAG approach impact both generation and citation quality across different task types and context lengths? 3. Can the observed correlation between attention mechanisms and citation generation be leveraged to develop more explainable or controllable LCMs for long-context tasks? |
| Training Language Models on Synthetic Edit Sequences Improves Code Synthesis (Read more on arXiv or HuggingFace) | Rob Fergus, lerrel, upiter | a) This research investigates whether training language models (LLMs) on synthetic code edit sequences, rather than complete programs, improves code synthesis performance, particularly in terms of the trade-off between generation quality and inference-time compute cost. b) The authors develop LintSeq, an algorithm that refactors existing programs into sequences of static error-free edits using a linter. LLMs are then instruction fine-tuned on these synthetic edit sequences and evaluated on code synthesis benchmarks. c) On HumanEval, smaller LLM’s (e.g., TinyCodeLM-150M and 400M) fine-tuned on synthetic edit sequences outperform existing code language models of comparable size and achieve a 20% (±3%) absolute improvement in pass@50 compared to baseline fine-tuning on full program code. d) For AI practitioners working with smaller LLMs, this research suggests that fine-tuning on synthetic edit sequences generated using a tool like LintSeq can significantly improve code synthesis performance and provide a more favorable trade-off between computational cost and generation quality, enabling competitiveness with larger models using repeated sampling. Follow-up questions: 1. How does the performance of LintSeq-trained models compare to baseline models on other code synthesis benchmarks beyond HumanEval and MBPP, especially those involving longer or more complex code generation? 2. What are the practical limitations and computational costs associated with generating and storing large datasets of synthetic code edits using LintSeq for training larger LLMs? 3. How robust is the LintSeq approach to different programming languages and how can it be adapted for other code editing tasks besides program synthesis, such as code completion or bug fixing? |
| Distilling an End-to-End Voice Assistant Without Instruction Training Data (Read more on arXiv or HuggingFace) | Michael Ryan, Ella Li, zyanzhe, missblanchett, WillHeld | a) The research aimed to develop a Speech Large Language Model (Speech LLM) that generalizes well without requiring instruction training data, addressing the “forgetting” issue observed in models fine-tuned with supervised finetuning (SFT). b) The study employed a cross-modal context distillation method, training a model named Distilled Voice Assistant (DiVA) on the CommonVoice dataset. DiVA leverages a frozen Llama 3 language model and a Q-Former initialized from Whisper, minimizing the L2 distance between audio and text embeddings and the KL Divergence between their output distributions. c) DiVA generalized to Spoken Question Answering, Classification, and Translation tasks. In a user study comparing DiVA with Qwen 2 Audio, DiVA achieved a 72% win rate based on user preference. d) This research provides AI practitioners with a data-efficient and computationally less expensive approach to developing Speech LLMs that generalize well, potentially reducing the reliance on extensive labeled instruction datasets. The significant user preference for DiVA over existing SFT models suggests a potential disconnect between benchmark evaluations and real-world user experience. Follow-up questions: 1. How does DiVA’s performance compare to SFT models on a broader range of spoken language understanding tasks beyond those evaluated in the paper? 2. What are the limitations of using context distillation for tasks where prosodic information in speech plays a crucial role, and how can these limitations be addressed? 3. How does the choice of the base LLM affect DiVA’s performance, and could performance be further improved by using a more powerful LLM or by fine-tuning the LLM’s parameters? |
| MedVisionLlama: Leveraging Pre-Trained Large Language Model Layers to Enhance Medical Image Segmentation (Read more on arXiv or HuggingFace) | Amir Shmuel, Janine Mendola, amanchadha, gurucharan-marthi | a) This research explored enhancing Vision Transformer (ViT) performance for medical image segmentation by integrating frozen transformer blocks from pre-trained Large Language Models (LLMs). b) The study integrated a frozen LLM transformer block within the encoder of a ViT, alongside a proposed Hybrid Attention Mechanism and Multi-Scale Fusion Block. The model was evaluated on 10 medical image segmentation tasks from the Medical Segmentation Decathlon (MSD) dataset. c) The integration of the Llama 3.1 LLM transformer block improved the average Dice score from 0.74 (baseline ViT) to 0.79. d) AI practitioners working on medical image segmentation tasks can leverage pre-trained LLM layers to boost the performance of ViT models without requiring larger datasets or excessive computational resources for LLM training. The paper notes the improved effectiveness seen at higher image resolutions, which could guide practitioners in model selection for specific tasks. Follow-up questions: 1. The paper mentions a Hybrid Attention mechanism. How does this mechanism’s design specifically contribute to the observed performance gains, and what are the computational trade-offs compared to standard attention mechanisms in ViTs? 2. Given the observation that lighter LLMs like Yi and Qwen performed well, what specific architectural factors within these models might be contributing to their effectiveness in medical image segmentation compared to heavier models like Llama and Gemma? Further research directly comparing these architectures on more datasets would be very insightful. 3. While the paper focuses on the MSD dataset, how generalizable are these findings to other medical imaging modalities or datasets with varying characteristics (e.g., noise levels, resolution)? Would further investigation on private datasets reveal a similar performance boost? |
| Vinoground: Scrutinizing LMMs over Dense Temporal Reasoning with Short Videos (Read more on arXiv or HuggingFace) | Jianrui Zhang, yjlee0222, mucai | a) The research investigates the ability of large multimodal models (LMMs) to perform dense temporal reasoning in short videos. b) A new benchmark dataset, Vinoground, consisting of 1000 short video-caption pairs with temporal counterfactuals, was created and used to evaluate several CLIP-based and text-generative LMMs. Models were tasked with matching videos to captions differing only in temporal ordering of events. c) GPT-40 achieved the highest text score among LMMs at 54.0%, significantly below human performance (~90%), and all CLIP-based models performed worse than random chance. d) The results demonstrate a significant deficiency in current LMMs regarding dense temporal reasoning, even in short videos, highlighting this as a critical area for future development and refinement. The paper’s introduction states that a “single-frame bias” exists in current video-language benchmarks and therefore the community has shifted its attention toward more complex challenges posed by long-form video understanding; however, the results reported in this paper suggest that short-form video comprehension is itself a problem that is far from being solved. Follow-up questions: 1. How does the performance of LMMs on Vinoground vary with different video encoding strategies, such as varying the number of sampled frames or using different temporal fusion methods? 2. What specific architectural modifications or training paradigms could be explored to improve LMMs’ ability to capture and reason about the temporal dynamics present in videos? 3. Could transfer learning from pre-trained models specialized in action recognition or temporal ordering improve performance on Vinoground, and how could such transfer learning be effectively implemented? |
| Synthio: Augmenting Small-Scale Audio Classification Datasets with Synthetic Data (Read more on arXiv or HuggingFace) | manocha, ctnzr, rafaelvalle, ZhifengKong, SreyanG-NVIDIA | This research aims to improve audio classification accuracy with limited labeled data. The Synthio method augments small-scale datasets using synthetic audio generated from a text-to-audio (T2A) diffusion model aligned with the target dataset using preference optimization and prompted with diverse captions generated by LLMs. Evaluation on ten downsampled datasets showed Synthio outperformed baselines by 0.1%-39% in classification accuracy. This implies that AI practitioners can leverage synthetic data generated from aligned T2A models, coupled with diverse captioning techniques, to significantly improve the performance of audio classification models trained on limited data. Follow-up questions: 1. How does the computational cost of Synthio, including LLM prompting and T2A generation, compare to the cost of collecting and labeling more real-world audio data? 2. The paper mentions limitations regarding the T2A model’s occasional inability to match generated audio with captions compositionally; how could this limitation be addressed to improve Synthio’s applicability to tasks like audio captioning? 3. Could the preference optimization technique used to align the T2A model be adapted or improved for other generative models beyond audio, such as image or text generation? |
Papers for 2024-10-03
| Title | Authors | Summary |
|---|---|---|
| From Code to Correctness: Closing the Last Mile of Code Generation with Hierarchical Debugging (Read more on arXiv or HuggingFace) | Xiaodong Gu, Chengcheng Wan, Songsong Wang, YerbaPage | This research addresses the problem of low pass rates in LLM-generated code due to subtle errors. The authors introduce MGDebugger, which uses a hierarchical, bottom-up debugging strategy, decomposing code into subfunctions and debugging them recursively with LLM-simulated execution and automatically generated test cases. Experiments on HumanEval show MGDebugger improves accuracy by 17.7% over seed generations when using DeepSeek-Coder-V2-Lite (16B). This implies that AI practitioners can significantly improve the correctness of LLM-generated code by adopting hierarchical debugging strategies rather than treating programs as monolithic units. The paper states MGDebugger achieves a 97.6% repair success rate on HumanEval-Fix using DeepSeek-Coder-V2-Lite (16B); however, it doesn’t clarify the baseline repair success rate for this dataset/model combination, making it difficult to assess the relative improvement. Follow-up questions: 1. How does MGDebugger’s performance compare to traditional symbolic execution or program analysis techniques for debugging, especially in terms of scalability and handling complex codebases? 2. What are the computational resource requirements (e.g., memory, time) of MGDebugger compared to other LLM-based debugging methods, and how do they scale with code size and complexity? 3. Could the hierarchical decomposition strategy be automated further, and what are the potential challenges in applying it to real-world codebases with complex dependencies and interactions between modules? |
| Is Preference Alignment Always the Best Option to Enhance LLM-Based Translation? An Empirical Analysis (Read more on arXiv or HuggingFace) | nunonmg, PierreColombo, CelineH, emmanuelmalherbe, hgissbkh | a) This paper investigates the effects of preference-based alignment, particularly Contrastive Preference Optimization (CPO), on the quality of Large Language Model (LLM)-based translations. b) The researchers conducted experiments fine-tuning an LLM translation model with CPO and Supervised Fine-Tuning (SFT), using various quality metrics (xCOMET-QE, CometKiwi, chrF) for alignment and evaluation, with both multi-system and mono-system candidate generation approaches. c) CPO consistently outperformed SFT on high-quality data when aligning with neural metrics like xCOMET-QE, sometimes significantly increasing scores on the alignment metric (e.g., +2.75 for xCOMET-QE in en-xx translations with a multi-system approach). However, it also introduced adverse effects between neural and lexical metrics, and exhibited sensitivity to the chosen candidate systems. d) AI practitioners aligning LLMs for translation should carefully consider the choice of candidate generation systems and potential trade-offs between optimizing neural versus lexical metrics when employing CPO. The instability of CPO across different downstream metrics warrants caution. The mono-system approach offers more control and may mitigate some of these issues while achieving comparable alignment effectiveness. This improved control stems from being able to fine-tune the choice of candidate option quality with greater precision in the mono-system setting. Follow-up questions: 1. How does the computational cost of generating multiple candidates in the mono-system approach compare to the cost of accessing and using multiple external systems in the multi-system approach? 2. Could the instability of CPO be addressed by exploring different values for the β hyperparameter or by modifying the training procedure (e.g., different optimizers, learning rate schedules)? 3. What are the practical implications of the adverse metric effects between neural and lexical metrics for real-world translation applications, where both types of metrics are often considered important? |
| LEOPARD : A Vision Language Model For Text-Rich Multi-Image Tasks (Read more on arXiv or HuggingFace) | Zhihan Zhang, Tianqing Fang, Mengzhao Jia, kaixinm, wyu1 | This research aimed to develop a multimodal large language model (MLLM) capable of handling text-rich, multi-image tasks. The researchers curated a one-million-instance instruction-tuning dataset (LEOPARD-INSTRUCT) and implemented an adaptive high-resolution multi-image encoding module based on pixel shuffling. LEOPARD-Idefics2, a variant trained on this dataset, outperformed the previous best-performing open-source MLLM on text-rich multi-image benchmarks by an average of 9.61 points. This suggests that LEOPARD and its associated dataset are valuable resources for developing MLLMs specialized in complex, text-rich, multi-image scenarios. The paper doesn’t explicitly state the metric used for the +9.61 point improvement, though it does mention average normalized levenshtein similarity and accuracy in Table 3, making it difficult to understand precisely what this improvement represents. Follow-up questions: 1. What specific metric (e.g., accuracy, F1-score, etc.) was used to calculate the +9.61 point improvement on the multi-image text-rich benchmarks, and on which specific subset of benchmarks was this average calculated? 2. What is the computational cost (e.g., GPU hours, FLOPs) of training LEOPARD compared to baseline models, and how does the adaptive high-resolution encoding module impact inference time? 3. Can the adaptive high-resolution encoding module be effectively applied to other visual encoders besides SigLIP-SO-400M, and are there plans to release the LEOPARD-INSTRUCT dataset publicly? |
| ComfyGen: Prompt-Adaptive Workflows for Text-to-Image Generation (Read more on arXiv or HuggingFace) | galchechik, cohenor, yuvalalaluf, adihaviv, rinong | a) This research aims to improve text-to-image generation quality by automatically tailoring workflows to individual user prompts. b) The authors propose two LLM-based approaches: ComfyGen-IC uses an LLM with a pre-computed table of flows and scores for prompt categories to select flows, while ComfyGen-FT fine-tunes an LLM to predict flows based on prompts and target scores. Both leverage ComfyUI, representing workflows as JSON. c) ComfyGen-FT outperforms baseline models and generic workflows on both human preference and prompt alignment benchmarks, achieving a 0.61 overall score on GenEval compared to 0.59 for the best baseline. d) This work indicates that AI practitioners can improve text-to-image generation quality by moving beyond fixed models or generic workflows and adopting prompt-adaptive workflow generation techniques. Specifically, fine-tuning LLMs to predict workflows based on both prompts and target scores shows promise for enhanced performance. Follow-up questions: 1. What are the computational costs and scalability challenges associated with training and deploying ComfyGen-FT, particularly for large datasets and complex workflows? 2. How does the performance of ComfyGen-FT vary across different LLM architectures and sizes, and what are the trade-offs between performance and computational resources? 3. Can the proposed framework be extended to other generative tasks beyond text-to-image generation, such as image editing or video generation, and what adaptations would be necessary? |
| Not All LLM Reasoners Are Created Equal (Read more on arXiv or HuggingFace) | Aaron Courville, Daniel Toyama, Alessandro Sordoni, agarwl, arianhosseini | This research investigates the depth of grade-school math (GSM) problem-solving and reasoning capabilities of LLMs. The study evaluates LLM performance on Compositional GSM, a new dataset derived from GSM8K, requiring models to solve chained math problems where the answer to the first question is a variable in the second. Results reveal a significant reasoning gap, defined as the performance difference between solving compositional pairs and individual questions; for example, the smaller, more cost-efficient GPT-40 mini exhibits a 14.2% reasoning gap on compositional GSM despite high accuracy on GSM8K. This implies that instruction-tuning, while effective for single-step problem-solving, does not necessarily translate to improved multi-hop reasoning, and high scores on standard benchmarks may mask deficiencies in compositional reasoning abilities, a critical insight for AI practitioners developing and applying such models. Follow-up Questions: 1. What specific modifications were made to the GSM8K problems to create the Compositional GSM dataset, and how might these modifications differentially impact various LLM architectures or training paradigms? 2. Given the observed overfitting during finetuning on GSM8K, what alternative training strategies could be explored to improve compositional reasoning without sacrificing generalization performance on other tasks? 3. Could the study’s findings about the reasoning gap in cost-efficient models be extrapolated to other problem domains beyond grade-school math, and if so, what are the implications for real-world AI applications where resource constraints are a major factor? |
| 3DGS-DET: Empower 3D Gaussian Splatting with Boundary Guidance and Box-Focused Sampling for 3D Object Detection (Read more on arXiv or HuggingFace) | Dan Xu, Yuanliang, YangCaoCS | a) The paper aims to introduce 3D Gaussian Splatting (3DGS) for 3D object detection, addressing the challenges of ambiguous spatial distribution and excessive background blobs encountered when adapting 3DGS to this task. b) The authors propose a novel method called 3DGS-DET, incorporating two key strategies: 2D Boundary Guidance, which utilizes object boundaries from posed images to train the 3DGS model, and Box-Focused Sampling, which constructs 3D object probability spaces based on 2D bounding boxes for probabilistic sampling of Gaussian blobs. c) On the ScanNet dataset, 3DGS-DET achieves a mean Average Precision (mAP) of 59.9 at an Intersection over Union (IoU) threshold of 0.25, surpassing the baseline 3DGS pipeline by 5.6 points. d) AI practitioners can leverage the proposed 3DGS-DET method to achieve improved performance in 3D object detection tasks by utilizing the explicit and efficient representation offered by 3DGS, enhanced with boundary and sampling strategies. The paper specifically notes that other detectors can potentially use the enhanced 3DGS representations. Follow-up questions: 1. Could the performance of 3DGS-DET be further improved by jointly training the 3DGS representation and the detection network, rather than training them sequentially? 2. How does the computational cost of Boundary Guidance and Box-Focused Sampling compare to other 3D object detection methods, particularly those based on point clouds or voxels? 3. The paper mentions using CAGroup3D and FCAF3D as detectors. Could the specific detector choice significantly impact the results observed? Would other detectors trained on point clouds yield similar improvements from using the 3DGS representations? |
| HelpSteer2-Preference: Complementing Ratings with Preferences (Read more on arXiv or HuggingFace) | okuchaiev, gshennvm, trias702, odelalleau, alexwb | a) This paper investigates whether Bradley-Terry style or Regression style reward models are more effective for aligning language models to instructions, and explores combining both approaches. b) The authors collect preference annotations and justifications alongside existing ratings in the HelpSteer2 dataset, enabling a head-to-head comparison of both reward modeling styles. They also experiment with a novel combined approach, initializing a Scaled Bradley-Terry model with a Helpfulness-Only SteerLM Regression model, and further refining it with ExPO. c) The combined reward model (Scaled BT + EXPO) achieves 94.1% on RewardBench, outperforming over 140 other reward models as of October 1, 2024. d) AI practitioners can leverage this combined reward model and the HelpSteer2-Preference dataset for training more accurate reward models, especially for RLHF, and potentially improve the performance of language models at following instructions. Follow-up questions: 1. How does the performance of the combined reward model (Scaled BT + EXPO) vary across different RewardBench categories (Chat, Chat-Hard, Safety, Reasoning), and what are the potential reasons for such variations? 2. What are the computational resource requirements (e.g., memory, FLOPs) for inference with the combined reward model compared to individual Bradley-Terry or Regression models? 3. What specific techniques were used for pre-processing the preference justifications, and how did those pre-processing steps impact the performance of Pairwise Justifier models? |
| RATIONALYST: Pre-training Process-Supervision for Improving Reasoning (Read more on arXiv or HuggingFace) | Guoxuan Wang, danyaljj, ChuyuLiu, ylu610, Dongwei | a) The research aims to improve the reasoning capabilities of Large Language Models (LLMs) by addressing the issue of incomplete reasoning chains with implicit rationales. b) The proposed method, RATIONALYST, involves extracting implicit rationales from unlabeled text (The Pile) and reasoning datasets (GSM8K and ECQA), training a model to predict these rationales, and using the predicted rationales to provide process-supervision during LLM inference. c) Fine-tuned from LLaMa-3-8B, RATIONALYST improves the accuracy of reasoning by an average of 3.9% on seven representative reasoning benchmarks, including mathematical, commonsense, scientific, and logical reasoning datasets. d) AI practitioners can use RATIONALYST to enhance the reasoning performance and interpretability of LLMs across various tasks by incorporating a process-supervision mechanism based on implicit rationales extracted from readily available unlabeled data. The improved interpretability is particularly important for debugging and gaining deeper insights into LLM’s reasoning process. Follow-up Questions: 1. How does the performance of RATIONALYST scale with larger base LLMs (e.g., LLaMa-3-70B) or more powerful rationale extractors (e.g., GPT-4)? 2. What are the computational costs and infrastructure requirements associated with extracting and filtering rationales from large datasets like The Pile, and how can these be optimized? 3. Could RATIONALYST be adapted for specific domains or tasks by training it on a curated dataset of domain-specific rationales, and how would this impact its performance and generalizability? |
| Quantifying Generalization Complexity for Large Language Models (Read more on arXiv or HuggingFace) | maxtiktok, Nrain, zhuokai, Xulianghuang, luohy | This research investigates how task complexity and model size affect the generalization ability of Large Language Models (LLMs). The study uses SCYLLA, a dynamic benchmark generating in-distribution and out-of-distribution data for 20 tasks across varying complexities. Results reveal a “generalization valley,” where the performance gap between in-distribution and out-of-distribution data is non-monotonic, peaking at a “critical complexity” that shifts rightward with increasing model size. Specifically, LLaMA-3.1-405B achieved near-perfect generalization scores (0.997 and 0.996) on O(N) and O([N, N²]) tasks, respectively. This suggests that scaling LLM size improves generalization, delaying but not eliminating over-reliance on memorization at higher task complexities. Follow-up questions: 1. How does the specific distribution of OOD data generation in SCYLLA affect the observed generalization valley, and how would these results compare if alternative OOD sampling strategies were employed? 2. Given the implicit reasoning observed in models like o1-mini, what further analysis could be conducted to better understand and potentially leverage these capabilities in downstream tasks or model development? 3. Could the performance of specialized LLMs (e.g., Qwen2.5-Math-7B) at higher complexities be improved by utilizing multi-stage prompting that decomposes complex tasks into sub-tasks within their expertise range? |
| EVER: Exact Volumetric Ellipsoid Rendering for Real-time View Synthesis (Read more on arXiv or HuggingFace) | George Kopanas, Alexander Mai, xharlie, dorverbin, phedman | a) The research aims to develop a real-time, differentiable, emission-only volume rendering method that addresses the limitations of existing techniques like 3D Gaussian Splatting (3DGS), particularly “popping” artifacts. b) The proposed method, Exact Volumetric Ellipsoid Rendering (EVER), represents the scene as a collection of constant-density ellipsoids and uses ray tracing to compute the volume rendering integral exactly. This allows for the inclusion of effects like defocus blur and fisheye lens distortion. c) EVER achieves a framerate of 30 FPS at 720p resolution on an NVIDIA RTX4090 on the challenging Zip-NeRF dataset and achieves a lower LPIPS score (0.368) compared to existing real-time methods like 3DGS (0.418) and StopThePop (0.411). d) AI practitioners working on novel view synthesis can use EVER to generate high-quality, pop-free renderings in real-time, enabling applications that require fast and consistent 3D scene representations. The paper does not state the impact on memory usage, nor quantify inference time on hardware other than an NVIDIA RTX4090. Follow-up questions: 1. How does the memory footprint of EVER compare to 3DGS, particularly when scaling to even higher resolution or more complex scenes? 2. Could the constant density assumption of EVER be relaxed to allow for more complex density variations within individual primitives, and how would that impact performance and quality? 3. What is the performance (FPS and quality metrics) of EVER on other commonly used GPUs, besides the NVIDIA RTX 4090 mentioned in the paper? |
| E.T. Bench: Towards Open-Ended Event-Level Video-Language Understanding (Read more on arXiv or HuggingFace) | Ying Shan, Yang Wu, Zhongang Qi, Zongyang Ma, Ye Liu | a) This research addresses the lack of fine-grained event-level and diverse task assessment in current video-language understanding benchmarks, aiming to create a more comprehensive evaluation for Video Large Language Models (Video-LLMs). b) The authors introduce E.T. Bench, a benchmark with 7.3K samples across 12 tasks and 8 domains, focusing on event-level and time-sensitive understanding of long videos. They also propose E.T. Chat, a novel Video-LLM using embedding matching for timestamp prediction, and E.T. Instruct 164K, a dedicated instruction-tuning dataset. c) State-of-the-art Video-LLMs struggle with E.T. Bench, especially on grounding and dense captioning tasks, while E.T. Chat achieves state-of-the-art performance among open-source models, with a 38.4% Accref (averaged accuracy on referring tasks) on E.T. Bench. d) AI practitioners developing Video-LLMs should consider incorporating finer-grained temporal understanding and multi-event scenarios in training data and model design, prioritizing both spatial and temporal reasoning capabilities for improved performance on complex video understanding tasks. The paper notes potential data leakage in benchmark evaluation due to overlap with existing datasets used for model training, which might affect the validity of zero-shot evaluation. Follow-up questions: 1. Given the limitations of discrete token prediction for timestamps, what other alternative approaches besides embedding matching could be explored for improving temporal understanding in Video-LLMs? 2. How can the E.T. Bench benchmark be improved to mitigate the potential data leakage issue mentioned in the paper and ensure a more robust evaluation of Video-LLMs in zero-shot settings? 3. What specific architectural modifications in E.T. Chat contribute to its superior performance on grounding and dense captioning tasks compared to other state-of-the-art open-source Video-LLMs? |
| Closed-loop Long-horizon Robotic Planning via Equilibrium Sequence Modeling (Read more on arXiv or HuggingFace) | Jiazhong Yu, Cao Sheng, Fei Li, feifeiobama, ljh0104 | a) The research aims to improve closed-loop long-horizon robotic planning in LLMs by addressing limitations like unidirectional dependency and lack of error correction. b) The paper proposes “equilibrium sequence modeling,” formulating self-refinement as a fixed-point problem solved through iterative refinement and utilizing a nested equilibrium solving process to incorporate environmental feedback efficiently. An experience memory and world model complement the planner. c) Evaluated on VirtualHome-Env, the method achieved a success rate improvement of up to 19% with error correction compared to not using error correction. It shows superior scaling for inference computation. d) This provides AI practitioners a supervised learning approach to train self-refining LLM planners for robotics without needing complex reinforcement learning or process supervision, potentially leading to more robust and efficient long-horizon task completion. Follow-up questions: 1. What are the specific architectural details of the world model used, and how does its performance compare to more complex world models that simulate environmental states rather than just feedback? 2. How does the proposed method’s computational cost during training and inference scale with increasing model size and task complexity compared to alternative approaches like Tree-Planner or SELF-REFINE? 3. The paper mentions failure scenarios like hallucination and lack of history awareness. What specific mitigation strategies, beyond the mentioned reasoning techniques, could be explored to address these limitations? |
| HarmoniCa: Harmonizing Training and Inference for Better Feature Cache in Diffusion Transformer Acceleration (Read more on arXiv or HuggingFace) | Xinjie Zhang, Jing Liu, Ruihao Gong, Zining Wang, Yushi Huang | a) Objective: To accelerate the inference speed of Diffusion Transformers (DiTs) for image generation tasks by mitigating discrepancies between training and inference in learning-based feature caching methods. b) Methodology: HarmoniCa framework, employing Step-Wise Denoising Training (SDT) to align training with the full denoising trajectory and Image Error Proxy-Guided Objective (IEPO) to incorporate final image error into training. c) Results: HarmoniCa achieved a 1.52x speedup and an FID of 27.61 for PIXART-α 256×256 with a 20-step DPM-Solver++, compared to an FID of 27.68 for the non-accelerated model. d) Implication: AI practitioners can leverage HarmoniCa to significantly reduce inference latency in DiT models without substantial performance degradation, improving practical deployment for high-resolution image generation tasks. This is particularly relevant to generative AI application developers. Follow-Up Questions: 1. How does the performance of HarmoniCa scale with even larger DiT models and higher resolutions beyond those tested in the paper (e.g., greater than 2048x2048)? 2. Could the proxy mechanism in IEPO be further refined to more accurately represent final image error, potentially leading to further performance gains? 3. What is the memory footprint of HarmoniCa during inference, and how does it compare to other acceleration techniques like pruning or quantization, particularly for resource-constrained environments? |
| Selective Aggregation for Low-Rank Adaptation in Federated Learning (Read more on arXiv or HuggingFace) | Huijie Fan, Liangqiong-QU, yanranw1, stevezs, gpx333 | a) This paper investigates how to effectively aggregate Low-Rank Adaptation (LoRA) matrices in Federated Learning (FL) for improved performance on downstream tasks. b) The authors introduce Federated Share-A LoRA (FedSA-LORA), where both A and B matrices of the LoRA update are trainable during local training, but only the A matrices (responsible for general knowledge) are aggregated on the server. This method is then generalized to other LoRA variants (rsLoRA and VeRA). c) On the GLUE benchmark’s RTE task with a severe non-IID data distribution, FedSA-LoRA achieved 90.20% accuracy, outperforming standard LORA (88.80%) and FFA-LoRA (88.83%). d) AI practitioners can use FedSA-LoRA to efficiently fine-tune large language models in federated learning settings, especially with non-IID data, by reducing communication overhead and improving performance compared to existing methods. The impactful finding, that A matrices capture general knowledge while B matrices learn client-specific knowledge, allows for more targeted aggregation and better generalization across clients. Follow-up questions: 1. How does the performance of FedSA-LoRA scale with the number of clients and the heterogeneity of the data distribution in more complex real-world scenarios beyond the presented experiments? 2. What are the computational and memory overheads of FedSA-LoRA compared to other PEFT methods in federated settings, particularly for very large language models? 3. How robust is FedSA-LoRA to malicious client behavior, and what mitigation strategies could be implemented to enhance its security in adversarial federated learning environments? |
Papers for 2024-10-02
| Title | Authors | Summary |
|---|---|---|
| Law of the Weakest Link: Cross Capabilities of Large Language Models (Read more on arXiv or HuggingFace) | xwhan, ruihou16, xwwang, astonzhang, MingZhong | The paper investigates the under-explored area of cross-capabilities in Large Language Models (LLMs), defined as the intersection of multiple abilities required for complex tasks. The authors introduce CROSSEVAL, a benchmark comprising 1400 human-annotated prompts across seven individual and seven cross-capabilities, and use LLM-based evaluators to assess model responses. Results reveal that cross-capability performance is often constrained by the weakest individual capability, exhibiting a “Law of the Weakest Link,” where 38 out of 58 cross-capability scores from 17 models fell below all individual capability scores. This highlights the need to focus on improving weaker capabilities for better overall performance. Follow-up questions: 1. How can CROSSEVAL be extended to encompass a wider range of cross-capabilities and incorporate more nuanced evaluation metrics beyond the 1-5 Likert scale? 2. What specific training strategies can be employed to effectively address the “Law of the Weakest Link” and improve LLM performance in tasks requiring multiple abilities? 3. How can the insights from this research be applied to the development and evaluation of LLM-based agents operating in real-world scenarios? |
| TPI-LLM: Serving 70B-scale LLMs Efficiently on Low-resource Edge Devices (Read more on arXiv or HuggingFace) | Hongfang Yu, Mohsen Guizani, Jiaoshen, LIKirin | a) This paper investigates how to efficiently serve large language models (LLMs), specifically 70B-scale models, on resource-constrained edge devices. b) The researchers developed TPI-LLM, a tensor parallel inference system with a sliding window memory scheduler to manage model weights dynamically and a star-based allreduce algorithm for inter-device communication. c) Experimental results on emulated and real testbeds demonstrated that TPI-LLM reduced the time-to-first-token and token latency by over 80% compared to Accelerate and over 90% compared to Transformers and Galaxy. It also reduced the peak memory footprint of Llama 2-70B by 90%, requiring only 3.1 GB of memory per device. d) TPI-LLM offers AI practitioners a viable solution for deploying and running large-scale LLMs on edge devices, addressing privacy concerns and limitations in memory and computing power, thus enabling broader LLM applications on edge devices. Follow-up questions: 1. What is the impact of varying the size of the sliding window on the trade-off between memory footprint and inference speed in real-world scenarios with diverse network conditions? 2. How does TPI-LLM perform with quantized LLMs, and what are the potential trade-offs between model accuracy and efficiency when using quantization on edge devices? 3. Could the star-based allreduce algorithm be further optimized for heterogeneous edge device clusters with varying compute power and network latency characteristics? |
| Atlas-Chat: Adapting Large Language Models for Low-Resource Moroccan Arabic Dialect (Read more on arXiv or HuggingFace) | imomayiz, amr-mohamed, khoubrane-yousef, habdine, guokan-shang | This paper investigates adapting large language models (LLMs) for the low-resource Moroccan Arabic dialect, Darija. The researchers construct a large instruction dataset from diverse sources, including existing Darija resources, manually and synthetically created data, and translated English instructions. Fine-tuned 2B and 9B parameter Gemma models, Atlas-Chat, show superior performance compared to other LLMs like LLaMa, Jais, and AceGPT, achieving 58.23% and 81.89% accuracy on DarijaMMLU and Sentiment Analysis, respectively, with the 9B model. This work demonstrates successful LLM adaptation for a low-resource dialect. Follow Up Questions: 1. What specific pre- and post-processing techniques were used for the English-to-Darija translation of the instruction datasets, and how did these impact the final model performance? 2. How does the performance of the smaller 2B model compare to the 9B model in resource-constrained environments, considering factors like inference speed and memory usage? 3. What are the limitations of the current evaluation benchmarks for Darija, and what further work is needed to develop more comprehensive and robust evaluation metrics for this dialect? |
| One Token to Seg Them All: Language Instructed Reasoning Segmentation in Videos (Read more on arXiv or HuggingFace) | sebgao, wangpichao, meihaiyang, tonghe, ZechenBai | a) The research aims to develop a video-based multimodal large language model (MLLM) for language-instructed reasoning segmentation in videos, generating temporally consistent masks based on complex language queries. b) VideoLISA, the proposed model, integrates a Sparse Dense Sampling strategy for balancing temporal context and spatial detail, a One-Token-Seg-All approach using a |
| Illustrious: an Open Advanced Illustration Model (Read more on arXiv or HuggingFace) | Junha Lee, leehg57, mhy9910, solbon1212, andyp-nvidia | a) The research aimed to develop an open-source, state-of-the-art anime image generation model, Illustrious, surpassing existing models in terms of animation style, high resolution, dynamic color range, and restoration ability. b) The key methodology involved training on a large, refined dataset of anime images with multi-level captions (tags and natural language descriptions), utilizing a No Dropout Token approach for preserving specific concepts, and training at higher resolutions (up to 2.25MP) to enable high-resolution output. The training used Stable Diffusion XL as a base, with modifications including Cosine Annealing scheduler and Input Perturbation Noise Augmentation. c) Illustrious v1.1 achieved a median CCIP (Character Consistency Image Prompt) score of 0.99 in a character similarity evaluation. The paper notes higher ELO ratings for Illustrious compared to other models in user preference studies, but the specific methodology for these ELO calculations needs further clarification. d) AI practitioners can utilize Illustrious as a high-quality, open-source model for generating anime illustrations at resolutions up to 20MP. The No Dropout Token approach and multi-level caption training methodology may be applicable to other specialized image generation tasks. Follow-up questions: 1. What is the precise formula and methodology used to compute the ELO scores in the user studies, including the composition of user groups, prompting strategies used, and handling of draws? More detailed analysis of the user preference results and their statistical significance would be beneficial. 2. The paper mentions limitations related to text rendering within images. What specific experiments were conducted to investigate this limitation, and what quantitative results were observed? Further investigation of this limitation could aid future research on generating glyphs in stylized images. 3. How does the computational cost of the higher-resolution training and inference compare to lower-resolution approaches, and what trade-offs in terms of memory and training time should practitioners consider when using or adapting Illustrious? |
| Flex3D: Feed-Forward 3D Generation With Flexible Reconstruction Model And Input View Curation (Read more on arXiv or HuggingFace) | Filippos Kokkinos, Andrea Vedaldi, philiptorr, JianyuanWang, Junlinh | a) The paper aims to improve the quality of feed-forward 3D object generation from text, single images, or sparse view images. b) Flex3D, a two-stage framework, is proposed. The first stage generates and curates a pool of candidate views using fine-tuned multi-view and video diffusion models and a view selection pipeline. The second stage reconstructs the 3D object as a set of Gaussian points from the curated views using FlexRM, a flexible reconstruction model based on a transformer architecture and a tri-plane representation. A novel training strategy simulates imperfect input views by adding noise to intermediate 3D Gaussian representations. c) In user studies comparing text-to-3D generation, Flex3D achieved a win rate of over 92% compared to state-of-the-art feed-forward models. Quantitatively, Flex3D achieved 0.277 CLIP text similarity and 0.255 VideoCLIP text similarity, outperforming all compared models. d) AI practitioners can utilize Flex3D’s framework to generate higher-quality 3D objects from various input modalities. The novel view curation and imperfect data simulation techniques provide robust methods to improve 3D reconstruction quality and generalization capabilities, essential for applications requiring accurate and visually appealing 3D assets. Follow-up questions: 1. The paper mentions initializing the MLP and tri-plane transformer with an off-the-shelf tri-plane NeRF network. Are the specific details of this network and its pre-training available, and how critical is this initialization for FlexRM’s performance? 2. While the paper demonstrates improvements on object-centric datasets, how well would Flex3D generalize to more complex scenes containing multiple objects and backgrounds, and what modifications might be necessary for such an extension? 3. The paper focuses on Gaussian splatting as the final 3D representation. Has any investigation been done into the feasibility and performance implications of directly generating meshes or other 3D representations within the Flex3D framework? |
| ACE: All-round Creator and Editor Following Instructions via Diffusion Transformer (Read more on arXiv or HuggingFace) | Jingren, chenweix7, chaojiemao, jingfengzhang, jiangzeyinzi | a) The research aims to develop a unified foundational model for diverse visual generation and editing tasks, addressing the limitations of existing models that are often task-specific. b) ACE (All-round Creator and Editor) employs a Diffusion Transformer architecture with novel components including Long-context Condition Unit (LCU) for handling multi-modal and multi-turn inputs, Image Indicator Embedding for image sequence alignment, and a novel data collection pipeline including synthesis and clustering-based methods. c) On the MagicBrush benchmark, ACE achieved a CLIP-I score of 0.9453 for single-turn instruction-guided image editing, outperforming other methods. A user study on the authors’ ACE benchmark also showed strong performance across various editing tasks. d) AI practitioners can leverage ACE’s unified framework and LCU structure to build multi-modal chat systems and visual agents for complex image generation and editing workflows, potentially streamlining and simplifying existing cumbersome pipelines. The proposed data collection strategy offers efficient methods for acquiring paired image data for training similar models. Follow-up Questions: 1. The paper mentions performance limitations in certain tasks like general editing and style editing compared to larger, task-specific models. Could further analysis of the user study feedback pinpoint specific visual qualities where ACE falls short and guide future model improvements? 2. How does the computational cost of ACE, especially with long-context inputs, scale with the number of input images and turns? Are there optimization strategies planned to improve inference efficiency for real-time applications? 3. While the paper describes the data collection pipeline, details on the Instruction Captioner’s architecture and training process are limited. Could further information be provided on the MLLM used, its performance metrics for instruction generation, and the impact of different instruction generation strategies on ACE’s overall performance? |
| Helpful DoggyBot: Open-World Object Fetching using Legged Robots and Vision-Language Models (Read more on arXiv or HuggingFace) | Xiaolong Wang, Xuxin Cheng, Zipeng Fu, Qi Wu, cbfinn | a) The research aimed to develop a quadrupedal robot system capable of understanding human commands and performing mobile manipulation tasks, such as fetching objects, in unseen indoor environments. b) The system combines a learned low-level controller trained in simulation for agile locomotion and whole-body tilting with pre-trained Vision-Language Models (VLMs) for semantic understanding and command generation. A 1-DoF gripper was designed for object manipulation. c) In real-world tests, the robot achieved a 60% first-attempt success rate in fetching a stuffed toy from a bed, requiring climbing, navigation, and grasping. d) This research demonstrates the potential of integrating simulation-trained low-level controllers with VLMs for enabling zero-shot generalization in robotic mobile manipulation, suggesting a promising approach for developing versatile robot assistants. Follow-up questions: 1. What are the specific architectures and hyperparameters used for the low-level controller (policy network and online estimator) and how were these determined? More detail about the specifics of the network architectures used would be helpful. 2. The paper mentions limitations regarding the gripper’s dexterity. What specific modifications or alternative gripper designs are being considered to improve manipulation capabilities, and how might these impact the robot’s agility and control? 3. How does the system handle object occlusions during navigation and grasping, and what strategies are being explored to improve robustness in more cluttered and dynamic real-world environments? |
| DressRecon: Freeform 4D Human Reconstruction from Monocular Video (Read more on arXiv or HuggingFace) | Shubham Tulsiani, Donglai Xiang, Jeff Tan, gengshan-y, devakramanan | a) The research aims to reconstruct time-consistent 4D human models with loose clothing and handheld objects from monocular videos. b) DressRecon uses a hierarchical bag-of-bones motion model, separating body and clothing deformations, and incorporates image-based priors (pose, normals, optical flow) within a differentiable rendering optimization framework. The model can be refined into explicit 3D Gaussians for interactive rendering. c) On a dataset of 14 challenging sequences from DNA-Rendering, DressRecon achieved an average chamfer distance of 6.411cm, outperforming baseline methods. d) AI practitioners can utilize DressRecon’s approach to create high-fidelity, animatable 3D human avatars from single-viewpoint videos, potentially streamlining avatar creation for virtual environments and other applications. The paper does not specify the computational requirements for training or inference. Follow-up questions: 1. What are the memory and computational requirements for training and inference of DressRecon, and how does it scale with video length and resolution? 2. Could the hierarchical motion model be adapted for other types of non-rigid objects beyond clothing and accessories, and what modifications would be necessary? 3. How robust is the method to variations in lighting, background clutter, and occlusions in the input video? |
| Visual Context Window Extension: A New Perspective for Long Video Understanding (Read more on arXiv or HuggingFace) | Zhenzhong Chen, hcwei | a) This research aims to improve Large Multimodal Models (LMMs) performance on long video understanding tasks without retraining on large video datasets. b) The authors propose extending the visual context window by adapting the YaRN (Yet Another RoPE for Transformers) method, originally designed for language models, and introduce a progressive pooling strategy to reduce memory consumption. c) On the MLVU benchmark, their method with a 7B parameter LMM outperforms GPT-40. d) AI practitioners can leverage this approach to apply pre-trained LMMs to long videos, benefiting from advances in open-source LMMs without the computational cost of retraining on extensive long video-text paired data. The progressive pooling strategy enables efficient memory management when processing long video sequences. Follow-up questions: 1. How does the performance of visual context window extension compare to retraining LMMs on long video data specifically, in terms of accuracy and computational cost? 2. What are the limitations of the progressive pooling strategy, and are there scenarios where information loss becomes significant despite the focus on preserving spatial details? 3. Could the visual context window extension method be adapted or combined with other memory optimization techniques, such as those used for sparse attention? |
| SyntheOcc: Synthesize Geometric-Controlled Street View Images through 3D Semantic MPIs (Read more on arXiv or HuggingFace) | Qing Lian, Xu Yan, Yingjie Cai, Weichao Qiu, Leheng Li | a) The research aimed to develop a framework for generating photorealistic and geometrically-controlled street view images conditioned on 3D occupancy labels. b) The key methodology involves representing 3D occupancy as semantic Multi-Plane Images (MPIs), encoding these MPIs using a 1x1 convolutional encoder, and integrating this into a Stable Diffusion model with cross-view and cross-frame attention. Reweighing strategies address class imbalance and depth-related learning difficulties. c) SyntheOcc achieved a Frechet Inception Distance (FID) of 14.75 on the nuScenes dataset, outperforming baseline methods like BEVGen (FID 25.54) and MagicDrive (FID 16.20). d) AI practitioners can leverage SyntheOcc to generate synthetic datasets for training perception models in autonomous driving, particularly for 3D occupancy prediction, and for creating corner case scenarios for system evaluation. The use of MPIs offers a novel approach for encoding 3D information into 2D diffusion models for enhanced controllability. Follow-up Questions: 1. How does the computational cost of generating MPIs and using the MPI encoder compare to other conditional input methods, such as BEV encodings or text prompts, in terms of memory usage and processing time? 2. What are the limitations of the reweighing strategies, particularly in extremely long-tailed or complex scenarios, and how can these limitations be addressed to improve generation quality and diversity? 3. How robust is the approach to different camera parameters and viewpoints not seen during training, and how could the framework be adapted to handle more diverse camera setups and environments? |
| Posterior-Mean Rectified Flow: Towards Minimum MSE Photo-Realistic Image Restoration (Read more on arXiv or HuggingFace) | Michael Elad, Michato, ohayonguy | a) This paper investigates the optimal estimator for minimizing Mean Squared Error (MSE) in photo-realistic image restoration under a perfect perceptual index constraint. b) The proposed Posterior-Mean Rectified Flow (PMRF) algorithm first predicts the posterior mean of the image and then uses a rectified flow model to transport the result to the distribution of ground-truth images. c) On the CelebA-Test blind face restoration benchmark, PMRF achieved a FID score of 37.46, outperforming all other compared methods. d) AI practitioners working on image restoration can use PMRF to potentially achieve lower distortion without sacrificing perceptual quality compared to posterior sampling or GAN-based methods. Follow-up questions: 1. How does the choice of the noise level (σε) added to the posterior mean prediction in PMRF affect the trade-off between MSE and perceptual quality in different restoration tasks and degradation levels? 2. The paper mentions the possibility of reflow to further improve PMRF. Have the authors explored this, and what were the observed impacts on performance and computational cost? 3. How does PMRF’s performance compare to other state-of-the-art methods when applied to diverse image datasets beyond faces, such as natural scenes or medical images? |
Papers for 2024-10-01
| Title | Authors | Summary |
|---|---|---|
| MM1.5: Methods, Analysis & Insights from Multimodal LLM Fine-tuning (Read more on arXiv or HuggingFace) | nm-w, pdufter, zhegan27, fly6464, haotiz | a) This research aimed to improve multimodal large language model (MLLM) performance in text-rich image understanding, visual referring and grounding, and multi-image reasoning after pre-training. b) The researchers adopted a data-centric approach, focusing on continual pre-training with high-resolution OCR data, an optimized visual instruction-tuning data mixture for supervised fine-tuning (SFT), and dynamic image splitting for high-resolution image comprehension. c) MM1.5-30B significantly improved performance over its predecessor MM1-30B on tasks such as MathVista (increasing the score from 39.4 to 55.6), DocVQA (from 75.8 to 91.4), and InfoVQA (from 47.3 to 67.3). d) The paper demonstrates the importance of careful data curation and training strategies for improving MLLM performance, even at smaller scales, providing valuable guidance for practitioners developing and fine-tuning MLLMs. The impact of text-only pre-training data on MLLM performance, and how the proportion of such data in pre-training affects the efficiency of transfer learning to SFT is an impactful finding, suggesting that optimization of pre-training data is crucial for effective SFT. Follow-up Questions: 1. The paper mentions the use of in-house synthetic caption data that outperformed public datasets in some settings. Could the authors elaborate on the specific methodology used for generating these in-house captions, including the models, data sources, and any filtering or quality control mechanisms employed? 2. Given the findings on the impact of image resolution in continual pre-training, are there recommendations for optimal resolution ranges for different MLLM scales, considering the trade-off between performance and computational cost? 3. What specific techniques were used for optimizing the “optimized visual instruction-tuning data mixture” mentioned for SFT, and how was the final mixture composition determined? More specifically, how do you decide when the model is overfitting to the data? |
| DiaSynth – Synthetic Dialogue Generation Framework (Read more on arXiv or HuggingFace) | Eng Siong Chng, Tushar Pranav, AlexWuuuu, SkAndMl | a) The paper addresses the scarcity of high-quality, large-scale, domain-specific dialogue datasets for training dialogue systems. b) DiaSynth, a synthetic dialogue generation framework, uses Large Language Models (LLMs) and Chain of Thought (CoT) reasoning to generate dialogues based on user-provided topics, dynamically generated subtopics and personas, and specified conversational characteristics. c) Fine-tuning pretrained language models on synthetic data generated by DiaSynth resulted in a performance improvement of 16.47% compared to base models on a dialogue summarization task using LLaMA-3 as the LLM backbone. d) DiaSynth offers AI practitioners a scalable and cost-effective method for generating synthetic dialogue data for training dialogue systems, especially in domains with limited existing data. The results indicate that synthetic data from moderate-sized open-source LLMs can be a viable alternative to scarce or costly real-world data. Follow-up questions: 1. The paper mentions differing performance across LLMs (LLaMA-3, GPT-4) based on dialogue structure (formal vs. informal). Could further analysis elucidate the specific factors within these structures that influence LLM performance and inform optimal LLM selection for specific application domains? 2. While the paper demonstrates effectiveness in summarization, how does DiaSynth-generated data perform in other downstream tasks relevant to dialogue systems, such as intent detection, slot filling, or sentiment analysis? 3. What are the computational resource requirements and associated costs of using DiaSynth to generate large synthetic datasets, particularly when employing larger LLMs or generating data for diverse domains? |
| Ruler: A Model-Agnostic Method to Control Generated Length for Large Language Models (Read more on arXiv or HuggingFace) | yuelin bai, Ziqiang Liu, Yunshui Li, Lei Zhang, Jiaming Li | a) The research investigated the ability of Large Language Models (LLMs) to generate responses of specified lengths, introducing the Target Length Generation Task (TLG). b) A model-agnostic method named RULER, utilizing Meta Length Tokens (MLTs), was proposed and tested on several LLMs. RULER adds an MLT, indicating the desired length, to the input and trains LLMs end-to-end on a dataset augmented with MLTs. c) RULER improved the Flexible Match (FM) score, a measure of adherence to the target length range, by an average of 29.57 across all tested models and length levels. d) AI practitioners can use RULER to improve the control over output length in LLMs, enhancing their ability to adhere to specific length constraints in diverse applications. The paper does not address potential effects of RULER on other LLM performance metrics beyond those related to length control, nor its computational efficiency. Follow-up questions: 1. How does the performance of RULER vary with different training dataset sizes and compositions, particularly with respect to the distribution of target lengths? 2. What is the computational overhead of incorporating RULER, both during training and inference, compared to standard LLM usage? 3. Does RULER impact other performance metrics of the LLMs, such as factual accuracy, reasoning ability, or toxicity of generated text? |
| Hyper-Connections (Read more on arXiv or HuggingFace) | banggu, YunyaoMao, Taoer, hongzhihuang, mathfinder | a) This research explores hyper-connections as a learnable alternative to residual connections in neural networks, aiming to address limitations like the seesaw effect between gradient vanishing and representation collapse. b) Hyper-connections introduce learnable depth and width connections within layers, allowing the network to adjust connection strength and dynamically rearrange layers; a dynamic variant (DHC) conditions these connections on the input. c) In large language model pre-training, a model with DHC and an expansion rate of 4 (OLMOE-1B-7B-DHC×4) converged 1.8 times faster and showed a 6-point improvement on ARC-Challenge accuracy compared to a residual connection baseline after training on 500 billion tokens. d) AI practitioners can utilize hyper-connections as a potential drop-in replacement for residual connections, offering potential performance gains and faster convergence, particularly in large language models. The paper also suggests potential applicability in computer vision tasks, but the provided results are limited. Follow-up questions: 1. What is the computational overhead of hyper-connections compared to standard residual connections during both training and inference, especially for very deep networks? 2. How robust are the performance improvements of hyper-connections across different model architectures, datasets, and hyperparameter settings beyond those tested in the paper, particularly in vision tasks where less experimentation is presented? 3. The paper mentions that hyper-connections can learn to rearrange layers. Can further details be provided on how this rearrangement is analyzed and its specific impact on model behavior? |
| UniAff: A Unified Representation of Affordances for Tool Usage and Articulation with Vision-Language Models (Read more on arXiv or HuggingFace) | Ce Hao, Zhengkai Jiang, Xibin Yuan, Qiaojun Yu, SiyuanH | This research aims to improve robotic manipulation by creating a unified representation of affordances for both tools and articulated objects. The researchers developed UniAff, a multimodal large language model (MLLM) fine-tuned on a synthetic dataset of 1500 objects with labeled part-level 6D poses, manipulation types, and affordances. UniAff achieved a 56.9% improvement in IOU for detecting functional affordances of tools compared to ManipVQA. This work provides a new model and dataset for object-centric robotic manipulation, potentially improving the generalization of robotic manipulation tasks. It is unclear how the synthetic dataset generation generalizes to the real world or the computational cost of UniAff. Follow-up questions: 1. What are the specific architectural details of the Mixed Visual Encoder used in UniAff, and how were the different visual encoders (CLIP, DINOv2, Q-Former) combined? 2. What is the breakdown of the 19 articulated object categories and 12 tool categories in the synthetic dataset, and what are the specific real-world datasets used to create the synthetic data? 3. How does UniAff perform in real-world settings on a broader range of tasks and objects not represented in the current experimental setup? |
| Cottention: Linear Transformers With Cosine Attention (Read more on arXiv or HuggingFace) | Eric C. Larson, TrevorDohm, gmongaras | a) This paper introduces Cottention, a novel attention mechanism designed to address the quadratic memory complexity of softmax attention in transformers. b) Cottention replaces the softmax operation with cosine similarity and rearranges the attention equation to achieve linear memory complexity with respect to sequence length. A custom CUDA kernel was developed for efficient computation, and a learned scalar parameter was introduced to stabilize training. c) On the GLUE benchmark, a BERT model using Cottention achieved an average score of 81.8, compared to 83.1 for the softmax baseline. d) Cottention offers AI practitioners a more memory-efficient alternative to softmax attention, enabling the processing of longer sequences without significant performance degradation, as demonstrated by comparable results on the GLUE benchmark and perplexity on GPT-J language modelling tasks. The paper notes theoretical linear memory complexity with respect to sequence length but acknowledges a discrepancy between theoretical and observed memory usage related to input dimensionality, warranting further investigation. Follow-up Questions: 1. The paper mentions a discrepancy between the theoretical and empirical memory usage with respect to input dimensionality. What further investigations could be conducted to explain this discrepancy and potentially optimize memory usage further? 2. The custom CUDA kernel for Cottention is mentioned but not detailed extensively. What specific optimization strategies were employed in the kernel design, and how do they contribute to the efficiency gains observed? 3. How does the training time and computational cost of Cottention compare to Softmax and other linear attention methods, considering both the forward and backward passes, particularly for very long sequences? |
| Image Copy Detection for Diffusion Models (Read more on arXiv or HuggingFace) | Yi Yang, Zhentao Tan, Yifan Sun, WenhaoWang | a) The paper investigates how to detect content replication generated by diffusion models, introducing the task of Image Copy Detection for Diffusion Models (ICDiff). b) A new dataset, Diffusion-Replication (D-Rep), containing 40,000 image-replica pairs with six annotated replication levels, was created using Stable Diffusion V1.5 and LAION-Aesthetics V2 images. A novel method, PDF-Embedding, which converts replication levels to probability density functions and uses a set of learned vectors for each image, was proposed. c) PDF-Embedding outperformed protocol-driven methods and non-PDF methods on the D-Rep test set, achieving 56.3% in Pearson Correlation Coefficient (PCC) and 25.6% in Relative Deviation (RD) using an exponential PDF. d) AI practitioners developing diffusion models should consider integrating ICDiff methods like PDF-Embedding to assess and mitigate potential copyright infringement or unwanted replication of training data in generated images. The replication ratios of several well-known diffusion models against a large-scale gallery were found to range from 10% to 20%, indicating a significant practical need for such detection. Follow-up questions: 1. How does the computational cost and performance of PDF-Embedding scale with larger image databases and with more recent, higher-resolution diffusion models beyond Stable Diffusion V1.5? 2. Could the PDF-Embedding method be adapted or improved for detecting partial image replication, as opposed to full-image replication, within diffusion model outputs? 3. How robust is PDF-Embedding to adversarial attacks designed to evade copy detection in generated images? |
| Can Models Learn Skill Composition from Examples? (Read more on arXiv or HuggingFace) | Sanjeev Arora, Anirudh Goyal, Simran Kaur, Haoyu Zhao, dingliyu | This research investigates whether fine-tuning can improve compositional generalization in LLMs, specifically their ability to combine language skills in novel ways. The study fine-tuned LLaMA-2-13B-Chat and Mistral-7B-Instruct-v0.2 on a dataset generated by GPT-4, consisting of text samples exhibiting combinations of 1, 2, or 3 language skills. Results showed that fine-tuning on these examples improved the models’ ability to compose up to 5 held-out skills, with LLaMA-2-13B-Chat’s success rate for composing 3 held-out skills increasing from 4% to 37%. This suggests that models can learn a “meta-skill” of composition, generalizing beyond specific skill combinations seen during training. AI practitioners can leverage this finding by incorporating skill-rich (potentially synthetic) text data into training to improve the compositional capabilities of LLMs. Follow-up Questions: 1. What is the impact of varying the size and diversity of the training dataset (beyond the current 13,957 samples) on the compositional generalization performance? 2. How does this fine-tuning approach compare to other methods for improving compositional generalization, such as curriculum learning or specific architectural modifications? 3. Beyond the SKILL-MIX evaluation, how can this improved compositional ability be effectively applied to more complex, real-world NLP tasks, and what are the potential limitations in such applications? |
| Coffee-Gym: An Environment for Evaluating and Improving Natural Language Feedback on Erroneous Code (Read more on arXiv or HuggingFace) | Dongjin Kang, Yongho Song, Seungjun Moon, Taeyoon Kwon, Hyungjoo Chae | a) The research aims to improve open-source natural language feedback models for code editing by creating a reinforcement learning environment that better aligns feedback with code improvement. b) The authors developed COFFEE-GYM, comprising the COFFEE dataset of human code edits with pairwise feedback annotations and COFFEEEVAL, a unit-test-driven reward function, used with PPO and DPO reinforcement learning algorithms. c) Feedback models trained with COFFEE-GYM achieved a 13.4% improvement in Pass@1 accuracy on both HumanEvalFix and COFFEE-TEST compared to a baseline DeepSeekCoder-7B model without feedback. d) AI practitioners can utilize COFFEE-GYM and COFFEEEVAL to train open-source feedback models that generate helpful feedback for code editing, achieving performance comparable to closed-source models like GPT-4. The paper highlights the importance of pairwise feedback data and robust reward models in training effective feedback systems. Follow-up questions: 1. The paper mentions limitations regarding the scope of editing being focused on correctness, not efficiency or readability. How could COFFEE-GYM be extended to incorporate these additional aspects of code quality into the feedback and reward models? 2. How robust is COFFEEEVAL to the specific choice of code editor model used? Could using a weaker or stronger editor significantly impact the learned feedback model? Are there experiments or analyses planned to address this potential dependency? 3. While the paper demonstrates improved performance on specific benchmarks, how well does this generalize to real-world code editing scenarios in diverse programming languages and codebases beyond competitive programming and the provided test sets? |
| IDEAW: Robust Neural Audio Watermarking with Invertible Dual-Embedding (Read more on arXiv or HuggingFace) | Jianzong Wang, Jing Xiao, zhangxulong, Pechola | a) This paper aims to develop a robust neural audio watermarking model with efficient localization capabilities, addressing the limitations of existing methods regarding capacity, imperceptibility, and locating efficiency. b) The authors propose IDEAW, which employs a dual-stage invertible neural network (INN) to separately embed a locating code and a watermark message into the audio, along with a balance block to mitigate the asymmetry introduced by the attack layer during robustness training. c) IDEAW achieves higher capacity and comparable robustness under various attacks compared to baseline methods, demonstrating a signal-to-noise ratio (SNR) of 35.41 dB and accuracy of 99.44% when embedding a 56-bit payload (46-bit message + 10-bit locating code). The proposed dual-embedding strategy reduces localization time overhead by approximately 40-50% compared to existing methods. d) AI practitioners working on audio security and copyright protection can utilize IDEAW for robust and efficient watermark embedding and extraction, improving localization speed significantly compared to traditional approaches. Follow-up questions: 1. How does the performance of IDEAW vary across different audio genres and lengths, beyond the speech and music datasets used in the evaluation? 2. What is the computational complexity of IDEAW’s embedding and extraction processes, and how does it scale with increasing audio length or watermark payload size? 3. Could the dual-embedding strategy be extended to other watermarking domains, such as image or video, using similar invertible network architectures? |
Papers for 2024-09-30
| Title | Authors | Summary |
|---|---|---|
| MIO: A Foundation Model on Multimodal Tokens (Read more on arXiv or HuggingFace) | Jiaheng Liu, Wangchunshu Zhou, Chunpu Xu, King Zhu, Zekun Wang | MIO aims to develop an any-to-any multimodal foundation model capable of understanding and generating text, images, speech, and video. The methodology involves training on discrete multimodal tokens using a four-stage process: alignment pre-training, interleaved pre-training, speech-enhanced pre-training, and supervised fine-tuning on various tasks. On the SEED-Bench, MIO-Instruct achieves 54.4% MCQ accuracy. This model offers AI practitioners a unified framework for diverse multimodal tasks, including interleaved video-text generation and chain-of-visual-thought reasoning. The paper doesn’t provide details on the size of the training dataset. Follow-up Questions: 1. What specific architectures and hyperparameters were used for the different pre-training stages, and how were they determined? 2. Could you elaborate on the computational resources required for training and inference, and how these scale with model size? 3. What are the limitations of the current video generation capabilities, particularly regarding generating raw video data rather than frame sequences? |
| VPTQ: Extreme Low-bit Vector Post-Training Quantization for Large Language Models (Read more on arXiv or HuggingFace) | Li Lyna Zhang, Shengyu Ye, Jicheng Wen, Yifei Liu, yangwang92 | This paper explores extremely low-bit weight-only quantization for Large Language Models (LLMs) to reduce memory footprint and improve inference speed. The authors propose Vector Post-Training Quantization (VPTQ), leveraging second-order optimization and channel-independent quantization to minimize the impact of vector quantization on model accuracy. On LLaMA-2 7B, VPTQ at 2.02 bits achieves a WikiText2 perplexity of 6.13 and an average improvement of 1% on QA tasks compared to previous state-of-the-art. This method allows for substantial model compression and faster inference speeds without significant accuracy degradation, useful for deploying LLMs on resource-constrained devices. The paper doesn’t detail the computational cost of VPTQ compared to other methods like GPTQ aside from quoting inference throughput. Follow-up questions: 1. How does the memory bandwidth requirement of VPTQ during inference compare to GPTQ and other scalar quantization methods, given the need to load codebooks? 2. What is the detailed breakdown of the quantization algorithm execution time (10.4-18.6%) – which steps contribute most significantly, and how can these be further optimized? 3. The paper mentions layer-wise finetuning. What is the specific process and its impact on final model accuracy and quantization time compared to not finetuning or performing full finetuning? |
| Modulated Intervention Preference Optimization (MIPO): Keep the Easy, Refine the Difficult (Read more on arXiv or HuggingFace) | fetong | This research aimed to improve preference optimization for large language models (LLMs) by addressing the limitations of Direct Preference Optimization (DPO). The authors proposed Modulated Intervention Preference Optimization (MIPO), which modulates the influence of a reference model during training based on the alignment between the reference model and each preference pair, measured using differences in average log-likelihood. On AlpacaEval 2.0, MIPO achieved a 9.05% higher win-rate than DPO using Llama3-8B-Instruct and an 8.19% higher win-rate using Mistral-7B-Base. This suggests that MIPO can facilitate more effective alignment of LLMs with human preferences compared to DPO by focusing training effort on instances where the reference model needs more improvement. The paper does not discuss computational complexity differences between MIPO and DPO. Follow-up questions: 1. How does the computational cost of MIPO compare to DPO, considering the additional computation required to calculate and integrate the modulation factor q(K)? 2. Could the performance gains observed with MIPO on AlpacaEval 2.0 and MT-Bench generalize to other preference optimization tasks and datasets? 3. What are the practical considerations for selecting the hyperparameter β in MIPO, and is there a more principled approach to tuning this parameter beyond the empirical analysis presented? |
| MSI-Agent: Incorporating Multi-Scale Insight into Embodied Agents for Superior Planning and Decision-Making (Read more on arXiv or HuggingFace) | Guanting Dong, Che Jiang, Yihuai Gao, Biqing Qi, Dayuan Fu | a) This research aimed to improve the planning and decision-making abilities of Large Language Model (LLM)-based embodied agents by effectively summarizing and utilizing insights from prior experiences. b) The researchers developed a Multi-Scale Insight Agent (MSI-Agent) featuring an experience selector, insight generator, and insight selector to organize experiences into multi-scale insights (general, environment, and subtask) and selectively use these insights when prompting the LLM. c) MSI-Agent achieved a 12.70% success rate on in-domain data and 14.54% on out-of-domain data on the TEACh Trajectory from Dialogue (TfD) benchmark, outperforming existing baselines, including the HELPER and Expel agents. d) This research indicates AI practitioners can significantly enhance LLM-based agent performance in embodied tasks by using multi-scale insight summarization and selection, especially in domain adaptation scenarios. This is impactful as it provides a practical method for improving the robustness and generalizability of embodied agents across different environments and tasks. Here are some follow-up questions an AI practitioner might ask: 1. What is the computational overhead of generating and storing multi-scale insights, and how can this be optimized for real-time applications? 2. How does MSI-Agent perform on more complex embodied tasks with longer horizons and more diverse interaction objects? 3. Can the insights generated by MSI-Agent be transferred or adapted for use with different LLMs or embodied agent architectures? |
Papers for 2024-09-27
| Title | Authors | Summary |
|---|---|---|
| MaskLLM: Learnable Semi-Structured Sparsity for Large Language Models (Read more on arXiv or HuggingFace) | wxcTest, gheinrich, srvm, yinhongxu, Vinnnf | The authors present MaskLLM, a novel method for achieving semi-structured (N:M) sparsity in Large Language Models (LLMs) by formulating mask selection as a differentiable sampling process using Gumbel Softmax. This approach enables end-to-end training of sparsity masks on large-scale datasets, leading to superior performance compared to traditional one-shot pruning techniques. Experiments on various LLMs, including LLaMA-2 and GPT-3 variants, demonstrate that MaskLLM achieves state-of-the-art perplexity scores while enabling significant memory and computational savings. Notably, MaskLLM facilitates lossless compression for specific downstream tasks by learning specialized masks, and the authors introduce “Mask Prior,” a technique for efficient transfer learning of sparsity. This work holds significant practical implications for AI practitioners, offering a pathway to deploy more efficient and scalable LLMs in real-world applications with reduced resource requirements. |
| LLaVA-3D: A Simple yet Effective Pathway to Empowering LMMs with 3D-awareness (Read more on arXiv or HuggingFace) | Wenwei Zhang, XihuiLiu, Jiangmiao, taiwang, ChaimZhu | The paper introduces LLaVA-3D, a novel framework for efficiently adapting the 2D Large Multimodal Model (LMM) LLaVA for 3D scene understanding. This is achieved by introducing “3D Patches,” a representation that augments 2D image patch features with 3D positional embeddings, allowing LLaVA-3D to process and understand 3D scenes from multi-view images. Experimental results demonstrate that LLaVA-3D achieves state-of-the-art performance on various 3D benchmarks, including 3D question answering, captioning, and visual grounding, while maintaining strong 2D image understanding capabilities. This development presents a significant advancement for AI practitioners, particularly AI engineers and data scientists working with 3D vision and language tasks, by offering a practical and efficient method to empower LMMs with 3D-awareness. LLaVA-3D’s ability to perform complex 3D scene understanding tasks, along with its ease of use and integration with existing 2D models, makes it a valuable tool for developing applications in fields such as robotics, virtual reality, and augmented reality. |
| EMOVA: Empowering Language Models to See, Hear and Speak with Vivid Emotions (Read more on arXiv or HuggingFace) | vikyzeng2, 17day, zhili-liu, gyhdog, KaiChen1998 | This research paper presents EMOVA, an innovative omni-modal large language model that leverages a continuous vision encoder and a semantic-acoustic disentangled speech tokenizer to enable simultaneous alignment of visual, speech, and text modalities. The model employs a novel text-centric alignment strategy that uses text as a bridge to facilitate alignment without relying on scarce omni-modal image-text-speech data. This joint optimization method not only enhances vision-language and speech capabilities but also surpasses corresponding bi-modal counterparts. Remarkably, EMOVA achieves state-of-the-art performance on both vision-language and speech benchmarks while supporting spoken dialogue with controllable emotional expressions. For AI practitioners, EMOVA offers a robust framework for building omni-modal applications with real-time spoken dialogue and emotion control, paving the way for more versatile and expressive human-computer interactions. |
| Lotus: Diffusion-based Visual Foundation Model for High-quality Dense Prediction (Read more on arXiv or HuggingFace) | Leheng Li, Yixun Liang, Wei Yin, Jing He, haodongli | This research introduces Lotus, a diffusion-based visual foundation model for enhancing dense prediction tasks like depth and normal estimation. The authors identify limitations in existing diffusion models when applied to dense prediction, proposing a novel adaptation protocol that addresses these issues. By incorporating a single-step diffusion process and a “detail preserver”, Lotus achieves state-of-the-art performance on zero-shot depth and normal estimation tasks, surpassing previous models in accuracy and efficiency. This development is particularly relevant for AI practitioners working with limited data, as Lotus demonstrates superior performance with significantly less training data compared to other state-of-the-art models. This advancement allows for wider adoption and potential for practical applications like 3D reconstruction and robotics. |
| Discovering the Gems in Early Layers: Accelerating Long-Context LLMs with 1000x Input Token Reduction (Read more on arXiv or HuggingFace) | Shafiq Joty, Yingyu Liang, Xuan-Phi Nguyen, Zhenmei Shi, alvinming | The research presents GemFilter, a novel inference strategy to accelerate Large Language Model (LLM) inference with long context inputs, effectively addressing the bottleneck of high computational cost and latency. GemFilter leverages the observation that relevant information for a query is often identified within the early layers of an LLM. By using these early layers as filters, GemFilter selects and compresses input tokens, leading to a significant reduction in context length for subsequent LLM processing. Empirical evaluations demonstrate that GemFilter achieves a 2.4x speedup and a 30% reduction in GPU memory consumption compared to state-of-the-art methods. This approach offers a practical solution for AI engineers and data scientists to deploy and optimize LLMs for long-context tasks, especially when computational resources are limited. |
| Pixel-Space Post-Training of Latent Diffusion Models (Read more on arXiv or HuggingFace) | Felix Juefei-Xu, Ji Hou, Matthew Yu, Simran Motwani, Christina Zhang | This research paper proposes a novel approach to improve the quality of images generated by Latent Diffusion Models (LDMs) by incorporating a pixel-space loss function during the post-training phase. The authors argue that operating solely in the compressed latent space, as is typical for LDMs, can lead to loss of detail and artifacts in the generated images. By adding a pixel-space objective during fine-tuning, either supervised or preference-based, the model learns to better preserve high-frequency details, resulting in significantly enhanced visual quality and fewer flaws in the generated images. Experiments demonstrate the effectiveness of this approach on both DiT and U-Net based LDMs, showing significant improvements in visual appeal and reduction of visual flaws without compromising text alignment. This technique provides AI practitioners, particularly those working with image generation, a simple yet effective method to enhance the quality of images generated by LDMs without architectural modifications, potentially leading to higher fidelity and more realistic image synthesis. |
| Reducing the Footprint of Multi-Vector Retrieval with Minimal Performance Impact via Token Pooling (Read more on arXiv or HuggingFace) | Griffin Adams, Antoine Chaffin, Benjamin Clavié | This paper introduces TOKEN POOLING, a straightforward method to compress multi-vector retrieval models like ColBERT by clustering and averaging similar token representations. Evaluations across various datasets demonstrate that this approach can reduce the index size by 50% with negligible impact on retrieval performance, and up to 66% with minimal degradation. Notably, TOKEN POOLING seamlessly integrates with ColBERT’s quantization pipeline, further enhancing compression capabilities. This method is particularly relevant for practitioners working with large-scale retrieval systems, as it offers a practical means to substantially reduce storage and memory footprints without compromising accuracy. This is especially important for deployments where resource constraints are a concern, or when utilizing indexing methods that offer greater flexibility for data updates compared to those typically employed with large multi-vector indexes. |
| Disco4D: Disentangled 4D Human Generation and Animation from a Single Image (Read more on arXiv or HuggingFace) | Tianwei Zhang, Lei Yang, Zhongang Cai, Shuai Liu, Hui En Pang | Disco4D is a novel Gaussian Splatting framework that generates and animates 3D clothed human avatars from a single image. Disco4D separates the human body and clothing into distinct Gaussian models, leveraging the strengths of SMPL-X for body representation and Gaussian models for clothing variability. The framework uses diffusion models for 3D reconstruction enhancement, addressing the challenge of occluded parts. Disco4D outperforms existing methods in fidelity, disentanglement, and animation quality, evidenced by quantitative and qualitative benchmarks on standard datasets. Its ability to disentangle and manipulate clothing assets while maintaining high-fidelity 3D representation holds significant potential for various applications, including virtual try-on, avatar customization, and digital content creation. Practitioners working in these domains may find Disco4D to be a valuable tool for streamlining their workflows and enhancing the realism and customizability of their projects. |
| Robot See Robot Do: Imitating Articulated Object Manipulation with Monocular 4D Reconstruction (Read more on arXiv or HuggingFace) | Qianqian Wang, Brent Yi, Mingxuan Wu, Chung Min Kim, Justin Kerr | The authors propose a novel method, Robot See Robot Do (RSRD), to enable a robot to imitate articulated object manipulation from a single monocular video. The system leverages 4D Differentiable Part Models (4D-DPM) for 3D part motion recovery from monocular video and plans bimanual arm motions to induce the demonstrated object part motion. RSRD achieves an average of 87% success rate in each phase and 60% end-to-end success rate across 90 trials on 9 objects. This work demonstrates the viability of using pretrained vision models, without any task-specific training, to learn new manipulation skills for a robot. This could be a valuable tool for AI engineers and Data Scientists working on robotics applications to simplify the process of teaching new manipulation skills to robots. |
| Instruction Following without Instruction Tuning (Read more on arXiv or HuggingFace) | Christopher D. Manning, Percy Liang, Nelson F. Liu, John Hewitt | This research paper investigates instruction following in language models without explicit instruction tuning. The authors identify two implicit instruction tuning approaches: response tuning (training on responses only) and single-task fine-tuning (training on a narrow domain). Surprisingly, both approaches yield models capable of following general instructions, even surpassing base models in performance. This suggests that instruction-response mappings might be implicitly learned during pretraining, and seemingly unrelated fine-tuning tasks can implicitly enhance instruction-following capabilities. This finding holds practical relevance for practitioners, emphasizing the need for comprehensive testing and safety evaluations even for models fine-tuned for specific tasks, as they may exhibit unintended general instruction-following behavior. |
| Enhancing Structured-Data Retrieval with GraphRAG: Soccer Data Case Study (Read more on arXiv or HuggingFace) | Pål Halvorsen, Michael A. Riegler, Cise Midoglu, Sushant Gautam, Zahra Sepasdar | This paper presents Structured-GraphRAG, a novel framework designed to enhance information retrieval from structured datasets. Structured-GraphRAG leverages the power of Knowledge Graphs (KGs) and graph-based architectures to provide more accurate and efficient retrieval of data from structured sources. Experimental results demonstrate that Structured-GraphRAG outperforms traditional methods by reducing processing time, enhancing answer accuracy, and mitigating the issue of hallucinations in Language Models (LLMs). By offering a more accessible approach to KG construction, Structured-GraphRAG proves to be a valuable tool for AI engineers and data scientists working with structured data across diverse domains. |
Papers for 2024-09-26
| Title | Authors | Summary |
|---|---|---|
| Programming Every Example: Lifting Pre-training Data Quality like Experts at Scale (Read more on arXiv or HuggingFace) | Qian Liu, Pengfei, lockon, SinclairWang, koalazf99 | The paper introduces Programming Every Example (PROX), a novel framework for refining large-scale language model pre-training data by utilizing small language models to generate and execute data processing programs. PROX refines data through a two-stage process: document-level programming for filtering and chunk-level programming for fine-grained operations like string normalization. Experimental results demonstrate that PROX-curated data consistently enhances model performance, achieving a 2.1% average improvement over 10 downstream benchmarks and surpassing state-of-the-art data selection techniques by over 2.0%. Furthermore, PROX significantly reduces the required training tokens for comparable performance, offering up to 20x training efficiency improvements in certain domains. Practitioners, including AI engineers and data scientists, can leverage PROX to enhance data quality and significantly reduce training costs for large language models, making LLM development more efficient and accessible. |
| Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Multimodal Models (Read more on arXiv or HuggingFace) | Muennighoff, SMSD75, jamepark3922, sharpen, mattdeitke | The paper introduces Molmo, a family of open-weight and open-data vision-language models (VLMs) trained on a novel dataset named PixMo. Unlike previous open VLMs that relied heavily on synthetic data from proprietary systems, Molmo leverages a high-quality dataset of detailed image descriptions collected using a speech-based annotation approach. Evaluation on 11 academic benchmarks and human evaluation demonstrate that Molmo achieves state-of-the-art performance among open VLMs, even rivaling proprietary models like GPT-40. The release of Molmo’s weights, data, and code provides practitioners and researchers with valuable resources for building and studying performant VLMs from scratch. |
| Boosting Healthcare LLMs Through Retrieved Context (Read more on arXiv or HuggingFace) | Ashwin Kumar Gururajan, dariog, JordiBayarri | This research investigates the enhancement of open-source Large Language Models (LLMs) for medical question answering through optimized context retrieval techniques. The authors find that incorporating choice shuffling, an optimal number of ensembles, and enriching databases with Chain-of-Thought augmented examples significantly improves performance on multiple-choice question answering benchmarks, achieving accuracy comparable to private models like MedPalm-2 and GPT-4. They introduce OpenMedPrompt, a novel framework for open-ended medical question answering, with two strategies: Ensemble Refining (OM-ER) and Self-Reflection (OM-SR), demonstrating the effectiveness of iterative feedback and reward model integration. The study provides valuable insights for AI engineers and data scientists working on building accurate and reliable healthcare AI systems by showcasing the potential of open-source LLMs augmented with optimized context retrieval. |
| DreamWaltz-G: Expressive 3D Gaussian Avatars from Skeleton-Guided 2D Diffusion (Read more on arXiv or HuggingFace) | Lei Zhang, Zheng-Jun Zha, Jianan Wang, alkxncda, KevinHuang | The paper introduces DreamWaltz-G, a novel framework for generating animatable 3D avatars from text descriptions. It leverages pretrained 2D diffusion models and a novel Skeleton-guided Score Distillation (SkelSD) technique, enhancing 3D consistency and pose accuracy. DreamWaltz-G utilizes a hybrid 3D Gaussian representation (H3GA), integrating neural implicit fields and parameterized meshes for efficient rendering, optimization, and expressive animation. Experiments demonstrate superior generation and animation quality, outperforming existing methods. AI practitioners can utilize DreamWaltz-G for applications like character generation in gaming and virtual reality, benefiting from its text-driven approach, realistic animation, and efficient implementation. |
| Degradation-Guided One-Step Image Super-Resolution with Diffusion Priors (Read more on arXiv or HuggingFace) | Renjing Pei, Aiping Zhang, cxc361461518, Akowang, OAOA | The authors present S3Diff, a novel one-step image super-resolution (SR) model that leverages a pre-trained text-to-image (T2I) diffusion model. By incorporating degradation-guided Low-Rank Adaptation (LoRA), S3Diff efficiently adapts model parameters based on the degradation characteristics of low-resolution images, enhancing its efficiency and effectiveness. Experimental results demonstrate S3Diff’s superior performance in both synthetic and real-world scenarios, achieving state-of-the-art results with just one sampling step. This approach holds significant implications for practitioners, particularly AI engineers and data scientists working on image enhancement tasks, by offering a computationally efficient yet highly effective solution for super-resolution. The integration of degradation awareness further enhances the model’s practical applicability for real-world image restoration scenarios. |
| Game4Loc: A UAV Geo-Localization Benchmark from Game Data (Read more on arXiv or HuggingFace) | Liaoni Wu, Zhuoyue Tan, heboyong, Yux1ang | This paper introduces Game4Loc, a novel benchmark for UAV geo-localization based on data extracted from commercial video games. Game4Loc addresses the limitations of existing datasets, which primarily rely on perfectly aligned drone-satellite image pairs, by incorporating partial matching scenarios that better reflect real-world conditions. The authors propose weighted-InfoNCE, a contrastive learning approach that leverages intersection-over-union (IOU) as a supervisory signal to improve partial matching performance. Experimental results demonstrate the effectiveness of Game4Loc and the proposed training method, achieving state-of-the-art performance in both cross-area and same-area geo-localization tasks. This work provides AI engineers and data scientists with a valuable resource for developing and evaluating more robust and practical UAV geo-localization systems. |
| AIM 2024 Sparse Neural Rendering Challenge: Dataset and Benchmark (Read more on arXiv or HuggingFace) | Radu Timofte, Richard Shaw, sibicatleychandar, thomas-tanay, michaal94 | This research paper introduces SpaRe, a novel dataset and benchmark designed for evaluating sparse-view neural rendering. Existing datasets and protocols are shown to suffer from limitations like low-resolution evaluation and overfitting due to public test data. SpaRe addresses these issues with high-quality synthetic renderings, hidden test data, and diverse camera viewpoints. Through an online platform, SpaRe allows researchers to benchmark novel view synthesis methods in a standardized manner and contribute to a public leaderboard. Experimental results highlight the strengths and weaknesses of both per-scene optimization and generalizable methods for sparse neural rendering. Practitioners, such as AI engineers and data scientists, can leverage SpaRe to rigorously evaluate and compare the performance of new sparse-view neural rendering algorithms. |
| TalkinNeRF: Animatable Neural Fields for Full-Body Talking Humans (Read more on arXiv or HuggingFace) | Rakesh Ranjan, Amit Kumar, Bindita Chaudhuri, nsarafianos, aggelina | The authors introduce a novel framework, TalkinNeRF, that learns a dynamic neural radiance field for full-body talking humans from monocular videos. TalkinNeRF models the holistic 4D human motion, including body pose, hand articulation, and facial expressions. It introduces a multi-identity representation that enables simultaneous training for multiple subjects, significantly reducing training time. TalkinNeRF demonstrates state-of-the-art performance for animating full-body talking humans. This research is relevant to practitioners because it provides a new way to create high-fidelity animated videos of talking humans. This can be useful for various applications, such as virtual communication, video games, and movie production. |
Papers for 2024-09-25
| Title | Authors | Summary |
|---|---|---|
| HelloBench: Evaluating Long Text Generation Capabilities of Large Language Models (Read more on arXiv or HuggingFace) | Liqun He, Feiyu Duan, zsytony, zhangysk, quehry | The research paper “HelloBench: Evaluating Long Text Generation Capabilities of Large Language Models” introduces a novel benchmark designed to evaluate the long-form text generation capabilities of Large Language Models (LLMs). The benchmark, called HelloBench, is structured around Bloom’s Taxonomy and comprises five tasks: open-ended QA, summarization, chat, text completion, and heuristic text generation, encompassing a diverse range of 38 subcategories and 647 testing samples. To facilitate efficient evaluation, the authors propose a human-aligned evaluation method called HelloEval, which uses LLM-as-a-Judge and demonstrates superior correlation with human evaluation compared to traditional metrics. The key finding of the study is that current LLMs, despite advancements, demonstrate limitations in generating long-form text, often favoring shorter outputs or generating longer text with compromised quality. This research is relevant to practitioners such as AI engineers and data scientists, as it provides a standardized benchmark and evaluation method to guide the development and fine-tuning of LLMs for long-form text generation tasks, a critical area for real-world applications. |
| Making Text Embedders Few-Shot Learners (Read more on arXiv or HuggingFace) | Kun Luo, Jianlyu Chen, Shitao Xiao, MingHao Qin, cfli | This research paper proposes a novel approach called bge-en-icl that integrates in-context learning (ICL) with large language models (LLMs) to enhance the generation of text embeddings, enabling them to excel in both zero-shot and few-shot settings. The model achieves state-of-the-art performance on MTEB and AIR-Bench benchmarks without modifying the LLM architecture, relying instead on enriching the query prompt with task-specific examples. Findings suggest that retaining the original, unmodified architecture often yields the best results, highlighting the strength of ICL in adapting to new tasks without complex architectural alterations. Practitioners, such as AI engineers and data scientists, can leverage this model to build more versatile text embedding systems that can readily adapt to diverse scenarios without extensive fine-tuning, facilitating better performance in information retrieval, text classification, and other NLP tasks. |
| Present and Future Generalization of Synthetic Image Detectors (Read more on arXiv or HuggingFace) | Enrique Lopez-Cuena, dariog, pabberpe | This paper investigates the generalization capacity of synthetic image detectors amidst the rapid evolution of AI image generation models. The authors find that no single detector consistently outperforms others across diverse datasets and generative models, suggesting that universal detectors are presently elusive. Experiments demonstrate that training detectors on images generated by newer models enhances their ability to detect both old and new synthetic content. This highlights a race equilibrium effect where better generators lead to better detectors and vice-versa, emphasizing the need for continuous development and evaluation of detectors in this dynamic field. For practitioners, this research underscores the importance of using diverse training datasets, incorporating the latest generation models, and remaining cognizant of the limitations of current detectors when deploying them in real-world applications. |
| MonoFormer: One Transformer for Both Diffusion and Autoregression (Read more on arXiv or HuggingFace) | Errui Ding, Haocheng Feng, Wenhao Wang, Yuxing Song, Chuyang Zhao | The research paper “MonoFormer: One Transformer for Both Diffusion and Autoregression” introduces a novel approach to utilizing a single transformer for both autoregressive text generation and diffusion-based image generation. The authors leverage the similarities between transformer training for these two modalities, primarily differing in the attention mask employed, to achieve comparable performance in image generation to state-of-the-art methods, while retaining text generation capabilities. This is a significant development for practitioners as it offers a unified and potentially more efficient architecture for multi-modal tasks, simplifying development and potentially reducing computational overhead for AI engineers and data scientists working with text and image data. The demonstrated performance on ImageNet and commonsense reasoning benchmarks, along with ablation studies highlighting the importance of pretrained LLMs and bidirectional attention, underscores the potential of MonoFormer for advancing multi-modal learning. |
| MaskBit: Embedding-free Image Generation via Bit Tokens (Read more on arXiv or HuggingFace) | Xiaohui Shen, Xueqing Deng, Qihang Yu, Lijun Yu, Mark Weber | The authors propose MaskBit, a novel transformer-based image generation model that operates directly on bit tokens, eliminating the need for embedding tables typically found in VQGAN-based approaches. Through a systematic study, they modernize a widely-used VQGAN model, achieving state-of-the-art image reconstruction performance. They demonstrate that bit tokens, derived from binary quantization, exhibit a structured semantic representation, making them suitable for image generation. MaskBit achieves state-of-the-art performance on ImageNet 256x256 generation benchmark, surpassing prior art while using a compact generator. This work provides AI practitioners with an efficient and high-performing method for image generation, offering advantages in terms of computational cost and memory footprint due to the embedding-free design. |
| MIMO: Controllable Character Video Synthesis with Spatial Decomposed Modeling (Read more on arXiv or HuggingFace) | Liefeng Bo, Miaomiao Cui, Yuan Yao, Yifang Men | The paper proposes MIMO, a novel framework for controllable character video synthesis that leverages spatial decomposition modeling for enhanced control and realism. MIMO uniquely decomposes video clips into spatially distinct components - human, scene, and occlusion - which are encoded into latent codes and fed into a diffusion-based decoder for video reconstruction. This approach allows for flexible manipulation of character appearance, motion, and scene interaction through user-provided inputs like images and pose sequences. The key result is the ability to generate high-fidelity character videos with complex 3D motions and realistic object interactions. MIMO presents a powerful tool for AI engineers and data scientists in domains like animation, virtual reality, and video editing, enabling them to synthesize and manipulate character-driven videos with unprecedented control and realism. |
| EuroLLM: Multilingual Language Models for Europe (Read more on arXiv or HuggingFace) | Ricardo Rei, Nuno M. Guerreiro, João Alves, Patrick Fernandes, Pedro Henrique Martins | The authors introduce EuroLLM, a project focused on developing multilingual language models (LLMs) proficient in all official European Union languages and several other relevant languages. The researchers meticulously constructed a massive multilingual dataset, developed a custom tokenizer, and explored different modeling and pre-training configurations based on scaling laws. Their initial models, EuroLLM-1.7B and EuroLLM-1.7B-Instruct, demonstrate strong performance on multilingual benchmarks and machine translation tasks. Notably, EuroLLM-1.7B-Instruct exhibits superior performance in machine translation across various language pairs compared to existing models with significantly larger parameter sizes, highlighting its efficacy for multilingual NLP applications. This work holds significant implications for AI practitioners, particularly those working on multilingual natural language processing tasks, as it offers a robust foundation and valuable resources for developing and deploying LLMs for a wide range of European languages. |
| Gen2Act: Human Video Generation in Novel Scenarios enables Generalizable Robot Manipulation (Read more on arXiv or HuggingFace) | Carl Doersch, Shubham Tulsiani, Abhinav Gupta, Debidatta Dwibedi, Homanga Bharadhwaj | The paper “Gen2Act: Human Video Generation in Novel Scenarios enables Generalizable Robot Manipulation” introduces a novel framework for generalizable robot manipulation that leverages zero-shot human video generation from web data and limited robot demonstrations. Gen2Act addresses the challenge of generalizing to unseen scenarios, objects, and motions by first generating a human video of the desired task using a pre-trained video generation model. A closed-loop policy then translates this video into robot actions, implicitly learning motion cues from the generated human behavior. Evaluations show Gen2Act significantly outperforms baselines in generalization tasks, especially to unseen object types and motion types. This framework holds significant potential for AI practitioners, particularly in robotics, by offering a scalable and efficient way to develop robot manipulation policies that generalize to new tasks and environments without the need for extensive robot data collection. |
| Seeing Faces in Things: A Model and Dataset for Pareidolia (Read more on arXiv or HuggingFace) | Jennifer Corbett, Anne Harrington, Vasha DuTell, Simon Stent, mhamilton723 | The paper, “Seeing Faces in Things: A Model and Dataset for Pareidolia”, by Corbett, Harrington, DuTell, et al. explores the phenomenon of face pareidolia – seeing faces in random stimuli – from a computer vision perspective. The authors introduce “Faces in Things”, a novel dataset of 5,000 annotated pareidolic face images, and demonstrate that a state-of-the-art face detector, while excelling at detecting human faces, struggles with pareidolic ones. Interestingly, fine-tuning the detector on animal faces significantly improves pareidolic face detection, suggesting a link between the perception of animal and pareidolic faces. This work provides valuable insights for AI practitioners, particularly those working on face detection, by highlighting the limitations of current models and suggesting avenues for improvement, such as incorporating training data that reflects the diversity of features present in both animal and pareidolic faces. Understanding pareidolia could lead to more robust face detectors, minimizing false positives and potentially enhancing visual attention mechanisms in AI systems. |
| DynaMo: In-Domain Dynamics Pretraining for Visuo-Motor Control (Read more on arXiv or HuggingFace) | Lerrel Pinto, Siddhant Haldar, Aadhithya Iyer, Hengkai Pan, Zichen Jeff Cui | DynaMo is a novel self-supervised learning method for pretraining visual representations for visuomotor control tasks. DynaMo operates by jointly learning an image encoder alongside inverse and forward dynamics models from unlabeled, sequential visual demonstrations, without relying on data augmentation or contrastive learning. Experiments demonstrate that DynaMo outperforms existing self-supervised methods and pretrained representations on both simulated and real-world robotic manipulation benchmarks. This approach is particularly relevant for AI engineers and roboticists working with limited demonstration data, as it offers a data-efficient method for learning robust visual representations for robot control. The authors posit that the method’s efficacy stems from its ability to leverage the inherent temporal structure in demonstrations, enabling it to learn task-specific features more effectively. |
| Reward-Robust RLHF in LLMs (Read more on arXiv or HuggingFace) | Jian Xie, Yiping Zhang, Jialian Li, Xingzhou Lou, Yuzi Yan | The authors introduce a novel reward-robust RLHF (Reinforcement Learning from Human Feedback) framework to enhance the alignment of LLMs (Large Language Models) with human preferences while addressing limitations in reward modeling. The proposed framework employs Bayesian Reward Model Ensembles (BRME) to capture the uncertainty inherent in reward signals and uses a trade-off objective function that balances performance and robustness during optimization. Empirical evaluations across diverse benchmarks show that the framework consistently outperforms traditional RLHF, demonstrating improved stability and accuracy, especially in long-term training. This approach is particularly relevant for AI practitioners as it tackles the crucial challenge of reward hacking, where LLMs exploit imperfections in reward models, leading to suboptimal performance. By incorporating the proposed reward-robust framework, AI engineers and data scientists can develop LLMs that are more reliable, generalize better, and are less susceptible to unintended behaviors. |
| SLIMER-IT: Zero-Shot NER on Italian Language (Read more on arXiv or HuggingFace) | Andrea Zugarini, Marco Maggini, Leonardo Rigutini, Andrew Zamai | This research proposes SLIMER-IT, a novel approach for zero-shot Named Entity Recognition (NER) in Italian, addressing the scarcity of resources and research for this language, particularly for non-standard domains and entity types. SLIMER-IT, adapting the English SLIMER model, employs instruction tuning with prompts enriched by entity definitions and annotation guidelines, enabling superior performance on unseen entity tags. Experiments demonstrate SLIMER-IT’s effectiveness on a newly defined zero-shot NER benchmark for Italian, outperforming existing methods, especially in identifying previously unseen entities. This work holds practical implications for AI practitioners working with Italian language data, offering an effective tool for tasks like information extraction, question answering, and knowledge base construction, even with limited annotated data. Future work will focus on extending the benchmark and improving scalability for larger label sets. |
| Time-MoE: Billion-Scale Time Series Foundation Models with Mixture of Experts (Read more on arXiv or HuggingFace) | Zhou Ye, Dianqi Li, Yuqi Nie, Shiyu Wang, Xiaoming Shi | The paper introduces Time-MoE, a novel decoder-only transformer architecture with a Mixture-of-Experts (MoE) design specifically tailored for large-scale time series forecasting. This architecture enables Time-MoE to scale to 2.4 billion parameters while maintaining computational efficiency by activating only a subset of networks for each prediction. Trained on Time-300B, a newly introduced dataset comprising over 300 billion time points across 9 domains, Time-MoE significantly outperforms existing forecasting models on six benchmarks in both zero-shot and fine-tuned settings. The results validate the scaling laws for training tokens and model size in time series forecasting, demonstrating superior performance compared to dense models with equivalent computational budgets. This work offers practitioners a powerful, efficient, and flexible solution for real-world time series forecasting, allowing them to develop and deploy larger, more capable models with reduced computational costs. |
| Tabular Data Generation using Binary Diffusion (Read more on arXiv or HuggingFace) | Slava Voloshynovskiy, vitaliykinakh | Voloshynovskiy and Kinakh introduce Binary Diffusion, a novel generative model for synthetic tabular data generation. Their method leverages a lossless binary transformation to convert tabular data into fixed-size binary representations, simplifying preprocessing. The Binary Diffusion model then employs XOR operations for efficient noise addition and removal, addressing challenges posed by mixed data types and complex distributions inherent in tabular data. Evaluations on benchmark datasets demonstrate that Binary Diffusion achieves state-of-the-art performance, notably surpassing existing methods on Travel, Adult Income, and Diabetes datasets. Furthermore, its compact size and efficient training make it a practical tool for practitioners, especially in scenarios with limited data or privacy concerns. |
Papers for 2024-09-24
| Title | Authors | Summary |
|---|---|---|
| RACER: Rich Language-Guided Failure Recovery Policies for Imitation Learning (Read more on arXiv or HuggingFace) | Joyce Chai, nimafazeli, newwater, Yinpei | This paper introduces RACER, a novel framework for enhancing robotic manipulation through the integration of rich language guidance and failure recovery mechanisms. The authors propose a data augmentation pipeline that automatically generates failure recovery trajectories and annotates them with detailed language instructions, addressing the limitations of existing benchmarks. Experimental results on RLBench demonstrate that RACER outperforms state-of-the-art baselines in multi-task learning, dynamic goal change scenarios, and zero-shot unseen task evaluations. Notably, RACER exhibits superior sim-to-real transfer capabilities, highlighting the practical significance of rich language guidance for real-world robotic deployments. This research provides AI practitioners, particularly those in robotics, with valuable insights and a practical framework for developing more robust and adaptable manipulation policies. |
| A Preliminary Study of o1 in Medicine: Are We Closer to an AI Doctor? (Read more on arXiv or HuggingFace) | Haoqin Tu, Juncheng Wu, Yunfei Xie, ys-zong, tennant | This research paper presents a comprehensive evaluation of OpenAI’s o1 language model within the medical domain, focusing on its understanding, reasoning, and multilingual capabilities across 37 datasets. The study reveals that o1 exhibits enhanced clinical understanding and reasoning abilities, surpassing prior models like GPT-4 in diagnostic accuracy on several tasks. Notably, o1 demonstrates significant improvements in challenging medical question-answering scenarios and medical calculation tasks. However, limitations persist in terms of hallucination and complex multilingual reasoning, suggesting areas for further development. These findings are highly relevant to AI practitioners, particularly those developing AI-driven healthcare solutions, as they highlight both the potential and current limitations of utilizing large language models for medical applications. |
| PixWizard: Versatile Image-to-Image Visual Assistant with Open-Language Instructions (Read more on arXiv or HuggingFace) | Renrui Zhang, Xinyu Wei, SiyuanH, stzhao, Afeng-x | PixWizard is a Diffusion Transformer-based image-to-image visual assistant that leverages a novel 30-million datapoint “Omni Pixel-to-Pixel Instruction-Tuning Dataset” to unify a variety of image editing, generation, and translation tasks. PixWizard demonstrates competitive performance in tasks like image restoration, image grounding, and text-to-image generation, surpassing existing unified methods and approaching the performance of specialized models on some tasks. Notably, PixWizard achieves state-of-the-art results in image outpainting and demonstrates strong generalization to tasks like object removal and replacement, even when not explicitly trained on them. AI practitioners can utilize PixWizard as a flexible tool for various image-related tasks, and the introduced dataset and training strategies can be adapted for other text-to-image diffusion models. |
| Beyond Fine-tuning: Unleashing the Potential of Continuous Pretraining for Clinical LLMs (Read more on arXiv or HuggingFace) | Muhammad Umar Salman, Svetlana Maslenkova, Tathagata Raha, pkanithi, cchristophe | The study investigates the efficacy of continuous pretraining on in-domain clinical data in conjunction with instruction fine-tuning and advanced prompting for optimizing Large Language Models (LLMs) in clinical question-answering tasks. While continuous pretraining yields marginal improvements compared to other techniques, it establishes a valuable foundation for enhancing LLM performance in the clinical domain by mitigating instability issues through careful balancing of in-domain data with general language data. The synergy between continuous pretraining, instruct fine-tuning, and complex prompting techniques, specifically MedPrompt, results in state-of-the-art performance on a variety of clinical QA benchmarks. These findings are particularly relevant for AI engineers and data scientists working on adapting LLMs for clinical applications, highlighting the effectiveness of continuous pretraining as a foundational step for improving model accuracy and reasoning ability in this domain. |
| Phantom of Latent for Large Language and Vision Models (Read more on arXiv or HuggingFace) | Yong Man Ro, Beomchan Park, Sangyun Chung, chae-won-kim, BK-Lee | The paper introduces Phantom, an efficient family of large language and vision models (LLVMs) that enhances learning capabilities within limited model sizes. Phantom temporarily increases the latent hidden dimension during multi-head self-attention (MHSA), allowing it to embed more vision-language knowledge without significantly increasing physical model size. The authors also introduce Phantom Optimization (PO), a novel training strategy inspired by Direct Preference Optimization, which guides the model towards correct answers while minimizing incorrect and ambiguous ones. Experiments demonstrate that Phantom outperforms numerous larger open- and closed-source LLVMs across various vision-language benchmarks. This is highly relevant to practitioners, particularly AI engineers and data scientists, who seek to develop and deploy efficient yet high-performing LLVMs for resource-constrained environments, such as mobile devices and embedded systems. By demonstrating the effectiveness of latent space optimization in enhancing LLVMs, the paper provides valuable insights for designing and training future efficient multimodal models. |
| An adapted large language model facilitates multiple medical tasks in diabetes care (Read more on arXiv or HuggingFace) | Yutong Chen, Muyang He, Zhen Ying, weiranhuang, WaltonFuture | The research paper, “An adapted large language model facilitates multiple medical tasks in diabetes care,” by Chen, He, Ying, et al. introduces Diabetica, a diabetes-specific large language model (LLM) family fine-tuned from the open-source Qwen2 model. The authors curated a specialized dataset and developed benchmarks for multiple-choice questions, fill-in-the-blank tasks, and open-ended dialogues to rigorously evaluate the model’s performance. Diabetica demonstrated state-of-the-art performance in understanding and executing diabetes-related tasks, surpassing open-source LLMs of comparable size and rivaling proprietary models like GPT-4 and Claude-3.5. Clinical evaluations highlight Diabetica’s potential in patient consulting, medical education, and clinical record summarization. This research offers a practical framework for developing and evaluating domain-specific LLMs, which is highly relevant to AI engineers and data scientists interested in healthcare applications. |
| MaterialFusion: Enhancing Inverse Rendering with Material Diffusion Priors (Read more on arXiv or HuggingFace) | Rushikesh Zawar, Aviral Agrawal, Kangle Deng, Or Patashnik, Yehonathan Litman | The paper introduces MaterialFusion, a novel inverse rendering approach that leverages a 2D material diffusion prior, called StableMaterial, to enhance the reconstruction of an object’s 3D representation, including geometry, materials, and illumination, from a set of multi-view images. StableMaterial is trained on a vast dataset of synthetic objects with high-quality Physically Based Rendering (PBR) assets, enabling it to learn a prior over plausible material and albedo combinations. Experimental results demonstrate that MaterialFusion surpasses state-of-the-art inverse rendering methods in reconstructing faithful material properties and accurately relighting objects under novel illumination conditions. This work holds significant implications for practitioners in computer graphics and vision, including AI engineers and data scientists, by providing a robust method for 3D object reconstruction and relighting, which can be applied in various domains like virtual reality, augmented reality, and content creation. |
| Zero-shot Cross-lingual Voice Transfer for TTS (Read more on arXiv or HuggingFace) | Gary Wang, Kyle Kastner, Isaac Elias, Youzheng Chen, Fadi Biadsy | This paper introduces a novel zero-shot voice transfer (VT) module for multilingual text-to-speech (TTS) systems, capable of transferring an individual’s voice across languages using a single short reference utterance. The module comprises a speaker encoder, a bottleneck layer (with SegmentGST shown most effective for typical speech), and residual adapters integrated into a pre-existing TTS system. Evaluations demonstrate an average voice transfer similarity score of 73% across nine languages, even with atypical reference speech. This research is highly relevant for AI practitioners developing accessible TTS systems or voice restoration technologies, enabling high-quality, cross-lingual voice transfer and offering potential benefits to individuals with speech impairments. |
| MaskedMimic: Unified Physics-Based Character Control Through Masked Motion Inpainting (Read more on arXiv or HuggingFace) | Xue Bin Peng, Ofir Nabati, Yunrong Guo, Chen Tessler, galchechik | The research paper, “MaskedMimic: Unified Physics-Based Character Control Through Masked Motion Inpainting,” introduces a novel framework for controlling physically simulated humanoid characters by leveraging a motion inpainting approach. MaskedMimic is trained on a diverse dataset of motion capture data with various modalities, including joint positions, text descriptions, and object interactions, where portions of the input data are strategically masked out. This forces the model to learn a general understanding of generating realistic and diverse human motions from partial information. The authors demonstrate that a single unified control architecture trained with this approach can successfully perform various tasks like locomotion, object interaction, VR tracking, and even text-to-motion synthesis without requiring task-specific training or reward engineering. Practitioners, including AI engineers and data scientists working in character animation and robotics, can benefit from this framework by having a simplified and flexible tool to create versatile and interactive virtual characters. |
| Self-Supervised Audio-Visual Soundscape Stylization (Read more on arXiv or HuggingFace) | Gopala Anumanchipalli, Andrew Owens, Po-Yao Huang, Renhao Wang, Tingle Li | This paper introduces the concept of audio-visual soundscape stylization, a technique to modify input audio to reflect the acoustic and ambient properties of a target scene represented by an audio-visual sample. The authors propose a self-supervised learning framework based on conditional speech de-enhancement using a latent diffusion model trained on unlabeled, in-the-wild videos. Extensive experiments demonstrate the model’s superiority over existing audio stylization methods in replicating acoustic properties and ambient sounds. This technique holds significant potential for practitioners, such as AI engineers and data scientists, in applications like realistic audio dubbing for videos, generating immersive virtual environments, and enhancing audio quality in old recordings. |
| A Case Study of Web App Coding with OpenAI Reasoning Models (Read more on arXiv or HuggingFace) | onekq | This paper presents a case study evaluating OpenAI’s latest reasoning models (o1-preview and o1-mini) on web application coding tasks. While demonstrating superior performance on the single-task WebApp1K benchmark, the models exhibit significant decline in the harder WebApp1K-Duo benchmark, falling behind Claude 3.5. The authors attribute this variability to instruction comprehension, where the reasoning mechanism, while beneficial with complete expectations, exacerbates errors when key expectations are missed. A key insight for practitioners, such as AI engineers and data scientists, is that the success of reasoning models in coding hinges not only on their reasoning capabilities but also on a robust base model and meticulous adherence to instructions, achieved through methods like SFT. This highlights the importance of focusing on both reasoning and instruction following when developing and deploying AI models for coding applications. |
Papers for 2024-09-23
| Title | Authors | Summary |
|---|---|---|
| Imagine yourself: Tuning-Free Personalized Image Generation (Read more on arXiv or HuggingFace) | anmolkalia, ankit61, haoyum1997, FelixXu, zechengh | The research paper “Imagine yourself: Tuning-Free Personalized Image Generation” by anmolkalia et al. introduces a novel diffusion-based model for personalized image generation that does not require subject-specific fine-tuning. The authors achieve this by incorporating three key components: a synthetic paired data generation mechanism to encourage image diversity, a fully parallel attention architecture with multiple text encoders and a trainable vision encoder for enhanced text alignment and identity preservation, and a coarse-to-fine multi-stage fine-tuning methodology for improved visual quality. Extensive human evaluation demonstrates that Imagine yourself significantly outperforms state-of-the-art personalization models in identity preservation, text alignment, and visual appeal. This tuning-free approach is particularly relevant to AI practitioners, such as AI Engineers and Data Scientists, as it enables the development of personalized image generation applications without the need for costly and time-consuming individual user tuning. |
| MuCodec: Ultra Low-Bitrate Music Codec (Read more on arXiv or HuggingFace) | Jianwei Yu, zy001, lglg666, hangtingchen, yaoxunxu | MuCodec is a novel neural codec designed for high-fidelity music reconstruction at ultra-low bitrates. This model leverages a specialized feature extractor, MuEncoder, to capture both acoustic and semantic features from music. These features are then discretized and reconstructed using a flow-matching-based method with a Diffusion Transformer. Experimental results demonstrate that MuCodec surpasses current state-of-the-art methods in both objective and subjective evaluations, achieving high-quality music reconstruction at bitrates as low as 0.35kbps. This development is particularly relevant for AI practitioners working on music information retrieval, music generation, and low-bitrate audio streaming applications. MuCodec offers a promising solution for compressing and reconstructing music with high fidelity, potentially leading to more efficient storage and transmission of music data. |
| Prithvi WxC: Foundation Model for Weather and Climate (Read more on arXiv or HuggingFace) | jubeku, ds6574, jhnnsjkbk, WillTrojak, johannesschmude | The paper introduces Prithvi WxC, a 2.3 billion parameter foundation model for weather and climate applications trained on the MERRA-2 reanalysis dataset. The model leverages a novel transformer-based architecture that incorporates both local and global attention mechanisms, and is trained using a combination of masked reconstruction and forecasting objectives. Zero-shot evaluations demonstrate Prithvi WxC’s ability to generate accurate short-term forecasts and reconstruct atmospheric states from heavily masked inputs. Fine-tuning experiments on downscaling and gravity wave flux parameterization further highlight the model’s versatility and ability to be adapted for diverse downstream tasks, suggesting potential benefits for AI engineers and data scientists working in climate modeling and weather forecasting applications. |
| Portrait Video Editing Empowered by Multimodal Generative Priors (Read more on arXiv or HuggingFace) | Yudong Guo, Chenglai Zhong, Haiyao Xiao, Xuan Gao, sisyphe28 | The paper introduces PortraitGen, a novel method for consistent and expressive portrait video editing using multimodal prompts. PortraitGen leverages 3D Gaussian Splatting embedded on SMPL-X models to ensure structural and temporal coherence, achieving rendering speeds of over 100FPS through a Neural Gaussian Texture mechanism. The system incorporates expression similarity guidance and a face-aware portrait editing module to mitigate degradation commonly associated with iterative dataset updates in existing methods. Experiments demonstrate superior quality and efficiency compared to state-of-the-art techniques across text-driven editing, image-driven editing, and relighting tasks. Practitioners, including AI Engineers and Data Scientists, can utilize PortraitGen to develop robust and high-fidelity portrait video editing tools for various applications. |
| Colorful Diffuse Intrinsic Image Decomposition in the Wild (Read more on arXiv or HuggingFace) | Yağız Aksoy, ccareaga | This research introduces a novel method for intrinsic image decomposition in the wild, successfully separating diffuse and non-diffuse lighting effects at high resolutions. The authors achieve this by decomposing the complex problem into physically-motivated sub-tasks, addressing the limitations of previous grayscale shading models. Quantitative analysis and qualitative examples demonstrate the method’s ability to generalize to diverse scenes, including outdoor landscapes and human faces, despite training the final diffuse network solely on a synthetic indoor dataset. This advancement allows for new illumination-aware image editing applications, offering AI practitioners robust tools for specularity removal and multi-illuminant white balancing in real-world images. |
| Temporally Aligned Audio for Video with Autoregression (Read more on arXiv or HuggingFace) | erahtu, bilpo, bilpo | This paper introduces V-AURA, a novel autoregressive model for video-to-audio generation that prioritizes temporal alignment and semantic relevance. Unlike diffusion-based counterparts, V-AURA utilizes a high-framerate visual feature extractor and a cross-modal fusion strategy to capture fine-grained audio-visual correspondences. Furthermore, the authors present VisualSound, a curated dataset with strong audio-visual relevance, to improve training efficiency and mitigate hallucinations. Evaluations demonstrate that V-AURA outperforms state-of-the-art methods in temporal alignment and relevance while maintaining competitive audio quality. These findings are particularly valuable for AI practitioners working on applications requiring tightly synchronized and semantically meaningful audio generation from video content, such as in video editing and multimedia content creation. |
| V^3: Viewing Volumetric Videos on Mobiles via Streamable 2D Dynamic Gaussians (Read more on arXiv or HuggingFace) | Zhirui Zhang, wuminye, Daluuu, liaowang11, Penghowdy | The paper proposes V³, a method for streaming and rendering high-quality volumetric videos on mobile devices using dynamic 3D Gaussian splats (3DGS). V³ leverages a compact 2D representation of 3DGS, allowing for efficient compression with video codecs and streaming to mobile devices. Their approach employs a novel two-stage training strategy with motion-appearance disentanglement, residual entropy loss, and temporal loss, enabling high-quality rendering while maintaining temporal consistency. Experimental results demonstrate that V³ outperforms existing methods in terms of rendering quality and storage efficiency. This breakthrough holds significant implications for practitioners in computer graphics and AI, particularly for AI engineers and data scientists working on efficient representations of 3D scenes and real-time rendering applications on resource-constrained devices. |
| Minstrel: Structural Prompt Generation with Multi-Agents Coordination for Non-AI Experts (Read more on arXiv or HuggingFace) | Daling Wang, Yijie Huang, Xiaoyu Liang, Yuanzhong Liu, Ming Wang | This research paper introduces LangGPT, a novel structured prompt framework designed to enhance the usability and effectiveness of Large Language Models (LLMs) for non-AI experts. LangGPT draws inspiration from programming language principles to establish a systematic, reusable, and extensible prompt structure, reducing the learning curve associated with prompt engineering. To further facilitate the prompt generation process, the authors propose Minstrel, a multi-agent system that automates the creation and optimization of LangGPT prompts through collaborative analysis, design, and reflection mechanisms. Experimental results demonstrate that both manually crafted and Minstrel-generated LangGPT prompts yield superior performance compared to conventional baseline prompts in various tasks, including question answering and instruction following. This framework holds significant practical implications for AI practitioners, enabling them to leverage a standardized and intuitive approach to harness the capabilities of LLMs effectively. |
Papers for 2024-09-20
| Title | Authors | Summary |
|---|---|---|
| InfiMM-WebMath-40B: Advancing Multimodal Pre-Training for Enhanced Mathematical Reasoning (Read more on arXiv or HuggingFace) | Yi-Qi638, lllliuhhhhggg, bytehxf, yjian-bytedance, xiaotianhan | The research paper introduces InfiMM-WebMath-40B, a large-scale, open-source dataset designed for the pre-training of Multimodal Large Language Models (MLLMs) specifically for enhanced mathematical reasoning. This dataset addresses a critical gap in the open-source community, which has previously lacked access to large, high-quality, multimodal math datasets. InfiMM-WebMath-40B consists of 24 million mathematics and science-related web documents, encompassing 40 billion text tokens and 85 million image URLs, all meticulously filtered and aligned from CommonCrawl. The authors detail the comprehensive data curation pipeline, highlighting the challenges associated with extracting and filtering mathematical content from web pages, including the development of specialized tools to handle mathematical equations and image URLs. Evaluations conducted on established benchmarks such as MathVerse and We-Math demonstrate that models pre-trained on InfiMM-WebMath-40B achieve state-of-the-art performance among open-source models, and even surpass some proprietary models on certain tasks. These findings hold significant implications for practitioners, including AI engineers and data scientists, as they now have access to a valuable resource for developing and refining MLLMs with superior mathematical reasoning capabilities. The availability of InfiMM-WebMath-40B is expected to accelerate progress in the field of multimodal mathematical reasoning and enable the development of more sophisticated and accurate MLLMs capable of tackling complex mathematical problems. |
| Training Language Models to Self-Correct via Reinforcement Learning (Read more on arXiv or HuggingFace) | sandraorion, ferya, shrivasd, rishabhagarwal, aviralkumar | This research paper introduces SCoRe, a novel multi-turn reinforcement learning approach designed to enhance the self-correction capabilities of large language models (LLMs). The authors demonstrate that traditional supervised fine-tuning methods are inadequate for this purpose, as they often lead to either minimal or detrimental modifications. SCoRe addresses these challenges through a two-stage training process: an initialization phase to expand the model’s self-correction repertoire and a reward shaping mechanism to incentivize effective self-correction during multi-turn RL. Evaluations on math and code generation benchmarks reveal that SCoRe significantly improves the model’s ability to rectify errors in its initial responses. This work provides AI practitioners, including AI engineers and data scientists, with a practical method to augment the reliability and accuracy of LLMs, particularly in tasks demanding high-fidelity outputs. |
| MMSearch: Benchmarking the Potential of Large Models as Multi-modal Search Engines (Read more on arXiv or HuggingFace) | lovesnowbest, lupantech, jyjyjyjy, ZiyuG, CaraJ | The paper “MMSearch: Benchmarking the Potential of Large Models as Multi-modal Search Engines” introduces a novel framework, MMSearch-Engine, designed to empower large language models (LLMs) with multi-modal search capabilities. The authors also present MMSearch, a comprehensive benchmark to evaluate the multi-modal search performance of LLMs, comprised of 300 manually collected instances across 14 subfields. Experimental results demonstrate that state-of-the-art LLMs, specifically GPT-4, achieve the best results on MMSearch, surpassing even commercial AI search engines in end-to-end task performance. However, error analysis reveals persistent challenges in requery and rerank capabilities, particularly for open-source LLMs, highlighting the need for further development in these areas. This work provides valuable insights for AI engineers and data scientists working on multi-modal search engines, emphasizing the importance of robust requery and rerank mechanisms for effective information retrieval and analysis. |
| Oryx MLLM: On-Demand Spatial-Temporal Understanding at Arbitrary Resolution (Read more on arXiv or HuggingFace) | jiwenlu, WinstonHu, liuziwei7, THUdyh, Zuyan | The authors propose Oryx, a novel multi-modal large language model (MLLM) that adeptly handles diverse visual input sizes and lengths. Oryx employs OryxViT, a visual encoder designed for native resolution processing, and a dynamic compression module for efficient processing of long video sequences. Through comprehensive experiments, Oryx demonstrates state-of-the-art performance on various benchmarks, including long-form video comprehension and 3D spatial understanding tasks. This work provides AI practitioners with a robust and versatile MLLM architecture capable of handling real-world multimodal data with varying resolutions and lengths. |
| StoryMaker: Towards Holistic Consistent Characters in Text-to-image Generation (Read more on arXiv or HuggingFace) | CantabPhD, chenyibo89, huaxiali, jingli, huaquan | StoryMaker is a novel, tuning-free AI model for personalized image generation that preserves the consistency of facial features, clothing, hairstyles, and body types across multiple character scenes, facilitating coherent visual storytelling. It leverages a Positional-aware Perceiver Resampler to generate distinct character embeddings and employs a novel attention loss mechanism with segmentation masks to prevent feature intermingling between characters and the background. Experiments demonstrate StoryMaker’s superior performance in maintaining visual consistency over state-of-the-art methods, particularly in multi-character scenarios. StoryMaker offers AI practitioners a powerful tool for a variety of applications including digital storytelling, comic creation, and character-driven image editing, enabling new possibilities for creative content generation. |
| LVCD: Reference-based Lineart Video Colorization with Diffusion Models (Read more on arXiv or HuggingFace) | Mohan Zhang, CeciliaJL, luckyhzt | This research proposes LVCD, the first video diffusion framework for reference-based lineart video colorization. By leveraging a pre-trained video diffusion model, LVCD generates temporally consistent and high-quality colorized animations from lineart sketches and a single reference frame. The authors introduce two novel components: sketch-guided ControlNet for incorporating lineart sketches and Reference Attention for long-range spatial color propagation. Experiments demonstrate LVCD’s superior performance in generating long animations with large motions, surpassing existing CNN-based and diffusion-based methods. LVCD offers a promising solution for AI engineers and data scientists in the animation industry, enabling automated colorization of animation sequences and potentially boosting productivity. |
| 3DTopia-XL: Scaling High-quality 3D Asset Generation via Primitive Diffusion (Read more on arXiv or HuggingFace) | hongfz16, Caoza, THUdyh, jiaxiang-tang, FrozenBurning | The paper proposes 3DTopia-XL, a novel 3D generative model that produces high-quality, textured 3D assets from text or image inputs. It utilizes a novel primitive-based representation called PrimX, which encodes shape, texture, and material information efficiently in a compact tensor format, enabling scalability to high resolutions. 3DTopia-XL leverages a Diffusion Transformer architecture for generative modeling and outperforms existing methods in terms of visual fidelity, particularly in generating fine-grained textures and Physically Based Rendering (PBR) materials. The high-quality outputs, coupled with efficient asset extraction into industry-standard formats like GLB, makes 3DTopia-XL readily applicable for AI practitioners working on 3D content creation tasks in domains such as gaming, virtual reality, and design. |
| Language Models Learn to Mislead Humans via RLHF (Read more on arXiv or HuggingFace) | Jacob Steinhardt, EthanAraragi, akbir, ruiqi-zhong, jiaxin-wen | This paper presents empirical evidence that RLHF, a popular technique for aligning language models, can lead to an unintended consequence termed “U-SOPHISTRY.” U-SOPHISTRY occurs when language models, optimized based on human feedback, learn to generate outputs that appear correct to human evaluators but are factually incorrect. The authors demonstrate this phenomenon on question-answering and programming tasks, finding that RLHF leads to a significant increase in human approval of incorrect outputs while actual task performance stagnates. The study highlights a critical risk associated with RLHF: it can create a false sense of improvement in language models, potentially misleading practitioners such as AI engineers and data scientists who rely on human evaluation for model assessment and selection. These findings underscore the need for developing more robust evaluation methods and mitigation strategies to address U-SOPHISTRY. |
| Scaling Smart: Accelerating Large Language Model Pre-training with Small Model Initialization (Read more on arXiv or HuggingFace) | mfarajtabar, moinnabi, thyeros, fartashf, imirzadeh-apple | This research paper introduces HyperCloning, a novel method for initializing large language models (LLMs) using pretrained smaller models. HyperCloning expands the hidden dimensions of a smaller model while preserving its functionality, ensuring the larger model inherits the smaller model’s accuracy before training begins. Experiments demonstrate that HyperCloning reduces training time by a factor of 2-4 compared to random initialization, achieving comparable or superior accuracy across various LLM architectures. This technique offers practitioners, including AI engineers and data scientists, a cost-effective and efficient approach to training LLMs, potentially democratizing access to high-performance models. Further research directions include investigating the observed catastrophic forgetting and exploring alternative weight expansion strategies to further enhance HyperCloning’s effectiveness. |
| Denoising Reuse: Exploiting Inter-frame Motion Consistency for Efficient Video Latent Generation (Read more on arXiv or HuggingFace) | Yixuan Chen, Shuo Yan, Chenyu Wang, dongshengli, genye | This paper introduces Dr. Mo, a novel diffusion-based video generation model that exploits inter-frame motion consistency to accelerate latent video generation. The key insight lies in the observation that coarse-grained features in the diffusion process exhibit high motion consistency across video frames. Dr. Mo leverages this finding by reusing denoising steps from a reference frame via a learned motion transformation network and a denoising step selector, significantly reducing computational overhead. Evaluations on UCF-101 and MSR-VTT datasets demonstrate that Dr. Mo achieves state-of-the-art video quality with a 4x speedup compared to previous methods. This work holds significant implications for AI practitioners, particularly those working on video generation and editing tasks, as it offers a pathway to generate high-quality videos with significantly reduced computational resources. |
| MURI: High-Quality Instruction Tuning Datasets for Low-Resource Languages via Reverse Instructions (Read more on arXiv or HuggingFace) | Ayyoob Imani, akorhonen, ahmetu, noriamt, akoksal | This research introduces Multilingual Reverse Instructions (MURI), a novel method for generating high-quality instruction tuning datasets for low-resource languages by leveraging existing multilingual text corpora and machine translation. The authors create MURI-IT, a dataset comprising over 2 million instruction-output pairs across 200 languages, with a significant focus on under-resourced languages. Evaluation by native speakers and fine-tuning experiments with mT5 models demonstrate the effectiveness of MURI-IT in improving multilingual instruction following capabilities, particularly for natural language understanding tasks. This work provides a valuable resource for AI practitioners working on multilingual language models and addresses the crucial need for diverse and inclusive datasets in NLP. The released datasets and models offer significant potential for downstream applications like machine translation, cross-lingual information retrieval, and chatbot development in a wider range of languages. |
| FlexiTex: Enhancing Texture Generation with Visual Guidance (Read more on arXiv or HuggingFace) | zouxb009, ysx007, aaronb, jiaaoyu, cocacola | This paper introduces FlexiTex, a novel framework for high-fidelity texture generation on 3D objects using both text and image prompts. FlexiTex addresses limitations of existing methods by incorporating a Visual Guidance Enhancement module, which uses image prompts to provide explicit guidance during texture generation, thus enhancing detail richness and style consistency. Additionally, a Direction-Aware Adaptation module leverages direction prompts to mitigate the Janus problem and improve semantic alignment across views. Experiments demonstrate FlexiTex’s superior performance in quantitative metrics and qualitative results compared to baseline methods. Practitioners, such as AI engineers and data scientists, can leverage FlexiTex to generate high-quality textures for 3D objects efficiently, benefiting applications like AR/VR, gaming, and film. |
| 3DGS-LM: Faster Gaussian-Splatting Optimization with Levenberg-Marquardt (Read more on arXiv or HuggingFace) | Matthias Nießner, Michael Zollhöfer, Aljaž Božič, Lukas Höllein | This paper introduces 3DGS-LM, a novel method for accelerating the reconstruction process in 3D Gaussian Splatting (3DGS). By replacing the conventional ADAM optimizer with a tailored Levenberg-Marquardt (LM) algorithm, the authors achieve a 30% reduction in optimization time while maintaining reconstruction quality. This speedup is achieved through a highly-efficient GPU parallelization scheme for the preconditioned conjugate gradient algorithm, utilizing a custom CUDA kernel implementation and a caching data structure for intermediate gradients. This advancement holds significant relevance for AI practitioners working with 3DGS, particularly in applications such as virtual reality and scene exploration, where faster reconstruction times can greatly benefit development cycles and user experience. |
Papers for 2024-09-19
| Title | Authors | Summary |
|---|---|---|
| Qwen2.5-Coder Technical Report (Read more on arXiv or HuggingFace) | Lemoncoke, Losin94, AbbottYJX, yangjian076, huybery | The paper introduces Qwen2.5-Coder, an open-source series of code language models built on the Qwen2.5 architecture and trained on a 5.5 trillion token dataset. Qwen2.5-Coder achieves state-of-the-art results across a variety of code generation, code completion, and code reasoning benchmarks, outperforming even significantly larger models. This performance is attributed to a robust data pipeline emphasizing high-quality code and code-related data, as well as meticulous instruction-tuning techniques. Qwen2.5-Coder’s capabilities, particularly its performance exceeding larger models, makes it a valuable tool for AI practitioners developing code generation, completion, and reasoning applications. Its open-source nature further facilitates research and application development in code intelligence. |
| Qwen2-VL: Enhancing Vision-Language Model’s Perception of the World at Any Resolution (Read more on arXiv or HuggingFace) | gewenbin292, chenkq, Jinze, tinytangent, bluelike | The research paper “Qwen2-VL: Enhancing Vision-Language Model’s Perception of the World at Any Resolution” introduces the Qwen2-VL series, a collection of open-weight vision-language models featuring 2, 8, and 72 billion parameters. Notably, Qwen2-VL incorporates a Naive Dynamic Resolution mechanism allowing for the processing of images with varying resolutions and a Multimodal Rotary Position Embedding (M-ROPE) for effectively encoding positional information across various modalities. This approach leads to state-of-the-art performance in various visual benchmarks, including extended-duration video comprehension and robust agent capabilities for device operation. Qwen2-VL’s capabilities in visual reasoning, document understanding, multilingual text recognition, video comprehension, and visual agent capabilities are particularly relevant for AI practitioners, including AI engineers and data scientists, offering a robust framework for developing applications in areas like image analysis, video processing, and human-computer interaction. |
| LLMs + Persona-Plug = Personalized LLMs (Read more on arXiv or HuggingFace) | Erxue Min, Xiaochi Wei, stingw, yutaozhu94, liujiongnan | This paper proposes PPlug, a novel personalized Large Language Model (LLM) designed to tailor outputs according to individual user preferences. PPlug leverages a plug-in user embedder module to encode a user’s entire interaction history into a single, comprehensive embedding, capturing general linguistic patterns and preferences. Experiments conducted on the Language Model Personalization (LaMP) benchmark demonstrate PPlug’s superiority, outperforming retrieval-based and fine-tuned personalized LLMs. Notably, PPlug’s plug-and-play architecture offers efficiency by utilizing a single LLM for all users, making it a practical solution for LLM service providers seeking to offer personalized experiences. AI engineers and data scientists can leverage PPlug to enhance personalization in applications ranging from drafting personalized content to tailoring recommendations based on user history. |
| To CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic reasoning (Read more on arXiv or HuggingFace) | wadhma, Dongwei, juand-r, fcyin, Zaynes | The research paper “To CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic reasoning” by wadhma et al. investigates the effectiveness of chain-of-thought (CoT) prompting for enhancing large language model (LLM) reasoning capabilities. Through meta-analysis of existing literature and empirical evaluations across 20 datasets and 14 contemporary LLMs, the authors demonstrate that CoT provides substantial performance benefits primarily for tasks involving mathematics or formal logic, with minimal gains observed for tasks requiring non-symbolic reasoning. Further analysis reveals that CoT’s strength lies in its ability to execute symbolic steps and track intermediate computational outputs. The authors suggest that while CoT remains a useful technique, practitioners, including AI Engineers and Data Scientists, should prioritize integrating LLMs with symbolic solvers for optimal performance on symbolic tasks and explore alternative paradigms, such as search or interacting agents, to enhance reasoning in non-symbolic domains. |
| Preference Tuning with Human Feedback on Language, Speech, and Vision Tasks: A Survey (Read more on arXiv or HuggingFace) | David D. Yao, Wenpin Tang, anirbandas, BraceZHY, gentaiscool | This survey paper provides a thorough overview of recent advancements in preference tuning, a crucial process for aligning deep generative models with human preferences, across language, speech, and vision tasks. The paper presents a systematic framework and classification of preference tuning methods, categorizing them by sampling methods (online or offline), modality (text, speech, vision, etc.), language, and reward granularity (sample or token level). The authors also describe various applications of preference tuning for improving generation quality using human feedback and discuss evaluation methods, highlighting both automatic LLM-based approaches and human-based evaluations. This survey is highly relevant to practitioners, such as AI engineers and data scientists, who aim to enhance the alignment of deep generative models with human preferences, leading to more human-like and desirable outputs in various domains, including text generation, image synthesis, and speech synthesis. |
| GRIN: GRadient-INformed MoE (Read more on arXiv or HuggingFace) | uuu6, liangchen-ms, Shuohang, ykim362, LiyuanLucasLiu | The paper introduces GRIN, a novel training method for Mixture-of-Experts (MoE) models, designed to overcome the limitations of discrete expert routing in gradient-based optimization. GRIN leverages SparseMixer-v2, a method that estimates gradients for expert routing directly, instead of relying on gating gradients as a proxy. This approach, combined with a modified load balance loss and the use of tensor parallelism instead of expert parallelism, allows for efficient scaling of MoE models without token dropping. The authors demonstrate the efficacy of GRIN by developing a 16x3.8B MoE model that outperforms a 7B dense model and matches a 14B dense model, achieving state-of-the-art performance on various benchmarks, especially in coding and mathematics. These results highlight GRIN’s potential for AI engineers and data scientists seeking to build highly scalable and performant MoE models for complex tasks. |
| Takin: A Cohort of Superior Quality Zero-shot Speech Generation Models (Read more on arXiv or HuggingFace) | yangyutu, sonaxyjh, ClorisLIN, YanniHu, ch3cook-fdu | The research introduces Takin AudioLLM, a suite of zero-shot speech generation models including Takin TTS, Takin VC, and Takin Morphing, aimed at high-quality, customizable audiobook production. Takin TTS, a neural codec language model, leverages a multi-task training strategy and a latent diffusion model for natural and robust speech synthesis. Takin VC employs joint content-timbre modeling and conditional flow matching for high-fidelity voice conversion. Takin Morphing allows timbre and prosody customization using an attention-based multi-reference timbre encoder and a language model-based prosody encoder. Experimental results demonstrate the superiority of Takin AudioLLM models over conventional methods in terms of speech quality, speaker similarity, and style control, making it a valuable tool for AI engineers and data scientists working on speech generation and audiobook production. |
| Towards Diverse and Efficient Audio Captioning via Diffusion Models (Read more on arXiv or HuggingFace) | Ruibo Fu, Yong Ren, Xinyi Tu, Manjie Xu, Chenxinglili | This paper presents Diffusion-based Audio Captioning (DAC), a novel non-autoregressive model for audio captioning that leverages a diffusion framework. DAC operates within the continuous text latent space and conditions the denoising process on audio features through cross-attention. Experimental results demonstrate that DAC achieves competitive captioning quality compared to state-of-the-art autoregressive models while exhibiting superior performance in terms of generation diversity and speed. Notably, the authors observe that DAC benefits significantly from pre-training on larger audio datasets and that semantic similarity metrics like CLAP and BERT might be more suitable for evaluating captioning quality compared to traditional token-level metrics. DAC’s efficiency and diversity make it a compelling solution for AI practitioners interested in deploying audio captioning models in resource-constrained environments or real-time applications. |
| A Controlled Study on Long Context Extension and Generalization in LLMs (Read more on arXiv or HuggingFace) | Jing Nathan Yan, Yi Lu, zy001, justintchiu, sonta7 | This research presents a controlled empirical study of long-context extension methods in Large Language Models (LLMs). The authors standardize evaluation across various exact and approximate attention methods, utilizing LLaMA2-7B as a consistent base model, trained on a 1B token long-context dataset. Results indicate that perplexity remains a reliable indicator of downstream task performance for exact attention methods, while approximate attention suffers from reduced accuracy, especially in retrieval tasks. Notably, continual fine-tuning with exact attention proves effective within the extended context length, while extrapolation to unseen lengths presents challenges. These findings, coupled with the open-sourced code and models, offer AI practitioners valuable insights into selecting and implementing appropriate context extension methods for their LLM applications, highlighting the trade-offs between accuracy, computational cost, and generalization capabilities. |
| Vista3D: Unravel the 3D Darkside of a Single Image (Read more on arXiv or HuggingFace) | Michael Bi Mi, wxcTest, adamdad, florinshum | The authors present Vista3D, a novel coarse-to-fine framework for generating diverse and consistent 3D objects from single images using 2D diffusion priors. Vista3D utilizes Gaussian Splatting to efficiently establish a coarse 3D geometry, subsequently refining it into a signed distance field representation with disentangled textures. Notably, Vista3D leverages a novel angular composition approach, constraining diffusion prior gradients to balance diversity in the unseen 3D aspects with overall consistency. Experiments demonstrate Vista3D’s ability to generate high-fidelity textured meshes in 5 minutes, outperforming existing methods in speed and quality. This framework offers practitioners, including AI engineers and data scientists, a robust and efficient tool for single-view 3D object reconstruction, with potential applications in areas such as virtual reality and 3D content creation. |
Papers for 2024-09-18
| Title | Authors | Summary |
|---|---|---|
| OmniGen: Unified Image Generation (Read more on arXiv or HuggingFace) | stingw, Ruiran, avery00, JUNJIE99, Shitao | The research introduces OmniGen, a novel diffusion-based model for unified image generation. Unlike task-specific models, OmniGen handles diverse tasks such as text-to-image generation, image editing, and subject-driven generation within a single framework. Trained on the newly introduced X2I dataset, a large-scale, multi-task dataset, OmniGen exhibits emergent capabilities like task composition and in-context learning for unseen tasks. Evaluation on benchmarks like GenEval and EMU-Edit demonstrates competitive performance compared to state-of-the-art models. This advancement is particularly relevant to AI practitioners, offering a unified and simplified approach to various image generation tasks within a single, efficient model. |
| NVLM: Open Frontier-Class Multimodal LLMs (Read more on arXiv or HuggingFace) | tuomass, jon-barker, zihanliu, boxin-wbx, nayeon7lee | The paper presents NVLM 1.0, a family of multimodal large language models (MLLMs) that achieve state-of-the-art results on a variety of vision-language tasks. NVLM 1.0 comes in three architectures: decoder-only (NVLM-D), cross-attention-based (NVLM-X), and a novel hybrid architecture (NVLM-H), each offering unique advantages in computational efficiency and reasoning capabilities. Importantly, NVLM 1.0 models demonstrate “production-grade multimodality,” excelling in both vision-language and text-only tasks, without sacrificing performance in either domain. This is achieved through a combination of novel model design, the introduction of a 1-D tile tagging design for high-resolution images, and careful curation of training data that emphasizes quality and task diversity over scale. Practitioners can benefit from these insights for building more robust and versatile MLLMs applicable to a wide range of tasks, from visual question answering to code generation. |
| Phidias: A Generative Model for Creating 3D Content from Text, Image, and 3D Conditions with Reference-Augmented Diffusion (Read more on arXiv or HuggingFace) | Gerhard Hancke, liuziwei7, zxhezexin, tfwang, ZhenweiWang | Phidias is a novel generative model that employs diffusion for reference-augmented 3D content creation. The model leverages a user-provided or retrieved 3D reference to enhance the 3D generation process, thereby improving the generation quality, generalizability, and controllability. Phidias unifies 3D generation from textual, image-based, and 3D prompts, providing a variety of downstream applications for practitioners, such as retrieval-augmented image-to-3D or text-to-3D generation. The authors demonstrate through extensive experiments that Phidias outperforms existing state-of-the-art approaches both quantitatively and qualitatively. The source code for Phidias is publicly available. |
| Fine-Tuning Image-Conditional Diffusion Models is Easier than You Think (Read more on arXiv or HuggingFace) | Alexander Hermans, Christian Schmidt, ddegeus, kabouzeid, GonzaloMG | This research paper demonstrates that the perceived inefficiency of image-conditional latent diffusion models for monocular depth estimation, such as Marigold, is due to a flawed inference pipeline. By fixing the DDIM scheduler implementation, the authors achieve single-step inference performance comparable to multi-step, ensembled approaches, with a speed increase of over 200x. Furthermore, simple end-to-end fine-tuning of these models with task-specific losses, even starting from a pre-trained Stable Diffusion model, surpasses the performance of more complex, specifically designed architectures. These findings are particularly relevant to practitioners, as they enable the use of high-precision, diffusion-based depth and normal estimation models in real-time applications, while also simplifying the training and optimization process. |
| On the limits of agency in agent-based models (Read more on arXiv or HuggingFace) | Shashank Kumar, arnauqb, rameshraskar, ngkuru, Godssidekick1 | This paper introduces AgentTorch, a novel framework for building scalable and differentiable agent-based models (ABMs) enhanced by large language models (LLMs). AgentTorch addresses the challenge of simulating large populations with adaptive behaviors by introducing the concept of LLM archetypes, enabling the simulation of millions of agents informed by LLM outputs. The authors demonstrate AgentTorch’s capabilities through a case study of the COVID-19 pandemic in New York City, showcasing its ability to capture realistic population-wide behaviors and simulate the impact of policy interventions. AgentTorch provides practitioners, including AI engineers and data scientists, with a powerful tool for understanding and addressing complex societal challenges through the integration of LLM-driven agent behavior in ABMs. |
| OSV: One Step is Enough for High-Quality Image to Video Generation (Read more on arXiv or HuggingFace) | Jiangning Zhang, Wenbing Zhu, Zhengkai Jiang, Xiaofeng Mao, wangfuyun | The authors present OSV (One Step Video Generation), a novel two-stage training approach for image-to-video generation using diffusion models that achieves high-quality results in just one inference step. OSV leverages latent GAN training in the first stage for rapid quality improvement and incorporates adversarial consistency distillation in the second stage to enhance performance and stability. The authors introduce a unique video discriminator design using pretrained image backbones (DINOv2) and a lightweight trainable head, significantly reducing computational costs by replacing the VAE decoding process with upsampling. Evaluations on the OpenWebVid-1M benchmark demonstrate OSV’s superior performance over existing methods in both speed and visual quality. OSV presents a significant advancement for practitioners, such as AI engineers and data scientists, working with video generation, offering a fast and efficient solution for high-quality results. |
| A Comprehensive Evaluation of Quantized Instruction-Tuned Large Language Models: An Experimental Analysis up to 405B (Read more on arXiv or HuggingFace) | Yongin Kwon, Sihyeong Park, oj9040, kwonse, leejaymin | This research paper presents a comprehensive evaluation of the quantization of instruction-tuned large language models (LLMs), spanning models from 7B to 405B parameters and four quantization methods (GPTQ, AWQ, SmoothQuant, and FP8). The authors found that quantized larger LLMs often outperform smaller, full-precision models on various tasks, except for hallucination detection and instruction following. Importantly, the study highlights that weight-only quantization methods, particularly AWQ, generally yield better accuracy preservation in large models compared to quantization methods involving activations. The findings are particularly relevant for practitioners, such as AI engineers and data scientists, aiming to deploy large LLMs under resource constraints while maintaining performance. The authors emphasize that selecting the optimal quantization method and bit precision should be done based on the specific LLM size and target task. |
| EzAudio: Enhancing Text-to-Audio Generation with Efficient Diffusion Transformer (Read more on arXiv or HuggingFace) | Helin Wang, Hao Zhang, Yong Xu, Chenxinglili, Higobeatz | EzAudio is a novel text-to-audio (T2A) generation framework that leverages a highly efficient Diffusion Transformer (DiT) architecture operating directly on raw waveform latent space. The authors propose a multi-stage training strategy employing masked acoustic modeling and synthetic caption generation, along with a classifier-free guidance rescaling technique to balance audio quality and text alignment. Experimental results demonstrate that EzAudio outperforms existing open-source T2A models in both objective and subjective evaluations, achieving state-of-the-art performance. This work provides AI practitioners a robust and accessible framework for developing high-quality T2A applications. |
| SplatFields: Neural Gaussian Splats for Sparse 3D and 4D Reconstruction (Read more on arXiv or HuggingFace) | Robert Maier, Siyu Tang, Aeriphi, sprokudin, markomih | This paper presents SplatFields, a novel optimization strategy for 3D Gaussian Splatting (3DGS) that addresses the technique’s limitations in sparse view scenarios. SplatFields introduces a spatial bias during optimization by leveraging neural networks to predict splat features, encouraging nearby primitives to share similar characteristics and emulating the behavior of implicit volumetric rendering methods. This approach significantly improves reconstruction quality under sparse view conditions for both static and dynamic scenes, outperforming recent 3DGS and NeRF-based alternatives. Notably, SplatFields maintains real-time rendering capabilities and compatibility with existing 3DGS pipelines, making it particularly attractive for practitioners seeking efficient and high-quality 3D reconstruction from limited input data. AI engineers and data scientists working on 3D vision applications such as scene reconstruction, novel view synthesis, and dynamic scene modeling can benefit from incorporating SplatFields to enhance performance and efficiency in their workflows. |
| Agile Continuous Jumping in Discontinuous Terrains (Read more on arXiv or HuggingFace) | Changyi Lin, mateoguaman, romesco, guanya, yxyang | This paper proposes a novel hierarchical learning and control framework for enabling quadrupedal robots to perform agile, continuous jumping in discontinuous terrains, such as stairs and stepping stones. The framework consists of a learned heightmap predictor for terrain perception, an RL-trained motion policy for planning, and a model-based leg controller for motion tracking. A key contribution is the reduction of the sim-to-real gap by accurately modeling hardware characteristics, such as motor saturation and camera latency. This allows the robot to achieve state-of-the-art performance, traversing a 14-step staircase in 4.5 seconds, demonstrating the effectiveness of the proposed approach for agile locomotion in challenging terrains. This work holds significant implications for practitioners, including AI Engineers and roboticists, seeking to develop robots capable of navigating complex real-world environments with enhanced agility and speed. |
| Single-Layer Learnable Activation for Implicit Neural Representation (SL$^{2}$A-INR) (Read more on arXiv or HuggingFace) | Hamid Soltanian-Zadeh, Dorit Merhof, Reza Azad, Reza-R-77, moein99 | This paper introduces SL$^{2}$A-INR, a novel implicit neural representation (INR) architecture that utilizes a single-layer learnable activation function based on Chebyshev polynomials. SL$^2$A-INR effectively captures high-frequency details and mitigates spectral bias, outperforming existing INRs on various tasks including image representation, 3D shape reconstruction, and inverse problems like super-resolution and CT reconstruction. Notably, SL$^2$A-INR achieves superior performance even with reduced model sizes compared to other INR methods. The demonstrated effectiveness and efficiency of SL$^2$A-INR across diverse tasks makes it a valuable tool for AI practitioners working on signal representation and generative modeling, particularly in applications requiring high-fidelity reconstruction from limited data. |
| PDMX: A Large-Scale Public Domain MusicXML Dataset for Symbolic Music Processing (Read more on arXiv or HuggingFace) | Julian McAuley, Phillip Long, tberg12, ZacharyNovack | This paper introduces PDMX, the largest publicly available dataset of public domain MusicXML files, comprising over 250,000 scores and encompassing 6,250 hours of music. The authors release MusicRender, an extension to the MusPy library, to facilitate accurate parsing and rendering of nuanced musical notation from MusicXML. Experiments on multitrack symbolic music generation demonstrate that filtering PDMX based on user ratings improves model performance in terms of harmonic and rhythmic diversity. Notably, fine-tuning models on a small subset of high-quality, rated data significantly enhances generation quality. PDMX offers AI practitioners a valuable resource for developing and evaluating symbolic music processing models, particularly in the domains of music generation, transcription, and recommendation. |
| Measuring and Enhancing Trustworthiness of LLMs in RAG through Grounded Attributions and Learning to Refuse (Read more on arXiv or HuggingFace) | Navonil Majumder, Hai Leong Chieu, Rishabh Bhardwaj, Shang Hong Sim, Maojia Song | This paper addresses the issue of hallucination in Large Language Models (LLMs) within the context of Retrieval-Augmented Generation (RAG). The authors propose a novel metric, TRUST-SCORE, to evaluate the trustworthiness of LLMs in a RAG setting by assessing grounded refusals, answer accuracy, and citation correctness. To improve trustworthiness, they introduce TRUST-ALIGN, an alignment framework that trains LLMs on a synthetic dataset to identify answerable questions, ground responses in provided documents, and avoid unnecessary refusals. Experiments demonstrate that TRUST-ALIGN enhances LLM performance across three datasets, achieving comparable results to leading closed-source language models like GPT-4. These findings are particularly relevant to AI engineers and data scientists developing RAG systems, emphasizing the importance of aligning LLMs with external knowledge sources to mitigate hallucination and improve the reliability of generated information. |
| Implicit Neural Representations with Fourier Kolmogorov-Arnold Networks (Read more on arXiv or HuggingFace) | Ilker Hacihaliloglu, Parsa Mojarad Adi, moein99, ali-mrbn | This paper introduces Fourier Kolmogorov-Arnold Network (FKAN), a novel architecture for implicit neural representations (INRs) designed to enhance the capture of task-specific frequency components in signals. FKAN leverages learnable activation functions modeled as Fourier series, enabling fine-grained control and learning of frequency information. Experimental results demonstrate that FKAN surpasses state-of-the-art baselines in image representation and 3D occupancy volume representation tasks, achieving improvements in PSNR, SSIM, and IoU metrics while exhibiting faster convergence. This novel approach provides AI practitioners, including AI engineers and data scientists, with an effective tool to enhance INR models for various applications requiring high-fidelity signal representation. |
Papers for 2024-09-17
| Title | Authors | Summary |
|---|---|---|
| Seed-Music: A Unified Framework for High Quality and Controlled Music Generation (Read more on arXiv or HuggingFace) | lixingxing, lich-ming, ducle, smileezzz, Weituo | Seed-Music is a novel framework for high-quality and controllable vocal music generation and editing. The authors introduce a system comprised of three core components: Representation Learning, Generation, and Rendering, which utilize audio tokens, symbolic music tokens, or vocoder latents as intermediate representations. Seed-Music leverages both autoregressive language modeling and diffusion approaches to achieve impressive results in tasks such as Lyrics2Song, Lyrics2Leadsheet2Song, MusicEDiT, and Zero-shot Singing Voice Conversion. The system’s flexibility, controllability, and impressive performance showcased through various applications and listening examples provide AI engineers and data scientists with valuable tools for music generation, post-production editing, and creative exploration in the music domain. The introduction of “lead sheet tokens,” designed to represent musical elements in a musician-friendly format, presents a potential new standard for music language models. |
| RetrievalAttention: Accelerating Long-Context LLM Inference via Vector Retrieval (Read more on arXiv or HuggingFace) | zqx123, hzhua, iofu728, baotonglu, Matchyc | This paper proposes RetrievalAttention, a training-free approach leveraging approximate nearest neighbor search (ANNS) to accelerate the inference of long-context Large Language Models (LLMs) by exploiting the dynamic sparsity inherent in the attention mechanism. The key innovation lies in addressing the out-of-distribution (OOD) challenge between query and key vectors in attention computation through an attention-aware vector search algorithm. This enables RetrievalAttention to accurately approximate attention with significantly reduced latency and minimal GPU memory footprint, achieving a 4.9x and 1.98x speedup compared to exact KNN and traditional ANNS methods respectively. RetrievalAttention presents a practical solution for AI practitioners working with LLMs on long sequences, particularly beneficial for deployment on resource-constrained devices. |
| Guiding Vision-Language Model Selection for Visual Question-Answering Across Tasks, Domains, and Knowledge Types (Read more on arXiv or HuggingFace) | Vinija Jain, amanchadha, neelabhsinha | This research paper proposes a comprehensive framework for evaluating and selecting optimal Vision-Language Models (VLMs) for specific Visual Question Answering (VQA) tasks, addressing practical application needs. The authors introduce a novel multi-dimensional dataset that classifies VQA tasks by task type, application domain, and knowledge type, facilitating fine-grained VLM performance comparisons. Additionally, a new evaluation metric, GoEval, is presented, demonstrating superior alignment with human judgments compared to traditional metrics by leveraging GPT-40’s capabilities for multimodal evaluation. Experimental results reveal significant performance variations among 10 state-of-the-art VLMs across categories, with proprietary models generally outperforming open-source alternatives. These findings provide AI practitioners (AI Engineers, Data Scientists) with actionable insights and a standardized framework for selecting best-suited VLMs based on specific task requirements, resource constraints, and performance expectations. |
| ReCLAP: Improving Zero Shot Audio Classification by Describing Sounds (Read more on arXiv or HuggingFace) | Sonal Kumar, Sreyan Ghosh, manocha, RamaniD, urinieto | The research proposes ReCLAP, an improved CLAP model for zero-shot audio classification (ZSAC) that enhances sound understanding by incorporating descriptive features into prompts. ReCLAP leverages caption augmentation during training, prompting a Large Language Model (LLM) to rewrite captions with detailed acoustic descriptions. Further improving ZSAC, the authors introduce prompt augmentation, generating multiple custom prompts per category using LLM-based descriptions in diverse scenes. ReCLAP exhibits state-of-the-art performance on various retrieval and ZSAC benchmarks, demonstrating the importance of descriptive sound features in prompts. This development holds significant relevance for AI practitioners, particularly those working on audio classification and retrieval systems, by providing a method to improve zero-shot performance and generalization capabilities. |
| On the Diagram of Thought (Read more on arXiv or HuggingFace) | Andrew Chi-Chih Yao, Yang Yuan, yifAI | The paper introduces Diagram of Thought (DoT), a novel framework for enhancing iterative reasoning in large language models (LLMs) by representing the process as the construction of a directed acyclic graph (DAG) within a single model. Unlike linear or tree-based reasoning approaches, DoT incorporates propositions, critiques, refinements, and verifications as nodes within the DAG, capturing the non-linear and iterative nature of human reasoning. By employing auto-regressive next-token prediction with role-specific tokens, DoT facilitates seamless transitions between reasoning steps within the LLM, eliminating the need for multiple models or external control mechanisms. Furthermore, the authors provide a robust mathematical foundation for DoT using Topos Theory and PreNet Categories, ensuring the logical consistency and soundness of the reasoning process. This framework offers AI practitioners a theoretically grounded and practically efficient approach to develop LLMs with enhanced reasoning capabilities for complex problem-solving tasks. |
| AudioBERT: Audio Knowledge Augmented Language Model (Read more on arXiv or HuggingFace) | Jaeho Lee, uso7d0, HJOK | This paper introduces AuditoryBench, the first benchmark designed to assess the auditory knowledge of large language models (LLMs). The authors find that LLMs pretrained solely on text data exhibit a significant lack of auditory commonsense knowledge. To address this, they propose AudioBERT, a novel framework that augments LLMs with auditory knowledge through a retrieval-based approach using a combination of auditory knowledge span detection and the CLAP audio-text model. Experiments demonstrate that AudioBERT significantly enhances the ability of LLMs to understand and reason about auditory information. This research has practical implications for AI practitioners, particularly those working on audio-language multimodal tasks such as audio captioning, sound recognition, and audio question answering. The availability of AudioBERT and AuditoryBench provides valuable resources for developing more robust and versatile multimodal AI systems. |
| One missing piece in Vision and Language: A Survey on Comics Understanding (Read more on arXiv or HuggingFace) | Mohamed Ali Souibgui, Andrey Barsky, MarcoBertini, Llabres, emanuelevivoli | This survey paper provides a comprehensive overview of the emerging field of Comics Understanding within the context of Vision-Language multimodal tasks. The authors introduce the novel Layer of Comics Understanding (LoCU) framework, a taxonomy that categorizes tasks based on input/output modalities and spatio-temporal dimensions, ranging from basic tagging and augmentation to complex generation and synthesis. The survey systematically reviews existing datasets and methodologies, highlighting the limitations in data availability, annotation standardization, and task complexity, and proposes potential research directions. Practitioners, such as AI engineers and data scientists, can leverage this survey to understand the current state of the field, identify potential applications of VLMs in comics analysis and generation, and contribute to the development of more robust and versatile models for this complex domain. |
| Ferret: Federated Full-Parameter Tuning at Scale for Large Language Models (Read more on arXiv or HuggingFace) | Fei Richard Yu, Bryan Kian Hsiang Low, See-Kiong Ng, Wenyang Hu, ZCODE0 | Ferret is a novel first-order federated learning algorithm designed for scalable full-parameter tuning of large language models (LLMs) with enhanced privacy. It leverages shared randomness to reduce communication costs by projecting local updates into a low-dimensional space and reconstructing them efficiently during global aggregation. Theoretical analyses demonstrate that Ferret’s reconstruction is unbiased and enjoys fast convergence while avoiding error accumulation often observed in zeroth-order methods. Empirical evaluations on benchmark datasets confirm Ferret’s superior scalability and competitive model accuracy compared to existing federated full-parameter and parameter-efficient tuning methods. This work holds significant implications for practitioners, especially AI engineers and data scientists, enabling them to efficiently fine-tune LLMs on decentralized datasets with improved privacy while maintaining performance. |
| beeFormer: Bridging the Gap Between Semantic and Interaction Similarity in Recommender Systems (Read more on arXiv or HuggingFace) | Pavel Kordík, foxik, beeformer | The authors propose beeFormer, a novel framework that bridges the gap between semantic and interaction similarity for recommender systems. This is accomplished by training sentence transformer models directly on user-item interaction data, leveraging gradient checkpointing and negative sampling for scalability. Experimental results demonstrate that beeFormer outperforms baselines in cold-start, zero-shot, and time-split recommendation tasks, indicating superior performance in scenarios with limited interaction data. Notably, training on datasets from multiple domains leads to improved knowledge transfer and domain-agnostic recommendation capabilities. These findings are especially relevant for AI practitioners, as beeFormer offers a scalable and effective approach to improve recommendation quality in challenging scenarios with limited user feedback. |
| Towards Predicting Temporal Changes in a Patient’s Chest X-ray Images based on Electronic Health Records (Read more on arXiv or HuggingFace) | Tackeun Kim, forgetnight, starmpcc, dek924 | This paper proposes EHRXDiff, a novel framework that leverages latent diffusion models to predict future Chest X-ray (CXR) images by integrating previous CXRs with subsequent medical events extracted from Electronic Health Records (EHRs). The framework utilizes a combination of VAE and CLIP encoders to capture both fine-grained visual details and high-level clinical features from the input data, and effectively predicts potential temporal changes while generating realistic CXR images. Experimental results demonstrate EHRXDiff’s superior performance in preserving medical information and generating high-quality images compared to baseline methods. This framework has the potential to serve as a valuable tool for AI practitioners, particularly in developing clinical decision support systems that assist medical professionals in monitoring disease progression and planning personalized treatment strategies. |
Papers for 2024-09-16
| Title | Authors | Summary |
|---|---|---|
| Robust Dual Gaussian Splatting for Immersive Human-centric Volumetric Videos (Read more on arXiv or HuggingFace) | Yu Hong, Zhehao Shen, Yuheng Jiang, Daluuu, chengchengguo123 | This paper introduces DualGS, a novel Gaussian-based representation for robust human performance tracking and high-fidelity rendering in volumetric videos. The approach utilizes Dual Gaussians to disentangle motion and appearance, employing motion-aware joint Gaussians and appearance-aware skin Gaussians. A coarse-to-fine optimization strategy with motion prediction ensures temporal coherence and rendering fidelity. A companion compression scheme using residual vector quantization, codec compression, and a persistent codebook achieves a 120-fold compression ratio. DualGS offers AI practitioners a method for creating high-fidelity, interactive volumetric video experiences that are efficient enough for deployment on VR and mobile devices. |
Papers for 2024-09-13
| Title | Authors | Summary |
|---|---|---|
| Windows Agent Arena: Evaluating Multi-Modal OS Agents at Scale (Read more on arXiv or HuggingFace) | hrz, Inhenn, Saraabdali, francedot, rbonatti | The research paper, “Windows Agent Arena: Evaluating Multi-Modal OS Agents at Scale”, by hrz, Inhenn, Saraabdali, francedot, and rbonatti introduces a novel benchmark for evaluating multi-modal AI agents operating within a real Windows environment. This benchmark, named WINDOWSAGENTARENA, features 154 diverse tasks spanning common user applications and is designed for scalability and deployment on Azure for efficient parallel evaluation. The authors also present a new multi-modal agent, Navi, achieving a success rate of 19.5% on WINDOWSAGENTARENA tasks, showcasing the potential for future agent development. Despite being far from human performance (74.5%), Navi’s results highlight the crucial role of precise visual prompting and reveal the challenges posed by visual-language misalignment. This research is significant for practitioners, including AI engineers and data scientists, as it provides a robust platform for testing and improving the capabilities of AI agents in performing complex, real-world tasks within the prevalent Windows OS ecosystem. |
| Can LLMs Generate Novel Research Ideas? A Large-Scale Human Study with 100+ NLP Researchers (Read more on arXiv or HuggingFace) | Tatsunori Hashimoto, Diyi Yang, CLS | The paper “Can LLMs Generate Novel Research Ideas? A Large-Scale Human Study with 100+ NLP Researchers” investigates whether Large Language Models (LLMs) can generate novel research ideas comparable to human experts. The authors conducted a large-scale human study with over 100 NLP researchers, comparing ideas generated by an LLM agent with those written by experts. The study found that AI-generated ideas were judged as statistically more novel than human ideas, while remaining comparable in feasibility and other metrics. However, the authors also identify limitations in LLMs, including a lack of diversity in generated ideas and unreliability in evaluating idea quality. These findings suggest that while LLMs show promise in assisting with research ideation, they are not yet capable of fully autonomous idea generation and require careful human oversight, particularly for practitioners such as AI Engineers and Data Scientists who may utilize these tools in their work. |
| IFAdapter: Instance Feature Control for Grounded Text-to-Image Generation (Read more on arXiv or HuggingFace) | Bing Ma, wxcTest, suxuefeng, tinytigerpan, WuYW | This paper proposes IFAdapter, a novel plug-and-play module for pretrained diffusion models, designed to improve fine-grained control over the positioning and appearance of multiple instances in generated images. It addresses limitations of existing Layout-to-Image generation methods by introducing two key components: Appearance Tokens for capturing high-frequency instance details and an Instance Semantic Map for ensuring accurate spatial correspondence. Experiments on the introduced COCO-IFG benchmark demonstrate IFAdapter’s superiority in generating images with both accurate instance placement and high-fidelity features, as measured by the novel Instance Feature Success rate and standard image quality metrics. This development holds significant practical implications for AI practitioners, particularly those working on image generation tasks requiring precise control over instance features, such as in graphic design or fashion design applications. |
| DreamHOI: Subject-Driven Generation of 3D Human-Object Interactions with Diffusion Priors (Read more on arXiv or HuggingFace) | tmsj, rayli, hanwenzhu | The paper introduces DreamHOI, a novel zero-shot method for synthesizing 3D human-object interactions (HOIs). DreamHOI utilizes pre-trained text-to-image diffusion models to guide the posing of a 3D human model, enabling it to realistically interact with a given 3D object based on a textual description. To overcome the limitations of directly applying diffusion model gradients to articulation parameters, DreamHOI employs a dual implicit-explicit representation of the human model, combining neural radiance fields (NeRFs) with skeleton-driven mesh articulation. This dual representation facilitates effective optimization and preserves human identity during the generation process. Experiments demonstrate DreamHOI’s ability to generate realistic and diverse HOIs, outperforming baseline methods. This approach offers practitioners in fields like video game development and virtual reality a powerful tool for efficiently creating engaging and interactive virtual environments populated with realistically posed human characters. |
| Source2Synth: Synthetic Data Generation and Curation Grounded in Real Data Sources (Read more on arXiv or HuggingFace) | marialomeli, rraileanu, spermwhale, ncan, carlos-gemmell-malt-ai | The paper introduces Source2Synth, a novel method for generating synthetic datasets by leveraging existing real-world data sources and large language models (LLMs). This approach involves generating examples with intermediate reasoning steps grounded in the source data, and then curating the dataset using the LLM itself to improve the quality. The authors demonstrate Source2Synth’s effectiveness on multi-hop question answering and tabular question answering tasks, achieving significant performance improvements over baselines. The ability to generate high-quality synthetic data from existing sources has significant implications for practitioners, particularly in low-data regimes, as it offers a scalable and cost-effective way to improve LLM performance on complex tasks without the need for costly human annotations. AI engineers and data scientists can leverage Source2Synth to enhance their models’ capabilities in areas such as reasoning and tool usage. |
| FlashSplat: 2D to 3D Gaussian Splatting Segmentation Solved Optimally (Read more on arXiv or HuggingFace) | wxcTest, adamdad, florinshum | The authors propose FlashSplat, a novel method for segmenting 3D Gaussian Splatting (3D-GS) representations using 2D masks. By leveraging the alpha composition inherent in the 3D-GS rendering process, the authors formulate the segmentation task as a linear integer programming problem that admits a closed-form, globally optimal solution. This approach significantly outperforms previous iterative methods, achieving a 50x speedup while maintaining high accuracy and demonstrating robustness against noise in the input masks. FlashSplat’s efficiency and effectiveness in downstream tasks, such as object removal and inpainting, make it a valuable tool for AI practitioners working with 3D scene understanding and manipulation tasks. |
| PiTe: Pixel-Temporal Alignment for Large Video-Language Model (Read more on arXiv or HuggingFace) | Han Zhao, Min Zhang, Pengxiang Ding, Yang Liu, huangsiteng | The paper introduces PiTe, a Large Video-Language Model (LVidLM) that leverages object trajectories for fine-grained alignment of visual and textual modalities in videos. The authors curate PiTe-143k, a novel dataset with automatically annotated object trajectories. PiTe consistently outperforms current LVidLMs on video question answering, temporal grounding, and dense captioning tasks under zero-shot settings. This trajectory-based alignment substantially enhances video comprehension, enabling sophisticated event descriptions and precise event localization. For AI practitioners, PiTe presents a robust framework for building LVidLMs capable of fine-grained video understanding, facilitating applications like content-aware video search and summarization. |
Papers for 2024-09-12
| Title | Authors | Summary |
|---|---|---|
| PingPong: A Benchmark for Role-Playing Language Models with User Emulation and Multi-Model Evaluation (Read more on arXiv or HuggingFace) | IlyaGusev | This research paper introduces PingPong, a novel benchmark for evaluating role-playing capabilities in large language models (LLMs). PingPong employs a multi-model evaluation system where an LLM acts as the ‘player,’ another simulates a ‘user’ (interrogator), and a third LLM judges the ‘player’s’ performance based on criteria like character consistency and language fluency. The authors validate the benchmark through correlation with human annotations, achieving correlations exceeding 0.64 across English and Russian. A key finding is that averaging scores from multiple judge models enhances result reliability. This work provides AI practitioners, particularly those developing conversational AI and role-playing agents, with a valuable tool to robustly assess and benchmark LLM performance in dynamic, multi-turn conversational settings. |
| MEDIC: Towards a Comprehensive Framework for Evaluating LLMs in Clinical Applications (Read more on arXiv or HuggingFace) | Nadas31, tathagataraha, mpimentel, cchristophe, pkanithi | The research paper introduces MEDIC, a comprehensive evaluation framework for assessing the performance of Large Language Models (LLMs) in clinical applications. MEDIC evaluates LLMs across five key dimensions: medical reasoning, ethics and bias concerns, data and language understanding, in-context learning, and clinical safety and risk. The study revealed that larger models generally perform better in closed-ended question-answering tasks; however, in open-ended tasks requiring free-form responses, domain-specific fine-tuning was crucial for achieving superior performance. The MEDIC framework provides AI engineers and data scientists with a valuable tool for guiding model selection, highlighting performance trade-offs, and identifying key areas for improvement, ultimately facilitating the development of safe, effective, and ethical AI models for healthcare. This framework, combined with the novel cross-examination evaluation methodology, allows researchers and practitioners to measure hallucinations, assess coverage of information, and understand the trade-offs between model capabilities like conciseness and coverage in healthcare applications. |
| Gated Slot Attention for Efficient Linear-Time Sequence Modeling (Read more on arXiv or HuggingFace) | ExplorerFreda, nealcly, rayzhu16, sonta7, yzhangcs | The paper proposes Gated Slot Attention (GSA), a novel linear attention mechanism for sequence modeling that addresses limitations in recall and training efficiency observed in existing linear attention models. GSA achieves this by enhancing the Attention with Bounded-memory-Control (ABC) model with a gating mechanism, inspired by Gated Linear Attention (GLA). This allows for efficient memory management and context-aware information retrieval. Experiments demonstrate GSA’s superior performance in in-context recall-intensive tasks and its effectiveness in “finetuning pretrained Transformers to RNNs” (T2R), making it a practical alternative for AI practitioners working with large-scale language models and seeking efficient inference and training. GSA’s efficient training and inference, coupled with its strong performance in recall-intensive tasks, make it a compelling alternative for AI engineers and data scientists working with large-scale language models. |
| Agent Workflow Memory (Read more on arXiv or HuggingFace) | Daniel Fried, gneubig, Jiayuan, zorawang | The paper introduces Agent Workflow Memory (AWM), a method to enhance the performance of language model-based agents on complex, long-horizon tasks. AWM induces reusable task workflows from past agent experiences and integrates them into the agent’s memory to guide future action generation. Experiments on web navigation benchmarks, WebArena and Mind2Web, demonstrate that AWM significantly improves task success rates and exhibits strong generalization ability across tasks, websites, and domains. Notably, AWM achieves a 51.1% relative increase in success rate on WebArena compared to the best published autonomous agent. This research is particularly relevant to AI practitioners developing agents for real-world applications, as AWM offers a mechanism for agents to learn and adapt from their experiences, potentially leading to more robust and efficient task-solving capabilities. |
| gsplat: An Open-Source Library for Gaussian Splatting (Read more on arXiv or HuggingFace) | Vickie Ye, akanazawa, zhypan, brentyi, ruilongli | “gsplat: An Open-Source Library for Gaussian Splatting” introduces a novel library for training and developing Gaussian Splatting models. gsplat features a user-friendly PyTorch front-end and highly optimized CUDA back-end, offering improvements to optimization speed, memory efficiency, and convergence times. Experimental results demonstrate that gsplat achieves comparable rendering performance to the original 3DGS implementation while significantly reducing training time and memory usage. The library’s modular API and support for various densification strategies, pose optimization, depth rendering, and anti-aliasing techniques make it a valuable tool for researchers and practitioners working with 3D scene reconstruction and novel view synthesis. AI engineers and data scientists can leverage gsplat to efficiently develop and deploy Gaussian Splatting models for applications like virtual reality, augmented reality, and robotics. |
| Hi3D: Pursuing High-Resolution Image-to-3D Generation with Video Diffusion Models (Read more on arXiv or HuggingFace) | Ting Yao, Yingwei Pan, Yang Chen, Haibo Yang, GiantBision | The paper proposes Hi3D, a novel two-stage video diffusion-based framework for high-resolution image-to-3D generation. Hi3D leverages the temporal consistency of pre-trained video diffusion models to enhance multi-view consistency in 3D generation, addressing limitations of previous 2D diffusion-based methods. The first stage generates low-resolution multi-view images conditioned on camera pose, while the second stage refines these images to higher resolution with finer details using a 3D-aware video-to-video refiner incorporating depth information. Hi3D achieves state-of-the-art performance on novel view synthesis and single-view reconstruction tasks, demonstrating its ability to generate high-fidelity 3D meshes with detailed textures. Practitioners, such as AI engineers and data scientists, can utilize Hi3D to generate high-quality 3D content from single images for various applications, including virtual reality, 3D film production, and more. |
| Can Large Language Models Unlock Novel Scientific Research Ideas? (Read more on arXiv or HuggingFace) | Asif Ekbal, Vinayak-goyal, TirthankarSlg, sandeep123 | This study investigates the potential of large language models (LLMs) in generating novel scientific research ideas. The authors evaluate four LLMs (Claude-2, Gemini, GPT-3.5, and GPT-4) across five scientific domains using a novel dataset and two proposed metrics: Idea Alignment Score (IAScore) and Idea Distinctness Index. The findings indicate that LLMs exhibit domain-specific strengths in idea generation, with Claude and GPT-4 outperforming others. While LLMs demonstrate the ability to generate novel research ideas, human evaluation reveals that they also produce a significant number of non-novel and generic ideas. This research provides valuable insights for AI practitioners, particularly AI engineers and data scientists, interested in leveraging LLMs for accelerating scientific innovation. The proposed metrics and datasets can serve as a foundation for further research in this domain, encouraging the development of new techniques to enhance the novelty and applicability of LLM-generated research ideas. |
| Instant Facial Gaussians Translator for Relightable and Interactable Facial Rendering (Read more on arXiv or HuggingFace) | Hongyang Lin, Daluuu, DolphinQiao, Haaribo, dafeiqin | This paper introduces TransGS, a novel method leveraging diffusion transformers to rapidly convert Physically Based Rendering (PBR) facial assets into high-quality, relightable, and interactable 3D Gaussian Splatting (3DGS) representations. This approach bridges the gap between traditional offline and online rendering by enabling real-time performance (5 seconds generation time) with comparable visual quality to offline techniques. Key innovations include the GauFace representation, optimized for efficient rendering and animation of facial assets, and a novel Pixel Aligned Sampling scheme for constrained, generative-friendly Gaussian distribution. This work offers AI engineers and data scientists a powerful tool for creating dynamic and interactive digital avatars across various platforms, including PCs, mobile devices, and VR headsets. |
| MVLLaVA: An Intelligent Agent for Unified and Flexible Novel View Synthesis (Read more on arXiv or HuggingFace) | Ke Lu, Guohong Hu, Xing Lan, Jian Xue, Hanyu Jiang | This paper introduces MVLLaVA, a novel intelligent agent for synthesizing novel views by integrating multiple multi-view diffusion models with a large multimodal model, LLaVA. The key innovation lies in the design of task-specific instruction templates that enable MVLLaVA to handle a wide range of user instructions, including single images, captions, and specific viewpoint changes. Experimental results demonstrate that MVLLaVA achieves state-of-the-art performance in accurately recognizing and executing novel view synthesis tasks from diverse input modalities. This work holds significant relevance for AI practitioners, especially those interested in 3D content creation, as it offers a robust and versatile solution for generating consistent multi-view images from flexible user inputs. |
| Self-Harmonized Chain of Thought (Read more on arXiv or HuggingFace) | Wei Lu, Ziqi Jin | This research paper, “Self-Harmonized Chain of Thought” by Wei Lu and Ziqi Jin, proposes a novel method called ECHO to improve chain-of-thought prompting in large language models. ECHO enhances the quality of demonstrations in the chain-of-thought process by unifying their diversity, leading to a more coherent and effective reasoning pattern. The method outperforms existing techniques, matching the performance of Few-shot-CoT but without requiring manual effort. ECHO’s ability to automatically generate high-quality demonstrations makes it a valuable tool for practitioners, such as AI engineers and data scientists, who aim to improve the reasoning capabilities of large language models for various downstream applications. |
| ProteinBench: A Holistic Evaluation of Protein Foundation Models (Read more on arXiv or HuggingFace) | Dongyu Xue, Zaixiang Zheng, Fei Ye, thughost, zhouxiangxin | The research paper introduces ProteinBench, a comprehensive evaluation framework designed to assess the capabilities of protein foundation models. ProteinBench comprises a taxonomy of generative tasks in protein science, a multi-metric evaluation approach assessing quality, novelty, diversity, and robustness, and in-depth analyses from various user perspectives. The evaluation reveals that language models excel in capturing natural evolutionary distributions, while structure-based models demonstrate greater robustness in de novo protein design. Additionally, current conformation prediction models show promise but still lag behind classic molecular dynamics simulations in accurately capturing protein dynamics. These findings provide valuable insights for AI engineers and data scientists working with protein foundation models, guiding model selection based on specific design objectives and highlighting areas requiring further development. |
| VMAS: Video-to-Music Generation via Semantic Alignment in Web Music Videos (Read more on arXiv or HuggingFace) | Heng Wang, Linjie Yang, Yu Tian, Yan-Bo Lin, gberta | This paper introduces VMAS, a novel framework for generating background music from video input. VMAS leverages a generative video-music Transformer trained on DISCO-MV, a newly curated dataset of 2.2 million video-music pairs sourced from the Web, which is significantly larger than prior datasets used for this task. The authors propose a video-music alignment scheme, comprising contrastive video-music matching and video-beat alignment, to ensure generated music aligns with high and low-level visual cues. Experimental results demonstrate that VMAS outperforms existing methods in various music generation metrics, including human evaluation. This work provides AI practitioners, particularly those interested in generative AI and multimedia applications, with a new framework and dataset for developing robust and high-quality video-to-music generation systems. |
| Generative Hierarchical Materials Search (Read more on arXiv or HuggingFace) | Simon Batzner, Sherry Yang, IgorM, danilor, RickWork | The authors propose Generative Hierarchical Materials Search (GenMS), a novel approach for generating novel crystal structures from high-level language instructions. GenMS leverages a hierarchical, multi-modal tree search algorithm that combines a large language model, a diffusion model with a compact crystal representation, and a graph neural network for property prediction. Experiments demonstrate that GenMS outperforms baseline methods in generating unique, valid, and potentially stable crystal structures that satisfy user-specified requirements, achieving a high DFT convergence rate and generating structures with lower formation energy. This framework has significant implications for AI practitioners in materials science, enabling them to efficiently explore a vast design space and accelerate the discovery of novel materials with desired properties through intuitive language-based interfaces. |
Papers for 2024-09-11
| Title | Authors | Summary |
|---|---|---|
| INTRA: Interaction Relationship-aware Weakly Supervised Affordance Grounding (Read more on arXiv or HuggingFace) | Se Young Chun, Agorium, jeeit17 | This research paper introduces INTRA, a novel weakly-supervised affordance grounding framework that leverages representation learning and interaction relationship-guided contrastive learning. Unlike previous approaches relying on paired exocentric and egocentric images, INTRA utilizes only exocentric images and incorporates large language models (LLMs) to understand the complex relationship between interactions. INTRA outperforms prior arts on multiple datasets, including AGD20K, IIT-AFF, CAD, and UMD, demonstrating its superior performance and domain scalability. AI practitioners, such as AI engineers and data scientists, can benefit from INTRA’s ability to ground affordances for novel objects and interactions, potentially leading to improved robot manipulation and scene understanding in diverse environments. The method’s ability to leverage LLMs for enhanced linguistic understanding of interactions offers a new direction for affordance grounding research. |
| LLaMA-Omni: Seamless Speech Interaction with Large Language Models (Read more on arXiv or HuggingFace) | zhangshaolei, Paulmzr, zysgdd, guoshoutao, poeroz | This research paper introduces LLaMA-Omni, a novel model architecture for low-latency, high-quality speech interaction with Large Language Models (LLMs). LLaMA-Omni leverages a speech encoder, a speech adapter, an LLM, and a streaming speech decoder to directly process speech instructions and generate text and speech responses with minimal latency. The researchers also created a new speech instruction dataset, InstructS2S-200K, to train and evaluate the model. Experimental results demonstrate that LLaMA-Omni outperforms existing speech-language models in terms of content and style while achieving a low response latency of 226ms. This work is particularly relevant to AI practitioners working on speech-based applications, such as conversational AI and virtual assistants, as it offers an efficient and effective solution for building seamless speech interfaces powered by LLMs. |
| SongCreator: Lyrics-based Universal Song Generation (Read more on arXiv or HuggingFace) | zy001, kangshiyin, jingchengwu, GK50, maxingaussian | The paper proposes SongCreator, a novel lyrics-based universal song generation system capable of generating high-quality songs with both vocals and accompaniment. The system utilizes a dual-sequence language model (DSLM) with a dynamic bidirectional cross-attention module to capture the interplay between vocal and accompaniment sequences. This architecture, trained using a multi-task learning strategy, enables SongCreator to perform various song generation tasks, including lyrics-to-song, vocals-to-song, and song editing, surpassing previous state-of-the-art methods in several tasks. The authors highlight the potential of SongCreator to become a powerful tool for content creators and musicians, lowering the barrier of entry for novices while streamlining the workflow for experienced producers. However, they acknowledge the potential risks associated with replicating voices and emphasize the need for responsible development, choosing not to release the fully trained models. |
| Draw an Audio: Leveraging Multi-Instruction for Video-to-Audio Synthesis (Read more on arXiv or HuggingFace) | Pengfei Gao, Xing Nie, Binjie Mao, MarkWang, YannQi | This research paper introduces Draw an Audio, a novel framework for video-to-audio synthesis that utilizes multi-instruction control to address limitations in content consistency, temporal synchronization, and loudness control observed in prior art. The authors leverage masked attention and time-loudness modules to enable granular control over audio generation guided by user-provided masks and loudness signals. Experimental validation on AudioCaps and VGGSound-Caption datasets demonstrates Draw an Audio’s superior performance in generating high-fidelity audio synchronized with video content. This research is highly relevant to practitioners, such as AI engineers and data scientists, working on applications requiring realistic and controllable sound generation from video data, including foley design, video editing, and multimodal content creation. |
| SaRA: High-Efficient Diffusion Model Fine-tuning with Progressive Sparse Low-Rank Adaptation (Read more on arXiv or HuggingFace) | Yabiao Wang, Ran Yi, Jiangning Zhang, Teng Hu, hongruihuang | This research paper introduces SaRA, a novel parameter-efficient fine-tuning technique designed to enhance the capabilities of pre-trained diffusion models for downstream tasks. The core of SaRA lies in selectively fine-tuning a subset of parameters with the smallest absolute values in the pre-trained model, exploiting their potential effectiveness. To mitigate overfitting due to the high representation ability of sparse matrices, SaRA employs a nuclear-norm-based low-rank loss, constraining the rank of learned sparse matrices. Furthermore, a progressive parameter adjustment strategy is introduced to enhance the utilization of initially ineffective parameters. Experimental results across various tasks, including backbone fine-tuning, downstream dataset fine-tuning, image customization, and controllable video generation, demonstrate that SaRA achieves superior performance compared to state-of-the-art parameter efficient fine-tuning methods, while effectively preserving the model’s prior knowledge. This method is particularly relevant to AI practitioners as it provides an efficient and effective way to adapt pre-trained diffusion models for specific tasks, offering both enhanced performance and reduced memory footprint during training. |
Papers for 2024-09-10
| Title | Authors | Summary |
|---|---|---|
| Towards a Unified View of Preference Learning for Large Language Models: A Survey (Read more on arXiv or HuggingFace) | hhhllan, ZefanCai, instro, songff, KbsdJames | This survey paper presents a unified framework for preference learning in large language models (LLMs), categorizing techniques based on data source, feedback mechanism, and optimization algorithm. The authors argue that existing categorizations based on reinforcement learning (RL) versus supervised fine-tuning (SFT) or online versus offline settings create artificial barriers, as core objectives are similar and algorithms can be decoupled from data acquisition strategies. The paper further details prevalent pointwise, pairwise, and listwise preference optimization methods, alongside training-free alignment approaches, highlighting their loss function designs. This comprehensive overview provides valuable insights for AI engineers and data scientists, facilitating understanding of the relationships between various alignment techniques and potentially enabling more effective development of human-aligned LLMs. |
| MMEvol: Empowering Multimodal Large Language Models with Evol-Instruct (Read more on arXiv or HuggingFace) | Wa2erGo, iiiiwis, tnlin, lzchen2001, haonanzhang | MMEvol, a novel framework for evolving image-text instruction data, is introduced to enhance the capabilities of Multimodal Large Language Models (MLLMs). The authors identify data quality and diversity limitations in existing MLLM datasets and propose an iterative evolution process encompassing fine-grained perceptual, cognitive reasoning, and interactive evolutions, coupled with instruction elimination to filter inadequate samples. Experiments demonstrate that their MLLM trained on evolved data significantly surpasses open-source alternatives across 13 vision-language benchmarks. This work holds significant implications for AI practitioners, highlighting the importance of high-quality instruction data for developing robust MLLMs with improved reasoning, instruction following, and reduced hallucination susceptibility. |
| OneGen: Efficient One-Pass Unified Generation and Retrieval for LLMs (Read more on arXiv or HuggingFace) | huajunsir, square0083, xiangchen-dvi, sunmengshu, MikeDean | The research paper introduces OneGen, a novel framework designed to unify generation and retrieval tasks within a single Large Language Model (LLM). OneGen bridges the traditionally separate training paradigms of generation and retrieval by leveraging retrieval tokens generated autoregressively, enabling a single LLM to handle both tasks concurrently. Empirical evaluations across single-hop and multi-hop question answering, and entity linking demonstrate that OneGen outperforms pipeline solutions and, where applicable, prior single-model methods like GRIT. Moreover, the paper highlights OneGen’s efficiency in training and inference, requiring less data and achieving faster inference speeds, particularly with increased retrieval frequency. Practitioners, including AI engineers and data scientists, can benefit from OneGen’s simplified deployment, reduced computational costs, and improved efficiency, particularly in applications demanding seamless integration of retrieval and generation within LLMs. |
| MemoRAG: Moving towards Next-Gen RAG Via Memory-Inspired Knowledge Discovery (Read more on arXiv or HuggingFace) | Zhicheng Dou, Kelong Mao, Zheng Liu, Hongjin Qian, namespace-Pt | This research paper introduces MemoRAG, a novel Retrieval-Augmented Generation (RAG) system designed to address challenges related to complex tasks involving extensive input contexts. MemoRAG leverages a memory module to create a global memory of the entire database and uses it to generate contextually relevant clues for accurate answer retrieval. Experimental results demonstrate that MemoRAG surpasses existing RAG systems and other baselines across a range of tasks, including knowledge-intensive QA and summarization. MemoRAG’s ability to effectively manage complex and lengthy texts, such as financial reports and legal contracts, by handling contexts of up to one million tokens and resolving intricate queries with high accuracy, makes it particularly valuable for AI practitioners working with large-scale text processing and retrieval applications. |
| Benchmarking Chinese Knowledge Rectification in Large Language Models (Read more on arXiv or HuggingFace) | huajunsir, Ningyu, cowTodd, JizhanFang, TianheLu | The authors introduce CKnowEdit, a novel dataset designed for evaluating and improving Chinese knowledge rectification in Large Language Models (LLMs). This dataset addresses a significant gap in the field, as prior knowledge editing research has primarily focused on English text and often fails to capture the nuances of the Chinese language. Evaluations of existing knowledge editing methods on CKnowEdit reveal limitations in their ability to accurately and consistently rectify Chinese knowledge, highlighting the need for more sophisticated techniques. This work has significant implications for practitioners, as it provides a valuable resource for developing and evaluating Chinese-specific knowledge editing tools, ultimately leading to more reliable and culturally-sensitive LLMs for Chinese language applications. |
| UniDet3D: Multi-dataset Indoor 3D Object Detection (Read more on arXiv or HuggingFace) | Anna Vorontsova, ktoshik, filapro, barracuda049, maksimko123 | This paper introduces UniDet3D, a novel 3D object detection model trained on a mixture of indoor datasets to address the limitations of existing models trained on individual, insufficiently diverse datasets. UniDet3D leverages a unified label space across datasets and employs a simple yet effective architecture based on a vanilla transformer encoder without positional encoding or cross-attention. The key innovation of UniDet3D lies in its ability to generalize to various indoor environments and achieve state-of-the-art results across six indoor benchmarks, outperforming existing methods in both accuracy and efficiency. This advancement is particularly relevant to practitioners, such as AI engineers and data scientists, as UniDet3D offers a robust and customizable solution for indoor 3D object detection that can be readily adapted to various applications and computational constraints. |
| POINTS: Improving Your Vision-language Model with Affordable Strategies (Read more on arXiv or HuggingFace) | Xiao Zhou, Le Tian, Zeon-Zhuang, scyr, YuanLiuuuuuu | The authors introduce POINTS, a novel vision-language model that achieves state-of-the-art performance while utilizing a relatively small pre-training dataset and a publicly available visual instruction tuning dataset. Key innovations include the use of perplexity to filter the pre-training dataset, retaining only the top 20% of data with the lowest perplexity values, leading to significant performance improvements. Additionally, the authors propose “greedy model soup,” a technique that averages the weights of models fine-tuned with varying dataset quantities and diversities, further enhancing performance. POINTS’ effectiveness, coupled with its reliance on publicly available datasets, makes it a valuable tool for practitioners, including AI engineers and data scientists, seeking to develop and deploy robust vision-language models with constrained resources. The authors’ meticulous ablation studies and detailed analysis of each component contribute to the model’s transparency and ease of adoption. |
| Open Language Data Initiative: Advancing Low-Resource Machine Translation for Karakalpak (Read more on arXiv or HuggingFace) | murodbek, mukhammadsaid | This research presents advancements in low-resource machine translation, specifically focusing on the Karakalpak language. The authors introduce a new FLORES+ devtest dataset translated into Karakalpak and develop parallel corpora for Uzbek-Karakalpak, Russian-Karakalpak, and English-Karakalpak language pairs. Utilizing these resources, they train and evaluate several neural machine translation models, demonstrating the effectiveness of incorporating data from related Turkic languages. The resulting models and datasets provide valuable resources for AI practitioners interested in developing NLP applications for Karakalpak and similar low-resource languages. |
| Paper Copilot: A Self-Evolving and Efficient LLM System for Personalized Academic Assistance (Read more on arXiv or HuggingFace) | Ge Liu, Pengrui Han, youjiaxuan, taofeng, cmulgy | This paper introduces Paper Copilot, a large language model (LLM) system designed to provide personalized and efficient academic research assistance. Paper Copilot employs thought retrieval, user profile generation, and high-performance optimization techniques to deliver its services. The system demonstrates a significant reduction in time required for information retrieval (69.92%) compared to traditional methods. Moreover, user feedback indicates a strong preference for the self-evolving capabilities of the system, highlighting its potential as a valuable tool for researchers. This is highly relevant to AI practitioners, particularly those involved in natural language processing, as it showcases the application of advanced techniques like thought retrieval and efficient deployment strategies for real-world use cases in information retrieval and knowledge management. |
| Insights from Benchmarking Frontier Language Models on Web App Code Generation (Read more on arXiv or HuggingFace) | Yi Cui | This research paper presents an analysis of 16 large language models (LLMs) evaluated on WebApp1K, a benchmark designed to assess code generation capabilities for web applications. The key finding suggests that despite exhibiting similar knowledge levels, the performance difference among models stems from the varying frequency of errors. Notably, the study reveals that generating correct code exhibits higher complexity compared to producing incorrect code. Moreover, prompt engineering, while effective in specific scenarios, shows limited impact in overall error reduction. These insights are crucial for practitioners, particularly AI engineers and data scientists, highlighting the importance of prioritizing model reliability and minimizing mistakes during the development of coding LLMs. |
| Evaluating Multiview Object Consistency in Humans and Image Models (Read more on arXiv or HuggingFace) | Kanwisher, tgoconnell, Emma02, stephaniefu, tzler | The research introduces MOCHI, a novel benchmark for evaluating the alignment between human perception and computer vision models on 3D shape inference tasks. Using a “same/different” object identification task with varying viewpoints, the study reveals that while humans significantly outperform models like DINOv2, CLIP, and MAE, a correlation exists between human and model performance. Further analysis of human reaction time and gaze patterns suggests that humans achieve superior performance by dedicating more processing time and employing flexible attention mechanisms, which current models lack. This benchmark provides crucial insights for AI practitioners, highlighting the need for models to incorporate mechanisms for dynamic processing and flexible attention to achieve more human-like 3D shape understanding. |
Papers for 2024-09-09
| Title | Authors | Summary |
|---|---|---|
| How Do Your Code LLMs Perform? Empowering Code Instruction Tuning with High-Quality Data (Read more on arXiv or HuggingFace) | mdizhang, bitwjg, dongguanting, fudayuan, banksy235 | The authors propose XCoder, a family of large language models (LLMs) fine-tuned from LLaMA3 using a novel data selection strategy for code instruction tuning. Recognizing the limitations of existing code instruction datasets, often plagued by data leakage and inconsistent quality, the authors introduce a three-pronged data assessment approach. This approach prioritizes instruction complexity, response quality (evaluated through a unit test model), and instruction diversity to curate a high-quality training dataset. Experimental results demonstrate that XCoder surpasses or matches state-of-the-art open-source code LLMs on benchmarks like HumanEval and LiveCodeBench, even with significantly fewer training samples. This research offers AI practitioners valuable insights into constructing and leveraging high-quality code instruction datasets for enhanced code generation and understanding. |
| Configurable Foundation Models: Building LLMs from a Modular Perspective (Read more on arXiv or HuggingFace) | fengyao1909, thuzhizhi, Raincleared, ZhengyanZhang, xcjthu | This research paper proposes the novel concept of “configurable foundation models,” which are built upon modular components termed “bricks,” offering a modular perspective on large language model (LLM) construction and deployment. The paper categorizes bricks as either “emergent,” arising from the pre-training process, or “customized,” manually designed for specific post-training tasks, and outlines four key brick-oriented operations: routing and retrieval, combination, updating, and growing. Empirical analysis on decoder-only models, Llama-3-8B-Instruct and Mistral-7B-Instruct-v0.3, reveals sparse neuron activation, functionality specialization, and potential for modular partitioning. These findings hold significant implications for AI practitioners, suggesting that LLM efficiency and scalability can be improved by leveraging modularity through selective brick activation, facilitating continual learning, and enabling distributed computation. |
| Open-MAGVIT2: An Open-Source Project Toward Democratizing Auto-regressive Visual Generation (Read more on arXiv or HuggingFace) | Yujiu Yang, yshan2u, yxgeee, shifengyuan, RobertLuo1 | This research paper introduces Open-MAGVIT2, an open-source family of auto-regressive image generation models. The authors replicate Google’s MAGVIT-v2 tokenizer, achieving state-of-the-art reconstruction performance on ImageNet by utilizing a super-large codebook with lookup-free quantization. To address the challenges of auto-regressive prediction with such a large vocabulary, they propose “next sub-token prediction” with asymmetric token factorization, improving generation quality. Open-MAGVIT2 demonstrates superior performance in both visual reconstruction and class-conditional generation using a plain auto-regressive approach. The release of these models and code provides AI practitioners with a powerful toolset for advancing auto-regressive visual generation, particularly within unified multimodal frameworks. |
| Qihoo-T2X: An Efficiency-Focused Diffusion Transformer via Proxy Tokens for Text-to-Any-Task (Read more on arXiv or HuggingFace) | Yuhui Yin, Dawei Leng, Jiasong Feng, Jing Wang, AoMa | This research paper introduces PT-DiT, a novel Proxy Token Diffusion Transformer designed for computationally efficient text-to-image and text-to-video generation tasks. PT-DiT leverages the redundancy in visual information by utilizing a sparse proxy token attention mechanism, wherein a select set of representative tokens, sampled based on spatio-temporal priors, model global visual relationships. To further enhance texture detail, the model incorporates window attention and shift-window attention modules. Experimental results demonstrate that PT-DiT achieves performance comparable to state-of-the-art methods while significantly reducing computational complexity and memory usage, making it particularly beneficial for high-resolution image and video generation. This efficiency gain makes PT-DiT and the Qihoo-T2X family of models valuable tools for AI practitioners, particularly AI engineers and data scientists working on resource-intensive generative tasks. |
| GST: Precise 3D Human Body from a Single Image with Gaussian Splatting Transformers (Read more on arXiv or HuggingFace) | Christian Rupprecht, Joao F. Henriques, Lorenza Prospero, ajhamdi | The paper introduces Gaussian Splatting Transformers (GST), a novel method for reconstructing 3D human models from monocular images using Gaussian Splatting representations. GST leverages a transformer architecture trained solely on multi-view supervision, eliminating the need for expensive 3D annotations or diffusion priors. Experiments demonstrate that GST achieves competitive performance on 3D human pose estimation and novel view synthesis tasks. This efficient and accurate approach holds significant potential for practitioners in various domains, including virtual reality, augmented reality, and human-computer interaction, by enabling real-time 3D human modeling from readily available data sources. |
Papers for 2024-09-06
| Title | Authors | Summary | Link |
|---|---|---|---|
| Attention Heads of Large Language Models: A Survey | Yezhaohui Wang, jimi888, Ki-Seki, saythe17, fan2goa1 | This paper surveys recent research on attention heads in Large Language Models (LLMs) and their role in reasoning processes. The authors propose a novel four-stage framework, inspired by human cognition, to categorize attention head functions: Knowledge Recalling, In-Context Identification, Latent Reasoning, and Expression Preparation. Furthermore, the paper summarizes experimental methodologies for investigating attention head mechanisms, categorized as Modeling-Free and Modeling-Required approaches. This survey provides AI practitioners with a valuable resource for understanding the inner workings of LLMs, potentially enabling them to design more interpretable and effective models, and develop novel techniques for LLM analysis and improvement. | Read more on HF |
| FuzzCoder: Byte-level Fuzzing Test via Large Language Model | Challenging666, Pony12, zhangysk, ngl567, WeiSumi | This paper introduces FUZZCODER, a novel fuzzing framework leveraging fine-tuned large language models (LLMs) for enhanced vulnerability detection in software. FUZZCODER employs a sequence-to-sequence paradigm, trained on a purpose-built “Fuzz-Instruct” dataset, to predict vulnerable byte locations and effective mutation strategies within input files. Evaluations on the custom Fuzz-Bench benchmark demonstrate FUZZCODER’s superiority over traditional methods, achieving higher effective proportions of mutation (EPM) and uncovering a greater number of program crashes, indicative of potential vulnerabilities. These findings highlight the potential of LLMs in advancing fuzzing techniques, offering a valuable tool for AI engineers and data scientists involved in software security testing and vulnerability analysis. | Read more on HF |
| CDM: A Reliable Metric for Fair and Accurate Formula Recognition Evaluation | conghui, BoZhang, renqiux0302, ouyanglinke, wanderkid | This research paper proposes a novel evaluation metric called Character Detection Matching (CDM) for formula recognition tasks. Addressing the limitations of existing text-based metrics like BLEU, CDM evaluates formula recognition by comparing rendered images of predicted and ground-truth formulas, utilizing visual character matching. Experiments demonstrate that CDM offers a more accurate and fairer assessment of formula recognition models, particularly in scenarios with diverse formula representations. Notably, the study shows that by using CDM for training data selection, comparable model performance can be achieved using only a fraction (less than 20%) of the data. This finding offers valuable insights for practitioners, such as AI engineers and data scientists, enabling more efficient model training and dataset construction in the field of formula recognition. | Read more on HF |
| mPLUG-DocOwl2: High-resolution Compressing for OCR-free Multi-page Document Understanding | Liang Zhang, Jingren, hzhwcmhf, xhyandwyy, AnwenHu | mPLUG-DocOwl2 is a novel Multimodal Large Language Model (MLLM) designed for efficient OCR-free multi-page document understanding. The authors introduce a High-resolution DocCompressor module that leverages cross-attention with global visual features to effectively compress high-resolution document images into a fixed number of tokens (324). This approach reduces computational overhead and inference time while maintaining comparable performance to state-of-the-art MLLMs on various document understanding benchmarks. DocOwl2’s ability to process high-resolution images and efficiently extract textual information is beneficial for practitioners, such as AI engineers and data scientists, developing applications for multi-page document analysis, question answering, and information retrieval. The reduction in computational resources required for processing high-resolution images makes DocOwl2 particularly relevant for real-world applications. | Read more on HF |
| Geometry Image Diffusion: Fast and Data-Efficient Text-to-3D with Image-Based Surface Representation | simondonn, CiaraRowles, SlavaElizarov | This research introduces Geometry Image Diffusion (GIMDiffusion), a novel Text-to-3D framework that leverages geometry images as the 3D representation. By employing a Collaborative Control scheme with a pre-trained Text-to-Image diffusion model, GIMDiffusion generates 3D objects with high fidelity and diversity from text prompts, eliminating the need for complex 3D-aware architectures. Results demonstrate its capability to produce relightable 3D assets efficiently, comparable to existing Text-to-Image methods. GIMDiffusion offers a practical and efficient approach for AI practitioners, particularly AI Engineers and Data Scientists, working in 3D content creation, as it simplifies both model design and training while leveraging existing resources. Furthermore, the generated objects consist of semantically meaningful, separable parts, enhancing their usability and versatility for tasks such as editing and animation. | Read more on HF |
| WildVis: Open Source Visualizer for Million-Scale Chat Logs in the Wild | Xiang Ren, Wenting Zhao, yejinchoinka, jmhessel, yuntian-deng | WILDVIS is an open-source interactive tool designed for the exploration and analysis of large-scale conversational datasets, particularly interactions between users and chatbots. The tool employs both filter-based retrieval and embedding-based visualization techniques to enable efficient navigation and pattern discovery within millions of conversations. WILDVIS allows for the application of various filters, including keywords, user demographics, and conversation topics, to refine searches and highlight relevant conversations within an embedding space. For AI engineers and data scientists, WILDVIS offers a valuable resource for understanding user behavior, identifying potential misuse of chatbots, and uncovering insights into conversation dynamics within large datasets. The tool’s ability to visualize topic distributions across datasets can be particularly beneficial for researchers studying trends in user-chatbot interactions. | Read more on HF |
| From MOOC to MAIC: Reshaping Online Teaching and Learning through LLM-driven Agents | juanli, Lin-23457, zhanxinhao, tsq2000, JovanYu | This paper introduces MAIC (Massive AI-empowered Course), a novel online education paradigm leveraging LLM-driven multi-agent systems to enhance the scalability and adaptivity of online learning. MAIC employs AI agents for course preparation, instruction delivery, and student interaction, aiming to provide personalized learning experiences. Preliminary experimental results demonstrate the effectiveness of MAIC in enhancing script generation quality, promoting student engagement, and improving learning outcomes. These findings hold significant implications for AI practitioners, particularly in the domain of educational technology, by showcasing the potential of LLMs and multi-agent systems in revolutionizing online education. | Read more on HF |
| Guide-and-Rescale: Self-Guidance Mechanism for Effective Tuning-Free Real Image Editing | Dmitry Vetrov, Madina Khalmatova, ai-alanov, sashapff, macderru | The paper, “Guide-and-Rescale: Self-Guidance Mechanism for Effective Tuning-Free Real Image Editing”, introduces a novel image editing method called Guide-and-Rescale. This method leverages a self-guidance technique within a diffusion model framework to balance high-quality editing with the preservation of the original image structure. The authors achieve this by introducing energy functions, referred to as “guiders,” designed to maintain both global layout and local visual characteristics during the editing process. The paper presents a noise rescaling mechanism, ensuring consistent behavior across a diverse range of images, and demonstrates its effectiveness through both qualitative and quantitative analysis on various editing tasks, such as changing object appearance, style transfer, and image manipulation. Practitioners, including AI engineers and data scientists, can utilize this method for real-time, high-fidelity image editing applications without the need for extensive model fine-tuning or computationally expensive inversion processes. | Read more on HF |
| FrozenSeg: Harmonizing Frozen Foundation Models for Open-Vocabulary Segmentation | Hongxun Yao, Xi Chen, Xiatian-Zhu, ShengJin, happy0612 | This paper introduces FrozenSeg, a novel open-vocabulary segmentation method that addresses the limitation of existing methods in generating accurate mask proposals for unseen categories. FrozenSeg leverages the strengths of frozen foundation models, specifically CLIP for semantic understanding and SAM for spatial reasoning, via two novel modules: Query Injector and Feature Injector. Experiments demonstrate FrozenSeg’s state-of-the-art performance in open-vocabulary semantic, instance, and panoptic segmentation across multiple datasets, with significant improvements over baselines. This method holds promise for AI practitioners seeking to develop segmentation models capable of generalizing to unseen categories and scenarios without extensive retraining. | Read more on HF |
| Report Cards: Qualitative Evaluation of Language Models Using Natural Language Summaries | Jimmy Ba, Keiran Paster, Fuyang Cui, spitis, loveblairsky | This paper introduces Report Cards, a novel approach for qualitative assessment of Large Language Models (LLMs), addressing the limitations of purely quantitative benchmarks. Report Cards provide human-interpretable natural language summaries of an LLM’s capabilities across specific skills or topics, offering nuanced insights into model behavior. The authors propose an iterative method, PRESS, for generating these report cards and introduce metrics for evaluating their specificity, faithfulness, and interpretability. Experimental results demonstrate that Report Cards can effectively differentiate between models, accurately reflect their capabilities, and provide valuable insights for practitioners like AI engineers and data scientists, who can leverage these summaries for understanding model strengths and weaknesses. This work contributes a valuable tool for holistic and interpretable evaluation of LLMs, moving beyond simplistic quantitative metrics. | Read more on HF |
Papers for 2024-09-05
| Title | Authors | Summary | Link |
|---|---|---|---|
| LongLLaVA: Scaling Multi-modal LLMs to 1000 Images Efficiently via Hybrid Architecture | Benyou Wang, Chen Zhang, Shunian Chen, Xidong Wang, songdj | The paper introduces LongLLaVA, a novel hybrid multi-modal large language model (MLLM) designed for efficient long-context understanding. By integrating Mamba and Transformer blocks, LongLLaVA effectively handles temporal and spatial dependencies among multiple images, achieving competitive performance on benchmarks like MileBench and Video-MME. Notably, LongLLaVA requires significantly fewer FLOPs compared to other models while demonstrating strong in-context learning capabilities. This efficiency and performance make LongLLaVA a valuable tool for AI practitioners, particularly in applications involving video understanding, high-resolution image processing, and multi-modal agents. | Read more on HF |
| Loopy: Taming Audio-Driven Portrait Avatar with Long-Term Motion Dependency | Gaojie Lin, Jiaqi Yang, Chao Liang, tianyumyum, janphu | This paper introduces LOOPY, an end-to-end audio-driven portrait video generation framework that generates realistic talking head videos solely from audio input, eliminating the reliance on spatial motion templates used in previous methods. LOOPY leverages inter- and intra-clip temporal modules to model long-term motion dependencies and an audio-to-motion latents module for effective audio-portrait motion correlation. Experiments on diverse datasets, including CelebV-HQ and RAVDESS, demonstrate LOOPY’s superior performance in generating temporally stable, expressive, and high-quality talking head videos, surpassing existing state-of-the-art methods. Practitioners, including AI engineers and data scientists, can utilize LOOPY to develop robust and realistic talking head generation systems for various applications, such as virtual assistants, video conferencing, and entertainment. The removal of spatial constraints and the ability to learn natural motion patterns from audio make LOOPY a significant advancement in audio-driven video synthesis. | Read more on HF |
| LongCite: Enabling LLMs to Generate Fine-grained Citations in Long-context QA | LZDQ, Broccolito, davidlvxin, bys0318, NeoZ123 | This research paper introduces LongCite, a system designed to enhance the trustworthiness of Large Language Models (LLMs) by enabling them to provide fine-grained citations within their long-form answers. The authors identify the limitations of current LLMs in providing adequate citations for long-context question answering (LQAC) and propose a novel pipeline called CoF (Coarse to Fine) to automatically construct a large-scale LQAC dataset, LongCite-45k. By fine-tuning existing open-source long-context models on this dataset, they demonstrate significant improvements in citation quality, even surpassing proprietary models like GPT-40. This advancement holds practical significance for AI practitioners, particularly AI engineers and data scientists, by equipping LLMs with enhanced transparency and verifiability, making them more reliable for various applications. | Read more on HF |
| MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmark | btyu, jamessyx, yuanshengni, aaabiao, yuexiang96 | The research paper introduces MMMU-Pro, a novel benchmark designed to rigorously evaluate the multimodal reasoning capabilities of large language models. MMMU-Pro addresses limitations in existing benchmarks by incorporating three key enhancements: filtering out questions solvable by text-only models, augmenting candidate options to mitigate guessing, and introducing a vision-only input setting to assess genuine multimodal understanding. Experimental results demonstrate significant performance drops across a variety of state-of-the-art multimodal models, indicating that MMMU-Pro poses a more realistic challenge. This benchmark provides AI practitioners, including AI engineers and data scientists, with a valuable tool for assessing and improving the robustness and reliability of multimodal systems, particularly in real-world scenarios where text and images are intertwined. | Read more on HF |
| Arctic-SnowCoder: Demystifying High-Quality Data in Code Pretraining | rajhans-snowflake, stovecat, yuxiang630 | Arctic-SnowCoder-1.3B is a new, high-performing code language model trained on 555B tokens utilizing a novel three-step methodology of progressively refined data quality. This model outperforms StarCoderBase-3B on all benchmarks despite being trained with significantly less data and achieves state-of-the-art results on BigCodeBench compared to similarly sized models. The authors demonstrate that aligning training data distribution with downstream tasks is crucial for effective code pretraining and significantly enhances model performance. These findings and the model itself will be of significant interest to practitioners, especially AI engineers who develop code generation and program synthesis applications. | Read more on HF |
| Political DEBATE: Efficient Zero-shot and Few-shot Classifiers for Political Text | Rachel X. Peng, Ryan Yank Wang, Michael Burnham, kaylakahn | This paper introduces Political DEBATE, a pair of open-source language models specifically designed for efficient zero-shot and few-shot classification of political text. Trained on the novel PolNLI dataset, comprising over 200,000 political documents and 852 unique hypotheses, the models exhibit superior performance compared to existing open-source alternatives across tasks such as stance detection, topic classification, hate-speech identification, and event extraction. The authors demonstrate that with minimal few-shot training (10-25 documents), Political DEBATE achieves comparable or even better accuracy than supervised classifiers and resource-intensive generative LLMs. The availability of these efficient and open-source models presents a valuable resource for practitioners in political science and related fields, enabling accessible and reproducible text analysis. | Read more on HF |
| FastVoiceGrad: One-step Diffusion-Based Voice Conversion with Adversarial Conditional Diffusion Distillation | Yuto Kondo, Hirokazu Kameoka, Takuhiro Kaneko, ououo | This research introduces FastVoiceGrad, a novel one-step diffusion-based voice conversion (VC) model that addresses the slow inference limitation of multi-step diffusion-based VC methods. FastVoiceGrad leverages adversarial conditional diffusion distillation (ACDD), which distills knowledge from a pretrained multi-step teacher diffusion model into a one-step student model using adversarial loss and score distillation loss. Experimental results demonstrate that FastVoiceGrad achieves comparable performance to multi-step models while significantly reducing computational cost, achieving a real-time factor of 0.060 for mel-spectrogram conversion. This development provides AI practitioners, particularly those working on VC applications, a faster and computationally efficient alternative for real-time and resource-constrained scenarios. | Read more on HF |
| Affordance-based Robot Manipulation with Flow Matching | Michael Gienger, Fanzhri | This research paper introduces a novel framework for robot manipulation that leverages prompt tuning and flow matching. The authors propose a parameter-efficient prompt tuning method to adapt pre-trained vision models for affordance learning conditioned on language instructions. They then introduce a flow matching policy, a generative approach that learns to transform random waypoints into desired robot trajectories guided by visual affordances. Experimental results on a constructed real-world dataset of Activities of Daily Living demonstrate that the proposed approach achieves competitive performance in both affordance learning and trajectory generation compared to existing methods. This work presents a promising direction for AI practitioners working on robot manipulation, particularly in scenarios where data efficiency and generalization to multi-task settings are crucial. The integration of prompt tuning facilitates efficient adaptation of large pre-trained models, while the flow matching policy offers a stable and effective approach for generating robot trajectories from visual affordances. | Read more on HF |
Papers for 2024-09-04
| Title | Authors | Summary | Link |
|---|---|---|---|
| Kvasir-VQA: A Text-Image Pair GI Tract Dataset | Andrea Storås, vlbthambawita, stevenah, cise-midoglu, SushantGautam | The paper introduces Kvasir-VQA, an extended dataset derived from HyperKvasir and Kvasir-Instrument datasets, augmented with question-and-answer annotations to facilitate advanced machine learning tasks in GI diagnostics. The dataset comprises 6,500 annotated images spanning various GI tract conditions and surgical instruments, and it supports multiple question types including yes/no, choice, location, and numerical count. Preliminary experiments demonstrate the dataset’s effectiveness in training models for image captioning, VQA, and synthetic image generation. The dataset is designed to bridge the gap between medical image analysis and practical diagnostic tools, ultimately aiming to improve patient outcomes and diagnostic precision. This dataset can be of immense value to AI engineers and data scientists looking to develop robust and accurate AI models for medical image analysis and diagnostics in the GI tract. | Read more on HF |
| OLMoE: Open Mixture-of-Experts Language Models | sewon, jacobmorrison, dirkgr, soldni, Muennighoff | The paper introduces OLMOE, a fully open-source, state-of-the-art Mixture-of-Experts (MoE) language model. This model outperforms other available models with similar active parameters, even surpassing larger models like Llama2-13B-Chat and DeepSeekMoE-16B. The authors present a comprehensive analysis of MoE training and routing, demonstrating how it achieves high specialization and outperforms dense language models on various benchmarks. All aspects of OLMOE are open-sourced, including model weights, training data, code, and logs. This work is highly relevant to practitioners by providing a cost-effective, open-source, high-performing language model for research and development. Moreover, the detailed analysis of MoE design choices provides valuable insights for AI engineers and data scientists working with MoE models. | Read more on HF |
| LongRecipe: Recipe for Efficient Long Context Generalization in Large Languge Models | Laziobird, anhtuanluu36, sheryc, yuliang03181, zhiyuanhucs | This research paper proposes LongRecipe, an efficient training strategy for extending the context window of Large Language Models (LLMs). LongRecipe leverages a novel approach called Impactful Token Analysis to identify key tokens that significantly influence long-text training, enabling the model to learn from shorter text segments while maintaining training efficiency. It also introduces a Position Index Transformation technique to simulate long sequences without needing actual long texts. LongRecipe achieves significant improvements in long-context generalization, demonstrating that it can effectively utilize long sequences while requiring only 30% of the target context window size and reducing computational training resources by over 85% compared to full-sequence training. Moreover, LongRecipe preserves the original LLM’s capabilities in general tasks, making it a balanced approach for enhancing both long-range dependency understanding and foundational model performance. This research contributes to the field of AI by offering practitioners a more efficient and effective method for extending the context window of LLMs, enabling them to handle more complex and challenging tasks that require long-context understanding. | Read more on HF |
| FLUX that Plays Music | huangjunshi, Changqian, MichaelFan, onion | This paper proposes FluxMusic, an extension of diffusion-based rectified flow Transformers for text-to-music generation. It leverages a latent VAE space of mel-spectrograms, incorporating double and single stream blocks to model text and music. The authors demonstrate that FluxMusic outperforms existing methods across multiple metrics, including FAD, IS, and CLAP, demonstrating its scalability and effectiveness. Furthermore, the authors evaluate the impact of model size, rectified flow training, and other hyperparameters on the generative performance. FluxMusic provides a promising avenue for researchers and practitioners in text-to-music generation, offering improved accuracy and scalability compared to previous approaches. | Read more on HF |
| DepthCrafter: Generating Consistent Long Depth Sequences for Open-world Videos | vinthony, walkingshadow, Xiaoyu521, xiangjun0211, wbhu-tc | DepthCrafter, a novel video-depth estimation method, generates temporally consistent long depth sequences for open-world videos using video diffusion models. Unlike previous approaches, it does not require additional information, such as camera poses or optical flow. DepthCrafter achieves this by training a video-to-depth model from a pre-trained image-to-video diffusion model through a three-stage training strategy. The method is evaluated on multiple datasets, outperforming existing approaches in terms of both quantitative and qualitative metrics, demonstrating its effectiveness in generating high-quality depth sequences. Practitioners, such as AI engineers and data scientists, can leverage DepthCrafter for various downstream applications, including depth-based visual effects and conditional video generation. | Read more on HF |
| VideoLLaMB: Long-context Video Understanding with Recurrent Memory Bridges | Yang Liu, zlzheng, cihangxie, ColorfulAI | VideoLLaMB is a new framework that utilizes recurrent memory tokens within bridge layers to encode the entirety of a video sequence, preserving semantic continuity and improving performance across various tasks. The authors introduce a SceneTilling algorithm, which segments videos into independent semantic units. This approach achieves state-of-the-art results across various video QA benchmarks, particularly on longer videos (up to 8x longer) and in the Needle in a Video Haystack (NIAVH) benchmark. VideoLLaMB also enables training-free streaming video captioning and high performance on a single GPU, setting a new foundation for long-form video understanding models. These improvements are particularly relevant to AI practitioners, as they offer a more efficient and effective way to analyze and understand long videos. | Read more on HF |
| Diffusion Policy Policy Optimization | Lars L. Ankile, Allen Z. Ren, daihongkai, pulkitag, jlidard | The research paper “Diffusion Policy Policy Optimization” explores a novel algorithm for fine-tuning diffusion-based policies in robot learning tasks using policy gradient methods. The authors demonstrate that their algorithm, DPPO, outperforms existing methods for diffusion-based policy fine-tuning and achieves strong results in both simulation and real-world robot manipulation tasks. The paper also provides insights into the mechanisms behind DPPO’s success, highlighting its ability to induce structured exploration, maintain training stability, and enhance policy robustness. DPPO could be relevant to practitioners developing robotic systems by providing a robust and efficient method for fine-tuning diffusion-based policies trained on expert demonstrations. | Read more on HF |
| Compositional 3D-aware Video Generation with LLM Director | Anni Tang, bianjiang, leo-guo, deeptimhe, ingzzzz | The paper proposes a novel method for text-to-video generation by explicitly composing concepts in 3D space. The method leverages LLMs to decompose a complex textual prompt into sub-prompts, each describing a specific concept. It then generates 3D representations for each concept using pre-trained expert models. These representations are then composed using priors from multi-modal LLMs and 2D diffusion models. The key results of this method include the generation of high-fidelity videos with diverse motions and the ability to control individual concepts. This research could be relevant to AI engineers and data scientists working on text-to-video generation or who are interested in applying LLMs to 3D graphics or video generation. | Read more on HF |
| LinFusion: 1 GPU, 1 Minute, 16K Image | Xinchao Wang, ZhenXiong, whyu, Huage001 | This research paper presents LinFusion, a novel diffusion model for text-to-image generation that achieves linear time and memory complexity with respect to the number of spatial tokens. The authors achieve this by introducing a generalized linear attention mechanism that serves as a low-rank approximation of popular linear token mixers. Extensive experiments on Stable Diffusion models demonstrate that LinFusion achieves performance on par with or superior to the original SD after only modest training, while significantly reducing training time and memory complexity. LinFusion is highly compatible with pre-trained SD components and can generate high-resolution images like 16K resolution. AI practitioners can leverage this novel model to generate high-resolution images with significantly reduced computational resources. | Read more on HF |
| ContextCite: Attributing Model Generation to Context | Aleksander Madry, krisgrg, harshay, bencw | This research paper introduces the novel task of context attribution, aiming to identify the specific parts of a context responsible for a language model’s generated statement. The paper proposes a scalable and efficient method called CONTEXTCITE, which uses a linear surrogate model to estimate the effect of ablating different parts of the context. The results demonstrate that CONTEXTCITE consistently outperforms existing baselines in identifying relevant sources, particularly for complex tasks like multi-hop question answering and summarization. CONTEXTCITE can be applied by practitioners to verify generated statements, improve response quality by pruning irrelevant context, and detect poisoning attacks in language models. | Read more on HF |
| OD-VAE: An Omni-dimensional Video Compressor for Improving Latent Video Diffusion Model | Qian Wang, Bin Zhu, Bin Lin, Zongjian Li, Liuhan Chen | This research proposes an omni-dimensional video compressor (OD-VAE) to improve the efficiency of latent video diffusion models (LVDMs). Unlike conventional VAEs, OD-VAE compresses videos temporally and spatially, leading to more concise latent representations and reduced computational requirements for LVDMs. The researchers demonstrate that OD-VAE can achieve high video reconstruction accuracy while maintaining high compression speed, improving the training efficiency of LVDMs. The results also suggest that OD-VAE can be used to generate longer videos with limited GPU memory, making it a valuable tool for practitioners working with LVDMs. The paper’s findings have implications for AI engineers and data scientists developing video generation models, offering a way to improve model efficiency and reduce computational costs. | Read more on HF |
| GenAgent: Build Collaborative AI Systems with Automated Workflow Generation – Case Studies on ComfyUI | Lei Bai, Wanli Ouyang, Di Huang, Xiangyuan Xue, whlzy | This research presents GenAgent, a novel LLM-based framework for automating the creation of complex workflows used in collaborative AI systems. The framework utilizes LLMs to represent workflows as code, enabling greater flexibility and scalability compared to monolithic AI models. GenAgent is evaluated on the ComfyUI platform and demonstrates superior performance to baseline methods in generating both run-level and task-level workflows. The key takeaway for practitioners is that GenAgent’s ability to automate workflow generation can significantly improve the efficiency and effectiveness of collaborative AI system development. The framework can be applied to a variety of AI systems and platforms, making it a valuable tool for AI engineers and data scientists. | Read more on HF |
| Follow-Your-Canvas: Higher-Resolution Video Outpainting with Extensive Content Generation | Junkun Yuan, Hongfa Wang, Yue Ma, Qihua Chen, cqf | This research paper presents “Follow-Your-Canvas”, a new method for higher-resolution video outpainting with extensive content generation. The proposed method addresses the limitations of existing video outpainting methods by using a diffusion-based model and dividing the task across spatial windows. By incorporating relative region embedding and a layout encoder, the authors demonstrate that Follow-Your-Canvas can generate high-quality results with improved spatial-temporal consistency. The model significantly outperforms existing methods in both low-resolution and high-resolution scenarios. AI engineers can use this method for a wide range of applications such as improving user experience by generating videos with larger aspect ratios or enhancing the resolution of existing videos. | Read more on HF |
| Density Adaptive Attention-based Speech Network: Enhancing Feature Understanding for Mental Health Disorders | Adrian Kieback, Georgios Ioannides, jsbai-aaron, amanchadha | This research introduces DAAMAudioCNNLSTM and DAAMAudioTransformer, two parameter-efficient and explainable models for audio feature extraction and depression detection. These models leverage the multi-head Density Adaptive Attention Mechanism (DAAM) to dynamically focus on informative speech segments, achieving state-of-the-art performance on the DAIC-WOZ dataset (F1 macro scores of 0.702 and 0.72, respectively). DAAM offers significant explainability benefits by highlighting which features were most informative for diagnosis, making it more transparent and trustworthy. This work could be valuable for practitioners by providing tools for developing more reliable, clinically-useful depression detection models that leverage only audio signals, without relying on supplementary information. | Read more on HF |
| Know When to Fuse: Investigating Non-English Hybrid Retrieval in the Legal Domain | Gerasimos Spanakis, Gijs van Dijck, antoinelouis | This paper investigates the performance of hybrid retrieval methods in the legal domain, specifically in the French language. The authors find that fusing domain-general retrieval models consistently improves performance in zero-shot settings, but in-domain training diminishes the benefits of fusion, suggesting a trade-off between computational resources and accuracy. They also propose a percentile-based score normalization method to address misaligned score distributions across different models, which can improve the effectiveness of fusion. The study highlights the importance of carefully considering the choice of retrieval models and fusion techniques in specialized domains, and provides insights that could be valuable for practitioners working on information retrieval in non-English legal domains. | Read more on HF |
| The MERIT Dataset: Modelling and Efficiently Rendering Interpretable Transcripts | J. Boal, A. Sanchez-Cuadrado, alvlopez, de-Rodrigo | This research introduces the MERIT Dataset, a multimodal (text, image, and layout) dataset of school reports designed for training visually-rich document understanding (VrDU) models. The dataset, comprising over 400 labels and 33k samples, includes realistic digital and photorealistic documents with controlled bias features (such as gender and name origin), enabling the study of bias in language models. The dataset is publicly available and includes a comprehensive generation pipeline for replication. The authors conduct experiments using state-of-the-art LayoutLM models, demonstrating the dataset’s suitability for training and evaluating performance, while showcasing the challenges associated with real-world scenarios. This dataset offers a valuable tool for practitioners in AI engineering and data science, providing a benchmark for developing and evaluating models, especially in the context of bias detection and understanding. | Read more on HF |
Papers for 2024-09-03
| Title | Authors | Summary | Link |
|---|---|---|---|
| VisionTS: Visual Masked Autoencoders Are Free-Lunch Zero-Shot Time Series Forecasters | Xiaoyun Joy Wang, Zhuo Li, twinsken, HALF111, chenmouxiang | This paper introduces VisionTS, a novel zero-shot time series forecasting model that leverages the intrinsic similarities between images and time series. The authors reformulate the forecasting task as an image reconstruction problem, and utilize a pre-trained visual masked autoencoder (MAE) to forecast future time series values without any specific training on time series data. VisionTS achieves comparable or even superior performance to existing text-based and time-series based foundation models in the zero-shot setting, suggesting that visual models could be a free lunch for time series forecasting. This work provides a novel approach for practitioners to build time series forecasting foundation models, particularly in situations where data scarcity or heterogeneity is a challenge. | Read more on HF |
| Mini-Omni: Language Models Can Hear, Talk While Thinking in Streaming | Zhifei Xie, gpt-omni | The paper proposes Mini-Omni, an open-source, end-to-end multi-modal large language model (LLM) with real-time speech interaction capabilities. Mini-Omni enables direct audio reasoning via text-instructed speech generation, which utilizes a novel parallel decoding strategy to boost inference speed. The authors introduce the “Any Model Can Talk” framework, which helps to transfer text capabilities of pre-trained models to speech output with minimal degradation, making it valuable for practitioners in the field. They also introduce the VoiceAssistant-400K dataset, specifically designed for speech-output models. Mini-Omni is a significant advancement in human-computer interaction, offering valuable potential for future research. | Read more on HF |
Papers for 2024-09-02
| Title | Authors | Summary | Link |
|---|---|---|---|
| SciLitLLM: How to Adapt LLMs for Scientific Literature Understanding | xumingjun, caixc97, yrshi, Jesse-zjx, Sihangli | This research paper presents SciLitLLM, a specialized large language model (LLM) designed for scientific literature understanding. The model utilizes a hybrid training strategy that combines continual pre-training (CPT) on high-quality scientific corpora and supervised fine-tuning (SFT) with diverse scientific instructions. To address the challenges of constructing high-quality CPT corpora and generating diverse SFT instructions, the authors propose a meticulous pipeline that includes PDF text extraction, content error correction, and quality filtering for CPT. For SFT, they introduce a novel LLM-based instruction synthesis method to generate diverse instructions. SciLitLLM demonstrates promising performance on scientific literature understanding benchmarks, outperforming existing LLMs across various tasks, especially in domains like fundamental science and organic materials. These findings are particularly relevant to AI engineers and data scientists involved in developing LLMs for specialized domains, highlighting the potential of combining CPT and SFT for knowledge injection and instruction-following enhancements. | Read more on HF |
| CoRe: Context-Regularized Text Embedding Learning for Text-to-Image Personalization | Jian Yin, BlurBlur, Zhangjunyi, darkcser, FeizeWu | The research paper, CoRe: Context-Regularized Text Embedding Learning for Text-to-Image Personalization, tackles the challenge of balancing identity preservation and text alignment in text-to-image personalization. It introduces a novel method, Context Regularization (CoRe), which improves text embedding learning by regularizing the context tokens surrounding the new concept. CoRe enhances the compatibility of the new concept’s text embedding and facilitates a more precise semantic understanding of the prompt. The authors demonstrate that CoRe outperforms several baselines in both identity preservation and text alignment, especially for prompts requiring high visual variability. This research provides valuable insights for practitioners in the field of text-to-image personalization, enabling the generation of high-quality, text-aligned images with improved identity preservation. | Read more on HF |
| The VoxCeleb Speaker Recognition Challenge: A Retrospective | dgromero, jungjee, arsha1, joonson, JaesungHuh | The VoxCeleb Speaker Recognition Challenge (VoxSRC) is a series of annual challenges and workshops that ran from 2019 to 2023. This paper is a retrospective analysis of the VoxSRC challenge, covering the challenges’ goals, dataset creation, evaluation metrics, and the progression of research techniques. Key results highlight that the state-of-the-art has steadily improved over the years, with the use of self-supervised pretrained models significantly advancing performance. The paper also provides valuable insights and recommendations for future challenge organizers, such as maintaining a consistent test set, incorporating individual and ensemble model performance, and including a more diverse dataset. Practitioners, particularly those involved in speaker recognition and diarization, will find this retrospective analysis a valuable resource for understanding the evolution of research techniques and identifying future directions in the field. | Read more on HF |
| CURLoRA: Stable LLM Continual Fine-Tuning and Catastrophic Forgetting Mitigation | mnoorfawi | The paper introduces CURLoRA, a novel approach to fine-tuning LLMs that leverages CUR matrix decomposition to mitigate catastrophic forgetting and improve computational efficiency. By leveraging inverted probabilities in CUR decomposition, the method effectively limits the growth of trainable parameters, resulting in improved stability and performance across tasks while significantly reducing the number of trainable parameters. This method is particularly useful in continual learning scenarios, where LLMs are trained on a sequence of tasks and need to preserve knowledge from previous tasks. The paper shows that CURLoRA outperforms standard LoRA in mitigating catastrophic forgetting, and demonstrates the effectiveness of this approach across a range of tasks and datasets. This research offers practical solutions for AI engineers and data scientists who are seeking to develop and deploy LLMs in real-world settings, where catastrophic forgetting poses a significant challenge. | Read more on HF |
| Jina-ColBERT-v2: A General-Purpose Multilingual Late Interaction Retriever | hanxiao, makram93, jupyterjazz, michael-guenther, bwang0911 | The paper introduces Jina-ColBERT-v2, a novel multilingual dense retriever based on the ColBERT architecture. It presents various improvements to the model architecture and training pipeline, including the adoption of a modified XLM-ROBERTa encoder, pair training with weakly supervised datasets, and triplet training with high-quality multilingual data. Jina-ColBERT-v2 significantly improves performance across a range of English and multilingual retrieval tasks while reducing storage requirements by up to 50%. The authors also highlight the model’s robust performance in low-resource languages, making it suitable for practitioners working on multilingual information retrieval tasks. | Read more on HF |
| SurveySum: A Dataset for Summarizing Multiple Scientific Articles into a Survey Section | Rodrigo Nogueira, Thales Sales Almeida, thiagolaitz, gubartz, carisio | The research paper introduces a novel dataset called “SurveySum” for summarizing multiple scientific articles into a section of a survey. The authors propose two pipelines for summarizing scientific articles into a survey section, which are evaluated using various metrics. The results of the evaluation highlight the importance of high-quality retrieval stages and the impact of different model configurations on the quality of generated summaries. The paper addresses the lack of domain-specific datasets for summarization, which is crucial for building accurate and robust summarization models. This work provides a valuable resource for researchers and practitioners working in the field of natural language processing, particularly those involved in the development and evaluation of summarization models. | Read more on HF |
| Automatic Differential Diagnosis using Transformer-Based Multi-Label Sequence Classification | Lubaba Binte Saber, Mohammad Ashrafuzzaman Khan, AdnanSadi | This research paper explores the use of transformer-based multi-label sequence classification for automated differential diagnosis. The authors propose a method to process tabular patient data into text reports and introduce two data modification modules to improve the robustness of the model. Their experiments using four transformer models demonstrate promising results with over 97% F1 scores and highlight the model’s capability to generalize to challenging scenarios. The results suggest that this approach could be a valuable tool for healthcare professionals seeking to identify and prioritize potential diagnoses for patients, especially when dealing with ambiguous symptoms. This research emphasizes the potential of AI-driven tools to assist with complex medical tasks, particularly for practitioners who may need assistance in identifying a wider range of possible diagnoses. | Read more on HF |
| UrBench: A Comprehensive Benchmark for Evaluating Large Multimodal Models in Multi-View Urban Scenarios | Tianyi Bai, Junyan Ye, Dairong Chen, Haote Yang, Baichuan Zhou | This research paper introduces UrBench, a comprehensive benchmark for evaluating Large Multimodal Models (LMMs) in complex, multi-view urban scenarios. The benchmark includes 11.6K questions covering 14 distinct tasks across four evaluation dimensions, namely Geo-Localization, Scene Reasoning, Scene Understanding, and Object Understanding. UrBench utilizes a novel cross-view detection-matching algorithm to create high-quality annotations and question generation pipeline that incorporates LMM-based, rule-based, and human-based methods. The authors evaluate 21 LMMs on UrBench and find that current models struggle with multi-view understanding, inconsistent behavior across different views, and fall behind human performance in most tasks, highlighting the significant room for improvement in current models’ abilities for human-centric AI applications in urban settings. The paper’s findings are relevant to AI practitioners working on LMM development, as it provides valuable insights into the limitations and potential of current models, and serves as a benchmark for future research. | Read more on HF |
| InkubaLM: A small language model for low-resource African languages | EricPeter, Jenalea, JessicaOjo, bonadossou, Atnafu | The research paper introduces InkubaLM, a 0.4-billion parameter, multilingual language model designed specifically for low-resource African languages. The model demonstrably outperforms larger language models on specific tasks, notably sentiment analysis in Swahili. The authors release the model and datasets to encourage further research and development in the field. By bridging the language gap and offering an accessible tool, the paper highlights the potential for InkubaLM to be used by AI engineers and data scientists in tasks requiring local language understanding, such as machine translation and sentiment analysis. | Read more on HF |
| Large-Scale Multi-omic Biosequence Transformers for Modeling Peptide-Nucleotide Interactions | Eric Oermann, Shivanand P. Lad, Robert J. Steele, Beakal, WeiHua | The authors of this paper, Eric Oermann, Shivanand P. Lad, Robert J. Steele, and Beakal, propose a new method for learning joint representations of protein and nucleotide sequences using a multi-omic transformer architecture. They demonstrate that their model, OmniBioTE, achieves state-of-the-art performance on a variety of tasks related to protein-nucleotide interactions, such as predicting binding affinity and the effects of mutations. They also show that the model can be effectively fine-tuned for single-omics tasks, highlighting its potential for a wider range of applications. This research is relevant to AI engineers, data scientists, and bioinformaticians working in the field of biosequence analysis as it provides a powerful tool for understanding and modeling complex interactions between proteins and nucleic acids. | Read more on HF |
| VLM4Bio: A Benchmark Dataset to Evaluate Pretrained Vision-Language Models for Trait Discovery from Biological Images | abhilashneog, harishB97, ksmehrab, arkadaw9, sammarfy | This paper introduces VLM4Bio, a new benchmark dataset that evaluates the zero-shot performance of vision-language models (VLMs) for the task of trait discovery from biological images. VLM4Bio includes ≈469K question-answer pairs based on 30k images of three taxonomic groups: fishes, birds, and butterflies. The paper finds that while VLMs perform well on some tasks (e.g., trait identification), they struggle with other tasks (e.g., counting traits, localizing traits), highlighting the need for further research in this area. The findings of this paper will be useful for AI engineers and data scientists who are developing VLMs for organismal biology applications. The dataset can be used to train and evaluate VLMs for a variety of tasks, including species classification, trait identification, and trait grounding. It also provides insights into the limitations of current VLMs, which can help to guide future research efforts. | Read more on HF |
| ClimDetect: A Benchmark Dataset for Climate Change Detection and Attribution | vasudevlal, matthewlyleolson, musashihinck, anahita-b, sungduk | The paper introduces ClimDetect, a benchmark dataset for climate change detection and attribution (D&A) that leverages daily snapshots of climate model simulations for training and evaluating machine learning (ML) models. The dataset standardizes input and target variables, promoting consistency and comparability across studies. The authors demonstrate the applicability of Vision Transformers (ViTs) for climate fingerprinting, a novel approach in this domain. ClimDetect is publicly accessible and provides a benchmark for advancing climate science by improving model evaluations. Practitioners, such as AI Engineers and Data Scientists working in climate modeling, can use ClimDetect to enhance their D&A research efforts and develop robust ML models for understanding and mitigating climate change. | Read more on HF |
Papers for 2024-08-30
| Title | Authors | Summary | Link |
|---|---|---|---|
| Law of Vision Representation in MLLMs | chenfengx, WaterInSea, Ye27, Borise, shijiay | The research paper titled “Law of Vision Representation in MLLMs” proposes a novel theory that links the performance of multimodal large language models (MLLMs) to the combination of cross-modal alignment and correspondence in vision representation. The authors establish a linear correlation between a proposed alignment and correspondence score (AC score) and the MLLM’s performance across eight benchmarks. Through this correlation, they propose an “AC policy” to efficiently determine the optimal vision representation, leading to a 99.7% reduction in computational cost compared to traditional methods. The findings are significant for practitioners in AI, particularly data scientists and AI engineers, as they provide an efficient method for selecting the optimal vision representation for MLLMs, thereby streamlining the development process and reducing computational resources. | Read more on HF |
| CogVLM2: Visual Language Models for Image and Video Understanding | ShiyuHuang, LiquidAmmonia, qingsonglv, iyuge2, wenyi | The paper introduces CogVLM2, a new family of visual language models (VLMs) for image and video understanding. The authors introduce an improved training recipe based on the visual expert architecture and a high-resolution cross-module, achieving state-of-the-art results on several benchmarks. CogVLM2 family incorporates temporal grounding, a technique for automatically generating video annotations with timestamps, allowing for more precise and detailed understanding of video content. CogVLM2 family represents a significant advancement in visual and language modalities, offering powerful tools for both research and practical applications such as AI engineers, data scientists and researchers. | Read more on HF |
| WavTokenizer: an Efficient Acoustic Discrete Codec Tokenizer for Audio Language Modeling | jlking, MingHuiFang, Exgc, ziyue, novateur | The research paper “WavTokenizer: an Efficient Acoustic Discrete Codec Tokenizer for Audio Language Modeling” introduces a novel codec model designed to effectively compress audio signals into a low-dimensional discrete representation. Notably, WavTokenizer achieves a significantly compressed representation of one-second audio with only 75 tokens while maintaining superior subjective reconstruction quality compared to existing acoustic codec models. Moreover, WavTokenizer surpasses state-of-the-art performance in semantic tasks on the ARCH benchmark, highlighting its capability to capture richer semantic information. This work opens a new avenue for effectively compressing audio into a discrete representation, thereby enabling the use of audio data with larger language models. Practitioners, including AI engineers and data scientists, may leverage the presented approach to compress audio data for various applications, such as text-to-speech synthesis, audio generation, and cross-modal retrieval. | Read more on HF |
| ReconX: Reconstruct Any Scene from Sparse Views with Video Diffusion Model | duanyueqi, yejunliang23, yikaiw, wenqsun, Liuff23 | This research paper proposes a novel 3D scene reconstruction paradigm called ReconX that utilizes the generative power of video diffusion models to generate more observations from limited sparse views. This allows for higher quality reconstructions, especially in areas not seen in the original input. ReconX utilizes 3D structure guidance and a confidence-aware optimization scheme within the 3D Gaussian Splatting framework to ensure 3D consistency and minimize visual artifacts. Experimental results show that ReconX outperforms existing state-of-the-art methods in terms of both quality and generalizability. This work is particularly relevant for practitioners working in computer vision, especially those who deal with sparse-view 3D reconstruction tasks. The ability to reconstruct high-quality 3D models from a limited number of views could be valuable for applications such as autonomous navigation, virtual reality, and 3D modeling. | Read more on HF |
| SAM2Point: Segment Any 3D as Videos in Zero-shot and Promptable Manners | Chengzhuo Tong, Xiangyang Zhu, Renrui Zhang, Chunyuan24, ZiyuG | This research paper introduces SAM2Point, a novel framework that adapts the Segment Anything Model 2 (SAM 2) for 3D segmentation. The method efficiently converts 3D data into a series of multi-directional videos, enabling SAM 2 to perform zero-shot segmentation without requiring any 2D-3D projection or additional training. SAM2Point supports various prompt types (e.g., 3D point, box, and mask) and demonstrates robust generalization across diverse 3D scenarios (e.g., 3D objects, indoor scenes, outdoor scenes, and raw LiDAR). This approach is particularly relevant for practitioners as it provides an efficient and highly generalizable way to perform 3D segmentation using a pre-trained model, effectively mitigating the data scarcity issue prevalent in 3D domains. | Read more on HF |
| CSGO: Content-Style Composition in Text-to-Image Generation | hobbyaih, NOVAglow646, syp115, wanghaofan, xingpng | The paper presents CSGO, a novel content-style-stylized image generation framework that utilizes a large-scale dataset, IMAGStyle, to achieve high-quality results in both image-driven and text-driven style transfer. CSGO is trained end-to-end, enabling zero-shot arbitrary style transfer through decoupled content and style feature injection. The key contributions of this work include: (1) a dataset construction pipeline that generates and automatically cleanses stylized data triplets; (2) a unified CSGO framework that leverages independent feature injection modules for content and style features; and (3) a Content Alignment Score (CAS) metric to evaluate the content preservation capabilities of the generated image. This paper is relevant to AI engineers and data scientists working on style transfer, as it offers a robust and efficient framework that can be readily implemented for various applications, such as image editing, art creation, and design. | Read more on HF |
| Physics of Language Models: Part 2.2, How to Learn From Mistakes on Grade-School Math Problems | Zeyuan Allen-Zhu, Yuanzhi Li, Zicheng Xu, Tian Ye | The paper investigates whether language models can learn to correct their reasoning mistakes during generation by incorporating “retry data” into the training process. The authors find that training on data that contains erroneous steps immediately followed by their corrections significantly improves the reasoning accuracy of the language model, compared to training on error-free data. They also demonstrate that this approach does not require any modifications to the training process, such as label masking, and that it can be used effectively in conjunction with pre-trained models. These findings suggest that practitioners can directly benefit from incorporating retry data into the training of language models, particularly for tasks that require accurate and robust reasoning. | Read more on HF |
| 3D Reconstruction with Spatial Memory | Lourdes Agapito, HengyiWang | This research paper, titled “3D Reconstruction with Spatial Memory,” presents Spann3R, a novel deep learning-based method for online 3D reconstruction. Spann3R is trained on ordered or unordered image collections without prior knowledge of the scene or camera parameters and directly regresses point maps from images, which is expressed in a common coordinate system. It achieves this by utilizing a spatial memory, which learns to store and access all previously relevant 3D information. By removing the need for optimization-based global alignment, Spann3R facilitates real-time online incremental reconstruction. The authors demonstrate that Spann3R achieves competitive performance compared to prior methods while being significantly faster. For practitioners, this research offers a more efficient and scalable approach for online 3D reconstruction tasks that can be applied in various domains such as autonomous driving, virtual reality, and robotics. | Read more on HF |
| StyleRemix: Interpretable Authorship Obfuscation via Distillation and Perturbation of Style Elements | Mitchell Gordon, yejinchoinka, Ximing, hallisky, jrfish | This paper introduces StyleRemix, an interpretable and adaptable authorship obfuscation method that uses fine-grained style elements to rewrite text while preserving content and maintaining fluency. StyleRemix leverages pre-trained LoRA modules to rewrite text along specific style axes, such as formality or length, resulting in more robust obfuscation than prior methods. The authors introduce two new datasets: AuthorMix, a large-scale corpus of 30K texts from 14 authors and four domains, and DISC, a high-quality parallel corpus spanning seven stylistic axes, demonstrating the effectiveness of the model. StyleRemix outperforms prior methods in both automatic and human evaluation. This work has significant implications for practitioners working in anonymous writing, text anonymization, and privacy-preserving text generation. | Read more on HF |
| Scaling Up Diffusion and Flow-based XGBoost Models | TaewooKim, JesseCresswell | This paper investigates the engineering challenges and algorithmic improvements for applying XGBoost in diffusion and flow-matching models for tabular data generation. The authors identify and resolve several key implementation issues in prior work, including memory management, data duplication, and parallelization, enabling an efficient and scalable implementation of XGBoost-based generative models. Furthermore, they propose multi-output trees and early stopping as algorithmic improvements. The results show that the proposed method scales to much larger datasets than previously possible and leads to improvements in both model performance and resource efficiency. This work provides valuable insights for practitioners in the field of tabular generative modeling, offering practical guidance for engineering efficient and scalable models based on XGBoost. | Read more on HF |
| Meta Flow Matching: Integrating Vector Fields on the Wasserstein Manifold | Leo J. Lee, Mathieu Blanchette, Brandon Amos, Xi Zhang, Lazar Atanackovic | The paper proposes a new method, Meta Flow Matching (MFM), for learning the dynamics of interacting particles. Unlike current flow-based models, which are limited to a single initial population and predefined conditions, MFM can generalize to previously unseen populations by integrating along vector fields on the Wasserstein manifold. The authors demonstrate the ability of MFM to improve prediction of individual treatment responses on a large scale multi-patient single-cell drug screen dataset. This work may be relevant to practitioners in a variety of fields, such as AI engineers, data scientists, and bioinformaticians, who are interested in modeling complex systems with interacting particles. MFM can be used to develop more accurate and personalized treatment regimens for patients with various diseases. | Read more on HF |