arXiv:2505.11491v3 Announce Type: replace
Abstract: This study investigates why physics-informed machine learning (PIML) can fail in macroscopic traffic flow modeling. We define failure as cases where a PIML model underperforms both purely data-driven and purely physics-based baselines by a given threshold. Unlike in other fields, physics residuals themselves do not hinder optimization in this setting. Instead, effective updates require both data and physics gradients to form acute angles with the true gradient, a condition difficult to satisfy with low-resolution loop data. In such cases, neural networks cannot accurately approximate density and speed, and the constructed physics residuals, already degraded by discrete sampling and temporal averaging, lose their ability to capture PDE dynamics, which directly leads to PIML failure. Theoretically, although LWR and ARZ solutions are weak solutions, for piecewise $C^k$ initial data they remain $C^k$ off the shock set under mild conditions, which has Lebesgue measure zero. Thus, almost all detector or collocation points lie in smooth regions where residuals are valid, and the MLP's inability to exactly represent discontinuities is immaterial. Finally, we establish MSE lower bounds of physics residuals: higher-order models such as ARZ have strictly larger consistency error bounds than LWR under mild conditions. This explains why LWR-based PIML can outperform ARZ-based PIML even with high-resolution data, with the gap shrinking as resolution increases, consistent with prior empirical findings.
Science Journals
arXiv:2505.11765v5 Announce Type: replace
Abstract: Agents powered by advanced large language models (LLMs) have demonstrated impressive capabilities across diverse complex applications. Recently, Multi-Agent Systems (MAS), wherein multiple agents collaborate and communicate with each other, have exhibited enhanced capabilities in complex tasks, such as high-quality code generation and arithmetic reasoning. However, the development of such systems often relies on handcrafted methods, and the literature on systematic design and optimization of LLM-based MAS remains limited. In this work, we introduce \textbf{OMAC}, a general framework designed for holistic optimization of LLM-based MAS. Specifically, we identify five key optimization dimensions for MAS, encompassing both agent functionality and collaboration structure. Building upon these dimensions, we first propose a general algorithm, utilizing two actors termed the Semantic Initializer and the Contrastive Comparator, to optimize any single dimension. Then, we present an algorithm for joint optimization across multiple dimensions. Extensive experiments demonstrate the superior performance of OMAC on diverse tasks against recent approaches.
arXiv:2505.17964v2 Announce Type: replace
Abstract: Cycle count statistics are fundamental tools in statistics and engineering, with applications in motif counting, channel coding, and statistical inference of network and matrix data. However, how to compute high-order cycle count statistics efficiently is still an open problem. In this paper, we aim to derive Computationally Efficient Equivalent Forms (CEEF) for cycle count statistics of any given order, where we express each cycle count statistic equivalently as a linear combination of finitely many terms. Using the CEEF, we provide a much more efficient way to compute the cycle count statistics.
The CEEF problem has no known general solution and requires delicate combinatorial arguments together with extensive calculations. While this task is hard to accomplish by humans alone, it provides an ideal setting in which Artificial Intelligence (AI) can be useful. We solve the problem by combining several theorems we derive with powerful coding skills of modern AI systems. Our results leverage graph-theoretic arguments and yield new formulas for general cases that were previously unknown. We find that, although AI cannot solve the problem independently, it becomes highly effective when guided by humans through theorems we derive as well as a clear derivation strategy, step-by-step instructions, and carefully-written prompts.
We consider several statistical applications, including spiked matrix testing, estimation of weak spike eigenvalues, and pairwise network comparison. For each problem, we demonstrate that optimal statistical performance is achieved by using high-order cycle count statistics, and our CEEF formulas make their computation feasible on large-scale data sets.
arXiv:2505.20161v2 Announce Type: replace
Abstract: Effective generalization in language models depends critically on the diversity of their training data. Yet existing diversity metrics often fall short of this goal, relying on surface-level heuristics that are decoupled from model behavior. This motivates us to ask: What kind of diversity in training data actually drives generalization in language models -- and how can we measure and amplify it? Through large-scale empirical analyses spanning over 300 training runs, carefully controlled for data scale and quality, we show that data diversity can be a strong predictor of generalization in LLM reasoning -- as measured by average model performance on unseen out-of-distribution benchmarks. We introduce G-Vendi, a metric that quantifies diversity via the entropy of model-induced gradients. Despite using a small off-the-shelf proxy model for gradients, G-Vendi consistently outperforms alternative measures, achieving strong correlation (Spearman's $\rho \approx 0.9$) with out-of-distribution (OOD) performance on both natural language inference (NLI) and math reasoning tasks. Building on this insight, we present Prismatic Synthesis, a framework for generating diverse synthetic data by targeting underrepresented regions in gradient space. Experimental results show that Prismatic Synthesis consistently improves model performance as we scale synthetic data -- not just on in-distribution test but across unseen, out-of-distribution benchmarks -- significantly outperforming state-of-the-art models that rely on 20 times larger data generator than ours. For example, PrismMath-7B, our model distilled from a 32B LLM, outperforms R1-Distill-Qwen-7B -- the same base model trained on proprietary data generated by 671B R1 -- on 6 out of 7 challenging benchmarks.
arXiv:2505.24852v3 Announce Type: replace
Abstract: On-device learning at the edge enables low-latency, private personalization with improved long-term robustness and reduced maintenance costs. Yet, achieving scalable, low-power end-to-end on-chip learning, especially from real-world sequential data with a limited number of examples, is an open challenge. Indeed, accelerators supporting error backpropagation optimize for learning performance at the expense of inference efficiency, while simplified learning algorithms often fail to reach acceptable accuracy targets. In this work, we present Chameleon, leveraging three key contributions to solve these challenges. (i) A unified learning and inference architecture supports few-shot learning (FSL), continual learning (CL) and inference at only 0.5% area overhead to the inference logic. (ii) Long temporal dependencies are efficiently captured with temporal convolutional networks (TCNs), enabling the first demonstration of end-to-end on-chip FSL and CL on sequential data and inference on 16-kHz raw audio. (iii) A dual-mode, multiplier-free compute array allows either matching the power consumption of state-of-the-art inference-only keyword spotting (KWS) accelerators or enabling $4.3\times$ higher peak GOPS. Fabricated in 40-nm CMOS, Chameleon sets new accuracy records on Omniglot for end-to-end on-chip FSL (96.8%, 5-way 1-shot, 98.8%, 5-way 5-shot) and CL (82.2% final accuracy for learning 250 classes with 10 shots), while maintaining an inference accuracy of 93.3% on the 12-class Google Speech Commands dataset at an extreme-edge power budget of 3.1 $\mu$W.
arXiv:2506.00625v2 Announce Type: replace
Abstract: The imbalanced distribution of long-tailed data presents a significant challenge for deep learning models, causing them to prioritize head classes while neglecting tail classes. Two key factors contributing to low recognition accuracy are the deformed representation space and a biased classifier, stemming from insufficient semantic information in tail classes. To address these issues, we propose permutation-invariant and head-to-tail feature fusion (PI-H2T), a highly adaptable method. PI-H2T enhances the representation space through permutation-invariant representation fusion (PIF), yielding more clustered features and automatic class margins. Additionally, it adjusts the biased classifier by transferring semantic information from head to tail classes via head-to-tail fusion (H2TF), improving tail class diversity. Theoretical analysis and experiments show that PI-H2T optimizes both the representation space and decision boundaries. Its plug-and-play design ensures seamless integration into existing methods, providing a straightforward path to further performance improvements. Extensive experiments on long-tailed benchmarks confirm the effectiveness of PI-H2T.
arXiv:2503.06942v2 Announce Type: replace
Abstract: A typical real-time ad-serving funnel comprises ad targeting, conversion modeling (e.g., click-through rate prediction), budget pacing (bidding), and auction processes. While there is a wealth of research and articles on ad targeting and conversion modeling, budget pacing,a crucial component,lacks a systematic treatment specifically tailored for engineers in existing literature. This book aims to provide engineers with a practical yet relatively comprehensive introduction to budget pacing algorithms within the digital advertising domain.
arXiv:2506.01726v2 Announce Type: replace
Abstract: Many problems in Euclidean geometry, arising in computational design and fabrication, amount to a system of constraints, which is challenging to solve. We suggest a new general approach to the solution, which is to start with analogous problems in isotropic geometry. Isotropic geometry can be viewed as a structure-preserving simplification of Euclidean geometry. The solutions found in the isotropic case give insight and can initialize optimization algorithms to solve the original Euclidean problems. We illustrate this general approach with three examples: quad-mesh mechanisms, composite asymptotic-geodesic gridshells, and asymptotic gridshells with constant node angle.
arXiv:2506.03066v2 Announce Type: replace
Abstract: The link function, which characterizes the relationship between the preference for two trajectories and their returns, is a crucial component in designing RL algorithms that learn from preference feedback. Most existing methods, both theoretical and empirical, assume that the link function is known (often a logistic function based on the Bradley-Terry model), which is arguably restrictive given the complex nature of preferences, especially those of humans. To avoid mis-specification, this paper studies preference-based RL with an unknown link function and proposes a novel zeroth-order policy optimization algorithm called Sign-SZPO. Unlike typical zeroth-order methods, which rely on the known link function to estimate the value function differences and form a gradient estimator, Sign-SZPO only estimates the sign of the value function difference. It then constructs a parameter update direction that is positively correlated with the true policy gradient, eliminating the need to know the link function exactly. Under mild conditions, Sign-SZPO provably converges to a stationary policy with a polynomial rate in the number of policy iterations and trajectories per iteration. Empirical evaluations further demonstrate the robustness of Sign-SZPO under link function mis-specifications.
arXiv:2506.04122v3 Announce Type: replace
Abstract: Open-loop performance evaluation of 3D multi-object tracking in autonomous driving requires matching criteria that effectively penalize translational, shape, and orientation errors from the ego vehicle perspective. The prevailing criteria for determining true positives are Intersection over Union (IoU) and Centre-Point Distances (CPD). When IoU is extended from the 2D image plane to 3D volumetric overlap, it often falls below its acceptance threshold even with minor yaw misalignments, whereas CPD disregards orientation entirely. To address this limitation, we propose Contour Errors (CE) as an ego-centric criterion that employs Hausdorff-type reasoning to sparse bounding-box corner geometry by selecting the k-nearest ego-centric corners. This method provides a graded orientation sensitivity between the extremes of IoU, which overpenalizes, and CPD, which is orientation-blind. We evaluate Contour Errors against six baselines using the HOTA evaluation protocol on the nuScenes dataset, conditioned on proximity, yaw error, and a confidence threshold. At the standard IoU vehicle threshold, 47% of car and 75% of pedestrian CE-valid matches are rejected by IoU despite close contour proximity, while fewer than 0.1% of IoU-valid matches fail CE, resulting in a 4.2% higher match rate relative to total ground truth annotations. These results establish the ego-centric matching criterion as a primary factor for improving open-loop perception evaluation in safety-critical autonomous driving.
arXiv:2506.04141v2 Announce Type: replace
Abstract: The sequential structure of videos poses a challenge to the ability of multimodal large language models (MLLMs) to locate multi-frame evidence and conduct multimodal reasoning. However, existing video benchmarks mainly focus on understanding tasks, which only require models to match frames mentioned in the question (hereafter referred to as "question frame") and perceive a few adjacent frames. To address this gap, we propose MMR-V: A Benchmark for Multimodal Deep Reasoning in Videos. The benchmark is characterized by the following features. (1) Long-range, multi-frame reasoning: Models are required to infer and analyze evidence frames that may be far from the question frame. (2) Beyond perception: Questions cannot be answered through direct perception alone but require reasoning over hidden information. (3) Reliability: All tasks are manually annotated, referencing extensive real-world user understanding to align with common perceptions. (4) Confusability: Carefully designed distractor annotation strategies to reduce model shortcuts. MMR-V consists of 317 videos and 1,257 tasks. Our experiments reveal that current models still struggle with multi-modal reasoning; even the best-performing model, Gemini-2.5-pro, achieves only 64.3% accuracy. Additionally, current reasoning enhancement strategies (Chain-of-Thought and scaling test-time compute) bring limited gains. Error analysis indicates that the CoT demanded for multi-modal reasoning differs from it in textual reasoning, which partly explains the limited performance gains. We hope that MMR-V can inspire further research into enhancing multi-modal reasoning capabilities.
arXiv:2506.06570v4 Announce Type: replace
Abstract: As robotic systems become increasingly integrated into real-world environments, ranging from autonomous vehicles to household assistants, they inevitably encounter diverse and unstructured scenarios that lead to failures. While such failures pose safety and reliability challenges, they also provide rich perceptual data for improving system robustness. However, manually analyzing large-scale failure datasets is impractical and does not scale. In this work, we introduce the problem of unsupervised discovery of failure taxonomies from large volumes of raw failure logs, aiming to obtain semantically coherent and actionable failure modes directly from perceptual trajectories. Our approach first infers structured failure explanations from multimodal inputs using vision language reasoning, then clusters them in the resulting semantic reasoning space, discovering recurring failure modes rather than isolated episode-level descriptions. We evaluate our method across robotic manipulation, indoor navigation, and autonomous driving domains, demonstrating that the discovered taxonomies are consistent, interpretable, and useful in practice. In particular, we show that structured failure taxonomies guide targeted data collection for offline policy refinement and enhance runtime failure monitoring systems. Website: https://mllm-failure-clustering.github.io/
arXiv:2506.07691v2 Announce Type: replace
Abstract: Sparse Autoencoders (SAEs) are a cornerstone of mechanistic interpretability. Existing training methods inherit the Block Training paradigm from LLM pre-training, which introduces destructive gradient noise in instruct models due to attention leakage from unrelated contexts. Using GSNR analysis, we theoretically characterize this issue and propose Finetuning-aligned Sequential Training (FAST), a sequential training paradigm specifically designed for instruct models. FAST aligns SAE training with the data distribution and activation patterns of instruct models, substantially improving both reconstruction fidelity and feature interpretability. Experimental results show that FAST achieves higher GSNR, a significantly lower log-scaled MSE of 0.6468 compared to the baseline's 5.1985, and a near-zero Delta Loss (-0.51\% to 0.37\%). Moreover, on Llama-3.2-3B-it, FAST produces 21.1\% high-quality features, substantially outperforming baseline methods that achieve 7.0\% and 10.2\%. We further find that intervening on special token activations through SAEs can improve generation quality, revealing new opportunities for fine-grained control. Our codes are available as open source at https://github.com/Geaming2002/FAST.
arXiv:2506.08725v3 Announce Type: replace
Abstract: Misuses of t-SNE and UMAP in visual analytics have become increasingly common. For example, although t-SNE and UMAP projections often do not faithfully reflect the original distances between clusters, practitioners frequently use them to investigate inter-cluster relationships. We investigate why this misuse occurs, and discuss methods to prevent it. To that end, we first review 136 papers to verify the prevalence of the misuse. We then interview researchers who have used dimensionality reduction (DR) to understand why such misuse occurs. Finally, we interview DR experts to examine why previous efforts failed to address the misuse. We find that the misuse of t-SNE and UMAP stems primarily from limited DR literacy among practitioners, and that existing attempts to address this issue -- mostly based on academic papers -- have been ineffective. Based on these insights, we discuss potential future research directions to mitigate the misuse.
arXiv:2506.11585v2 Announce Type: replace
Abstract: We introduce OV-MAP, a novel approach to open-world 3D mapping for mobile robots by integrating open-features into 3D maps to enhance object recognition capabilities. A significant challenge arises when overlapping features from adjacent voxels reduce instance-level precision, as features spill over voxel boundaries, blending neighboring regions together. Our method overcomes this by employing a class-agnostic segmentation model to project 2D masks into 3D space, combined with a supplemented depth image created by merging raw and synthetic depth from point clouds. This approach, along with a 3D mask voting mechanism, enables accurate zero-shot 3D instance segmentation without relying on 3D supervised segmentation models. We assess the effectiveness of our method through comprehensive experiments on public datasets such as ScanNet200 and Replica, demonstrating superior zero-shot performance, robustness, and adaptability across diverse environments. Additionally, we conducted real-world experiments to demonstrate our method's adaptability and robustness when applied to diverse real-world environments.
arXiv:2102.12081v2 Announce Type: replace
Abstract: As novel applications spring up in future network scenarios, the requirements on network service capabilities for differentiated services or burst services are diverse. Aiming at the research of collaborative computing and resource allocation in edge scenarios, migrating computing tasks to the edge and cloud for computing requires a comprehensive consideration of energy consumption, bandwidth, and delay. Our paper proposes a collaboration mechanism based on computation offloading, which is flexible and customizable to meet the diversified requirements of differentiated networks. This mechanism handles the terminal's differentiated computing tasks by establishing a collaborative computation offloading model between the cloud server and edge server. Experiments show that our method has more significant improvements over regular optimization algorithms, including reducing the execution time of computing tasks, improving the utilization of server resources, and decreasing the terminal's energy consumption.
arXiv:2506.13507v2 Announce Type: replace
Abstract: This letter revisits update ordering in layered belief propagation (LBP) decoding of low-density parity-check (LDPC) codes. The closest probability-based schedule orders layers by check node unsatisfied probabilities estimated only from the channel observations, although these probabilities change once decoding messages are exchanged. We therefore refresh the check node unsatisfied probabilities during decoding and use them as dynamic priorities. The first schedule, Dyn-EBP, selects the most reliable available check node while ensuring that every check node is updated once in each iteration. The second schedule, Dyn-PEBP, adds a linear update-count penalty and allows limited repeated updates without letting a small subset of check nodes dominate the schedule. For 5G new radio LDPC base graph 1 codes with five iterations, the proposed schedules yield small BLER reductions relative to the channel-only probability schedule and remain competitive with LBP, LPHD scheduling, and RD-RBP. The gain is modest, but it shows that probability-based scheduling benefits from message-level refinement.
arXiv:2506.14121v3 Announce Type: replace
Abstract: Face super-resolution (FSR) under limited computational budgets remains challenging. Existing methods often treat all facial pixels equally, leading to suboptimal resource allocation and degraded performance. CNNs are sensitive to high-frequency facial features such as contours and outlines, while Mamba excels at capturing low-frequency attributes like facial color and texture with lower complexity than Transformers. Motivated by this, we propose FADPNet, a Frequency-Aware Dual-Path Network that decomposes facial features into low- and high-frequency components for dedicated processing. The low-frequency branch employs a Mamba-based Low-Frequency Enhancement Block (LFEB) that integrates state-space attention with squeeze-and-excitation to restore global interactions and emphasize informative channels. The high-frequency branch uses a CNN-based Depthwise Position-aware Attention (DPA) module to refine structural details, followed by a lightweight High-Frequency Refinement (HFR) module for further frequency-specific refinement. These designs enable FADPNet to achieve a strong balance between FSR quality and efficiency, outperforming existing methods. Codes will be available at https://github.com/IVIPLab/FADPNet.
arXiv:2506.18109v3 Announce Type: replace
Abstract: Polygon resonators are promising candidates for nanomechanical applications due to their compact architecture and high force sensitivity. Here, we develop a polygon-specific analytical model for perimeter modes by coupling the flexural motion of the polygon sides to tether torsion governed by a stress-modified Timoshenko-Gere (TG) equation. The model captures finite-wavelength torsional phase accumulation beyond the conventional short-tether approximation, revealing tether-length-dependent resonances and bending-torsion hybridization. Comparisons with finite-element-method (FEM) simulations show good agreement in the quasi-one-dimensional regime, while identifying the onset of deviations associated with neglected torsional deformation of the polygon sides. The model therefore provides a useful analytical tool for interpreting and optimizing high quality factor (Q) polygon resonators for cavity optomechanics and precision sensing.
arXiv:2506.18150v4 Announce Type: replace
Abstract: Fully Homomorphic Encryption (FHE) enables computation directly on encrypted data and privacy-preserving neural inference in the cloud. Existing solutions focus on models with dense inputs (e.g., CNNs and MLPs). Recommendation models (e.g., DLRM) pose a different challenge: sparse categorical inputs require private lookups into large embedding tables, which must be implemented using FHE's restrictive operators. Naive lookups incur significant communication and memory costs; prior work proposes compressing embedding tables at the expense of introducing large server-side compute costs (i.e., indicator function) and revealing embedding-table structure. We present HE-LRM, a performance optimized solution for executing recommendation with FHE. First, we develop an embedding compression technique using client-side digit decomposition that achieves 56$\times$ speedup over the state-of-the-art. Next, we propose a multi-embedding packing strategy that enables ciphertext SIMD-parallel lookups across multiple tables. We integrate HE-LRM into the open-source Orion FHE framework to demonstrate end-to-end encrypted DLRM inference. We evaluate HE-LRM on UCI (health prediction) and Criteo (click prediction), achieving inference latencies of 24 seconds on UCI and 228 to 489 seconds, respectively, on a single-threaded CPU. Finally, we show how GPU and ASIC FHE acceleration can reduce end-to-end latencies to seconds and even sub-seconds. Our code can be found at https://github.com/baahl-nyu/orion/tree/criteo-helrm.
arXiv:2506.20463v2 Announce Type: replace
Abstract: Generative artificial intelligence (GenAI) is transforming the educational landscape by augmenting learning paradigms. However, state-of-the-art GenAI systems driving this transformation are predominantly developed and controlled by a small number of private companies; there is little clarity about their data retention practices and limited user control over inputs and outputs. In the context of education, end-users lack the awareness of how to safely adopt GenAI in learning. This raises significant concerns, particularly when proprietary or personally identifiable educational information may be shared with external GenAI platforms. In response to these concerns, universities are developing their own usage guidelines and policies to balance innovation with academic integrity, privacy, and security. Our research seeks to understand these emerging guidelines, with a particular focus on the privacy and security implications of integrating GenAI tools into academic environments - an area that has received little attention to date. We conducted an in-depth qualitative analysis of GenAI-usage guidelines from 43 universities across 12 countries. Our findings reveal several key challenges, including barriers faced by universities in deploying privacy measures and adopting existing security frameworks. These insights lay the groundwork for designing more robust, privacy-aware GenAI guidelines for higher education.
arXiv:2507.00930v3 Announce Type: replace
Abstract: In inverse optimization, the goal is to find a minimum perturbation of weights that makes a prescribed feasible solution optimal. For matroids, the classical inverse problem fixes a target basis. We replace this fixed target by a subset constraint: given a matroid $M=(S,\mathcal{I})$, weights $w$, and a subset $S_0\subseteq S$, we specify how the family of maximum-weight bases relates to the bases contained in $S_0$. We study six natural variants. The positive variants require, respectively, that at least one basis contained in $S_0$ be optimal, that every basis contained in $S_0$ be optimal, or that the optimal bases be exactly the bases contained in $S_0$; we also study the three corresponding negated requirements. This framework captures partial inverse requirements such as forced or forbidden elements, as well as settings where undesirable optimal bases should be excluded.
We give a complete classification of these subset-constrained inverse matroid problems under the $\ell_\infty$- and $\ell_1$-norms. Under the $\ell_\infty$-norm, all six variants admit polynomial-time combinatorial algorithms (interpreting the variants with strict inequalities in the natural integral-weight setting). The algorithms are based on matroid exchange, uniform perturbations, and the connected-component structure of the restriction $M|S_0$. Under the $\ell_1$-norm, the picture changes sharply: the variant requiring at least one optimal basis contained in $S_0$ is strongly $\mathsf{NP}$-hard even for graphic matroids, whereas the remaining variants considered here admit polynomial-time algorithms. Thus the subset-constrained setting separates the two norms already for matroids, and even for spanning trees.
arXiv:2507.04384v4 Announce Type: replace
Abstract: Achieving safe, efficient, and kinematically feasible planning in dynamic environments remains a significant challenge, as planners must simultaneously handle moving obstacles, sensor uncertainty, and strict motion constraints. To address this problem, we propose an energy-parameterized diffusion planning framework that learns a conservative energy field to realize safe and stable generalization across diverse scenarios. The energy-parameterized diffusion formulation enables flexible integration of multiple constraints, allowing the planner to generalize to previously unseen environments without retraining. To ensure real-time safety during deployment, we further incorporate a lightweight safety filter that enforces safety and kinematic feasibility constraints in real-time. Additionally, we develop a scene-agnostic, MPC-based data generation pipeline to produce large-scale, dynamically feasible training trajectories. In simulation, the proposed method achieves real-time performance with a mean planning time of 0.21s and a low planning failure rate of 0.57%. Real-world experiments on the F1TENTH platform further validate the effectiveness of the proposed framework. Under sensor uncertainty in previously unseen dynamic environments, the planner consistently generates collision-free trajectories, which remain safe after being tracked by a simple controller, maintaining a mean obstacle clearance of 0.26 m, demonstrating strong robustness and practical applicability. Project page: https://rstp-comp-diffuser.github.io.
arXiv:2507.05515v3 Announce Type: replace
Abstract: Vision-language models (VLMs) are facing the challenges of understanding and following multimodal assembly instructions, particularly when fine-grained spatial reasoning and precise object state detection are required. In this work, we explore LEGO Co-builder, a hybrid benchmark combining real-world LEGO assembly logic with programmatically generated multimodal scenes. The dataset captures stepwise visual states and procedural instructions, allowing controlled evaluation of instruction-following, object detection, and state detection. We introduce a unified framework and assess leading VLMs such as GPT-4o, Gemini, and Qwen-VL, under zero-shot and fine-tuned settings. We also evaluated the framework using a reasoning-focused model, GLM-4.1-thinking. Our results show that while object detection achieved high performance (98.16% with fine-tuned InstructBLIP), fine-grained scene understanding and assembly state detection remain challenging: Fine-tuned MiniGPT-v2 reached only 37.52% F1 for identifying theme entities, and even advanced models such as GPT-4o achieved just 40.54% F1 on state detection. This highlights gaps in fine-grained visual understanding among existing models. We release the benchmark, codebase, and generation pipeline to support future research on multimodal assembly assistants grounded in real-world workflows.
arXiv:2507.05876v3 Announce Type: replace
Abstract: Large-scale training for distributed Machine Learning can cause congestion at bottleneck switch ports, leading to model staleness through update losses. This is particularly detrimental for asynchronous Distributed Reinforcement Learning (DRL) training, as stale updates are known to degrade convergence performance in asynchronous settings. This paper presents \textit{Shesha}, an in-network DRL accelerator engine, which \textit{opportunistically aggregates} asynchronously generated model updates \textit{on the fly} while they traverse the data plane queue. This aggregation operation motivates an alternative queue design, which we prototype and envision for future Top-of-Rack switches. We further present corresponding host-side transmission control in the face of possible congestion, taking advantage of in-network accelerator feedback. A quantification of model staleness, denoted Age-of-Model (AoM), together with a formal verifier allows us to reason on system-wide AoM objectives in multi DRL-cluster scenarios. Shesha shows significant reductions in model staleness and queue congestion, improving overall convergence behavior for asynchronous DRL workloads.