Forskningsradar

Science Journals

Peer-reviewade publikationer — 54780 artiklar

Softly Constrained Denoisers for Diffusion Models Applied to Partial Differential Equations
arXiv:2512.14980v4 Announce Type: replace Abstract: Diffusion models have become a powerful generative prior for solutions of partial differential equations (PDEs). Existing approaches enforce physical constraints either by adding the PDE residuals as loss regularizers or through inference-time adjustments. These methods bias the model away from the true data distribution, which is especially problematic when the governing PDE is misspecified. To circumvent these issues while making the most out of the PDE constraint, we introduce soft inductive biases into the denoiser architecture derived from the PDEs. We show that these softly constrained denoisers exploit constraint knowledge to improve compliance over standard denoisers, while maintaining enough flexibility to deviate from it in case of misspecification with respect to observed data.
TraceGraph: Shared Decision Landscapes for Diagnosing and Improving Agent Trajectories
arXiv:2605.31308v1 Announce Type: new Abstract: Agent benchmarks increasingly record rich interaction trajectories, yet evaluation often reduces each rollout to a pass rate or reward score. We introduce TraceGraph, a graph-based framework that turns released multi-model agent trajectories into shared decision landscapes. For each task, TraceGraph builds a graph over observable action-observation states from pooled rollouts before model identity is introduced. It then overlays outcome-informed productive cores and trap regions, and summarizes each rollout with three events: Access, Trap exposure, and Repair. Across trajectories spanning five benchmark splits, TraceGraph profiles reveal navigation differences hidden by aggregate scores and show that splits differ in whether they reward avoiding traps or recovering from them. The same TraceGraph landscape also motivates a trap-aware recovery pipeline for SWE-bench: aruntime detector fires on states matching historical trap regions, then lightweight continuation policies are evaluated from the same prefix. On fired states, the best pooled single-factor policy raises official resolved rate from 40.4% to 43.5% on the per-provider fired subset and from 41.0% to 44.8% on common-fired instances, with provider-specific active components. Overall, TraceGraph provides a process vocabulary for asking what agent benchmarks test, where models diverge on a shared landscape, and how failure regions can guide downstream improvement.
AR Forcing: Towards Long-Horizon Robot Navigation World Model
arXiv:2605.31314v1 Announce Type: new Abstract: The diffusion based robot navigation world models are typically trained using parallel supervision, while autoregressive inference is employed during path planning. This results in a distribution shift between training and inference, which destabilizes the performance over long-horizon prediction. We propose AR Forcing, an autoregressive training strategy, which integrates the standard diffusion loss into the autoregressive training loop. At each step, the model uses its own predictions to update the context and optimize the single step noise prediction objective, thereby explicitly exposing the model to the inference state distribution during training. Our method does not require additional discriminators or distribution-matching losses, retains the original diffusion framework and sampler, and is easy to integrate. Experiments on multi-domain navigation datasets (RECON, SCAND, HuRoN, TartanDrive) show that compared with strong baselines, AR Forcing improved the consistency of generated images during long-horizon navigation and the accuracy of predicted trajectories, enhancing robustness of the model in complex known and unknown environments. We will release the code soon.
Generalized Intention Modeling in Multi-Agent Reinforcement Learning
arXiv:2605.31318v1 Announce Type: new Abstract: Modeling an opponent's intent is critical for effective decision-making in non-cooperative, competitive, and general-sum multi-agent reinforcement learning. Existing opponent modeling methods encode intent using an embedding derived from episode information chosen a priori, such as the opponent's next action or a future environment state, and use this to guide the ego-agent's behavior. These approaches assume that the chosen information is universally representative of intent; however, we show empirically that this is not the case as intentions are often task- and environment-dependent. To address this, we introduce a task-adaptive opponent modeling framework that learns a performance-driven mixture of multiple intent representations. We further introduce a new intention representation that maximizes mutual information with the ego-agent's future returns, thereby capturing opponent information that is most directly relevant to performance. Our approach consistently matches or exceeds the performance of state-of-the-art baselines across diverse tasks and yields insights into when and why different opponent modeling strategies succeed.
Constrained Multi-Objective Reinforcement Learning with Max-Min Criterion
arXiv:2605.31388v1 Announce Type: new Abstract: Multi-Objective Reinforcement Learning (MORL) extends standard RL by optimizing policies with respect to multiple, often conflicting, objectives. While max-min MORL has emerged as an effective approach for promoting fairness, its applicability remains limited, particularly when constraints must be incorporated. In this paper, we propose a MORL framework that integrates the max-min criterion with explicit constraint satisfaction. We establish a theoretical foundation for the proposed framework and validate the resulting algorithm through convergence analysis and experiments in tabular settings. We further demonstrate the practical relevance of our approach in simulated building thermal control, multi-objective locomotion control, and greenhouse-gas-emission-aware traffic management. Across these domains, our method effectively balances fairness and constraint satisfaction in multi-objective decision-making.
Spectral density estimation for normal matrices
arXiv:2605.31430v1 Announce Type: new Abstract: The spectral density estimation problem asks for an algorithm that, given an $n\times n$ matrix $A$, outputs a probability measure that is a good approximation to the uniform distribution on the eigenvalues of $A$, called the spectral density of $A$. This paper considers the setting where $A$ is a large normal matrix that is accessible only through matrix-vector product queries. We provide an algorithm that makes just $m$ matrix-vector queries to $A$ and returns, with high probability, a measure within earth mover's distance $O(1/m+\log m/{\sqrt n})$ of the true spectral density of $A$. We provide a complementary lower bound that any algorithm producing an $\varepsilon$-approximation to the true spectral density for large matrices must make $\Omega(1/\varepsilon)$ matrix-vector queries. The lower bound holds even for the more restricted case of real symmetric input matrices. In combination with our upper bound, it shows that spectral density estimation is essentially no harder for complex normal matrices than for real symmetric matrices.
AdaptR1: Reinforcement Learning Based Adaptive Interleaved Thinking in Multi-hop Question Answering
arXiv:2605.31062v1 Announce Type: new Abstract: Large Language Models (LLMs) have achieved remarkable performance in complex reasoning tasks through Chain-of-Thought (CoT) prompting. However, this approach often leads to ``over-thinking,'' where models generate unnecessarily long reasoning traces for simple queries and incur avoidable inference cost. While recent work has explored adaptive reasoning, existing methods typically make a single query-level decision about whether to reason. This overlooks the dynamic nature of multi-step tasks, where the need for explicit reasoning varies across intermediate stages. To address this limitation, we introduce AdaptR1, a Reinforcement Learning (RL) based framework for adaptive interleaved thinking in multi-hop Question Answering (QA). Unlike previous approaches that require Supervised Fine-Tuning (SFT) for cold-start initialization, AdaptR1 uses a fully RL-based strategy with a quality-gated efficiency reward to dynamically allocate reasoning budgets at each step. Under the Graph-R1 setting, AdaptR1 reduces average think tokens by 69.71\%, with a 90.35\% reduction on HotpotQA, while maintaining performance comparable to or better than standard baselines. Furthermore, our analysis reveals that overthinking in multi-hop reasoning is not uniformly distributed but occurs predominantly during the initial planning stages, highlighting the effectiveness of step-wise adaptive budget allocation.
Value Functions as Supermartingale Certificates
arXiv:2605.31524v1 Announce Type: new Abstract: Certification methods for stochastic systems provide sufficient proof rules, based on real-valued supermartingale certificates, to determine the almost-sure satisfaction of $\omega$-regular properties (and therefore of linear temporal logic) over general state spaces, encompassing both countably infinite and continuous state spaces. Conversely, reinforcement learning (RL) methods for $\omega$-regular tasks have received considerable attention, but they typically lack formal guarantees that the learned policy satisfies the specification, except possibly for finite state and action spaces. We bridge these two lines of research by establishing a novel theoretical connection: under an appropriate reward, the value function associated to a policy that almost surely satisfies an $\omega$-regular property encodes a Streett supermartingale certificate for that specification. Our results, validated experimentally on finite Markov decision processes, hold for finite, countably infinite, and continuous state spaces, suggesting a principled route to certificate synthesis via RL.
amerta: A Python Library for Idealized 1D Saint--Venant Dam-Break Simulation
arXiv:2605.31011v1 Announce Type: new Abstract: The Saint-Venant shallow water equations (SWE) govern depth-integrated free-surface flows arising in dam-break inundation, flood routing, tsunami runup, and estuarine tidal dynamics. Closed-form analytical solutions exist only for highly idealized Riemann configurations, making rigorously verified numerical solvers essential. This work presents amerta, an open-source Python library that solves the one-dimensional frictionless Saint-Venant system on a uniform Cartesian grid using Monotone Upstream-centered Schemes for Conservation Laws (MUSCL) reconstruction with a minmod slope limiter, the Harten-Lax-van Leer-Contact (HLLC) approximate Riemann solver, and two-stage strong-stability-preserving Runge-Kutta (SSP-RK) time integration. Numba just-in-time (JIT) compilation accelerates the performance-critical kernels. The solver is verified end-to-end against the four canonical Riemann configurations: wet-bed dam break, dry-bed dam break, double rarefaction, and double shock. A six-component post-processing pipeline quantifies space-time topology, final-time error norms with empirical quantile decomposition, self-similarity collapse onto the analytical Riemann fan, integral-norm evolution, boundary-flux-corrected mass and energy diagnostics, and phase-plane analysis against analytical wave curves. The implementation conserves discrete mass to floating-point precision, satisfies discrete entropy admissibility identically, and reproduces all four analytical wave-curve geometries to within sub-centimetre accuracy in the depth-velocity phase plane. The complete source code, analytical-solution evaluators, post-processing scripts, and Network Common Data Format (NetCDF) archives are released under the MIT license.
Distributed optimal control problems governed by poroelasticity equations
arXiv:2605.30839v1 Announce Type: cross Abstract: In this paper, we propose and analyze a novel two-field symmetric formulation with solid displacement and fluid pressure as main unknowns for the Biot's consolidation model in poroelasticity. Firstly, we prove the well-posedness of the new formulation and then show the existence and uniqueness of optimal control where the fluid sources in the model act as a control variable. We prove a priori error estimates for the fully discrete scheme with backward Euler time discretization and a variational approximation of the control variable. A numerical example is presented to validate the performance of the proposed novel scheme.
Sequential Subspace Noise Injection Prevents Accuracy Collapse in Certified Unlearning
arXiv:2601.05134v2 Announce Type: replace Abstract: Certified unlearning based on differential privacy offers strong guarantees but remains largely impractical: the noisy fine-tuning approaches proposed so far achieve these guarantees but severely reduce model accuracy. We propose sequential noise scheduling, which distributes the noise budget across orthogonal subspaces of the parameter space, rather than injecting it all at once. This simple modification mitigates the destructive effect of noise while preserving the original certification guarantees. We extend the analysis of noisy fine-tuning to the subspace setting, proving that the same $(\varepsilon,\delta)$ privacy budget is retained. Empirical results on image classification benchmarks show that our approach substantially improves accuracy after unlearning while remaining robust to membership inference attacks. These results show that certified unlearning can achieve both rigorous guarantees and practical utility.
Shallow Electronic State Preparation for Quantum Chemistry with Quantum Monte Carlo Pre-Selection
arXiv:2605.31139v1 Announce Type: cross Abstract: Quantum computers hold great promise for molecular simulation, but noise remains a fundamental obstacle. We introduce a Quantum Monte Carlo (QMC) pre-screening procedure that constructs compact, physically motivated Givens rotation ans\"atze tailored to realistic quantum hardware. By identifying the most important wavefunction contributions early in a QMC simulation, we build circuits that are shallower that conventional alternatives while preserving number symmetry. Benchmarked on Quantinuum System Model H1, QMC-prescreened circuits outperform more complex ans\"atze under realistic noise conditions. The method offers a practical path toward chemical accuracy on quantum devices, by providing an adjustable trade-off between expressivity and circuit depth to generate shallow circuits suited to current high-noise devices, as well as deeper, more expressive circuits that can be deployed on future lower-noise devices.
The Nonparametric Kiefer-Weiss Problem
arXiv:2605.31465v1 Announce Type: cross Abstract: A nonparametric variant of the Kiefer-Weiss problem is proposed and solved. The objective is to minimize a weighted sum of the error probabilities of a binary sequential test subject to a constraint on its maximum expected sample size. This maximum is taken over all possible probability distributions on the given sequence space. First, it is shown that the nonparametric Kiefer-Weiss problem can be reduced to an optimal stopping problem. Then, the optimal stopping policy is derived under the assumption that at most k uses of randomization are permitted during any run of the test. The solution to the original problem is then obtained by letting k go to infinity. The optimal cost function is shown to be the solution of a nonlinear Bellman equation. The corresponding optimal stopping policy is shown to be based on a two-dimensional test statistic, with one component tracking the likelihood ratio and the other one tracking the expected remaining sample size. Critically, the stopping policy uses randomization to increase the remaining expected sample size for some runs, while stopping early for others. The optimal randomization rule is shown to be determined by a function that maps the likelihood ratio to an integer-valued sample size. Two approximations of this function are proposed that can be evaluated easily in practice. The results are illustrated with two numerical examples of nonparametric Kiefer-Weiss tests, one for a shift in the success probability of a Bernoulli distribution, and one for a shift in the mean of a normal distribution.
Closed-form linear moments of the two-dimensional angular central Gaussian distribution
arXiv:2605.31536v1 Announce Type: cross Abstract: The polar-angle marginal of a centred bivariate Gaussian distribution, obtained after integrating out the radial coordinate, gives the two-dimensional angular central Gaussian (ACG) distribution of Tyler. While its trigonometric and vector-valued moments have been studied in detail, to our knowledge there are no explicit closed-form expressions for the \emph{linear} moments $\mathbf{E}[\theta]$ and $\mathbf{E}[\theta^{2}]$ on the natural domain $\theta\in\left]-\pi/2,\pi/2\right[$. Here \textit{linear} refers to the ordinary moments $\int\theta^{k}f(\theta)\,d\theta$ of the angle regarded as a real-valued variable, in contrast to the circular (trigonometric) moments $\mathbf{E}[e^{ik\theta}]$ customary in directional statistics. We provide such expressions: the mean is a simple arctangent of the parameters, while the second moment is given by the real part of a dilogarithm. The derivation, based on a contour integration around the branch cut of $\arctan z$, is elementary. These quantities naturally arise in physics, where $\theta$ is interpreted as a real-valued phase rather than a circular variable.
Recovering the Shape of a Contact Line
arXiv:2605.31542v1 Announce Type: cross Abstract: We study the conditions for a three-phase contact line to return to a previous position. We drive a water-air-glass contact line between two horizontal plates, by slowly adding and removing water with a constant volume amplitude. For the first several cycles, the contact line ends each cycle with a different shape, in contrast with previously published work. Eventually the shapes begin to repeat, and the system has memory: a cycle with a smaller amplitude ends in a different shape, but even one cycle at the original amplitude recovers the steady-state shape. After a cycle at a larger amplitude, the steady-state shape is erased. We find that our tight control of the enclosed volume creates a global interaction, wherein only the least stable part of the contact line can move. Using theory and minimal models, we show that this interaction gives rise to the transient behaviors. Our study sheds light on the origins of reversibility and memory in a system where neither is guaranteed, and shows that the physics of contact line motion changes in a confined environment.
Towards a Framework for Social Mechanics
arXiv:2402.12638v3 Announce Type: replace Abstract: Social physics explores the possibility that mathematical structures developed in physics may provide useful descriptions of certain social phenomena. In this work, we propose an effective mechanical framework for modelling social change in terms of positions in a space of social stances, together with concepts analogous to motion, inertia, interaction, and force. A central feature of the framework is the introduction of position-dependent inertial responses, allowing susceptibility to social change to vary across stance-space. Within this setting, we investigate deterministic and stochastic models of social evolution, including free motion, effective interactions, and diffusion-driven dynamics. We also discuss Lagrangian and Hamiltonian formulations associated with the proposed framework. As an illustrative application, we model partisan preference distributions in United States presidential elections through effective drift and diffusion processes. The framework is intended as a phenomenological and exploratory approach to social dynamics rather than as a fundamental description of human behaviour.
HYGENE: A Diffusion-based Hypergraph Generation Method
arXiv:2408.16457v5 Announce Type: replace Abstract: Hypergraphs are powerful mathematical structures that can model complex, high-order relationships in various domains, including social networks, bioinformatics, and recommender systems. However, generating realistic and diverse hypergraphs remains challenging due to their inherent complexity and lack of effective generative models. In this paper, we introduce a diffusion-based Hypergraph Generation (HYGENE) method that addresses these challenges through a progressive local expansion approach. HYGENE works on the bipartite representation of hypergraphs, starting with a single pair of connected nodes and iteratively expanding it to form the target hypergraph. At each step, nodes and hyperedges are added in a localized manner using a denoising diffusion process, which allows for the construction of the global structure before refining local details. Our experiments demonstrated the effectiveness of HYGENE, proving its ability to closely mimic a variety of properties in hypergraphs. To the best of our knowledge, this is the first attempt to employ deep learning models for hypergraph generation, and our work aims to lay the groundwork for future research in this area.
Minibatch Optimal Transport and Perplexity Bound Estimation in Discrete Flow Matching
arXiv:2411.00759v5 Announce Type: replace Abstract: Discrete flow matching, a recent framework for modeling categorical data, has shown competitive performance with autoregressive models. However, unlike continuous flow matching, the rectification strategy cannot be applied due to the stochasticity of discrete paths, necessitating alternative methods to minimize state transitions. We propose a dynamic-optimal-transport-like minimization objective and derive its Kantorovich formulation for discrete flows with convex interpolants, where transport cost depends solely on inter-state dissimilarity and can be optimized via minibatch strategies. We show that such methods can reduce the number of transitions up to 32 times (1024 to 32) to reach the same generative perplexity without compromising diversity. Additionally, path nondeterminism in discrete flows precludes an instantaneous change-of-variables analogue, preventing precise probability estimation available to continuous flows. We therefore propose two upper bounds on perplexity, enabling principled training, evaluation and model comparison. Finally, we introduce Multimask Flows which outperform masked flows in generative perplexity without compromising diversity, particularly when utilizing minibatch Optimal Transport.
How can embedding models bind concepts?
arXiv:2605.31503v1 Announce Type: new Abstract: Humans easily determine which color belongs to which shape in multi-object scenes, an ability known as concept binding. Vision-language embedding models such as CLIP struggle with binding: they recognize individual concepts but fail to represent which concepts form which objects. Although CLIP behaves like a bag-of-concepts model in cross-modal retrieval, object information is recoverable from its image and text embeddings separately. We study this tension through the binding function, which maps concepts to scene embeddings. We find that scene embeddings decompose additively into object representations, explaining why uni-modal probes can recover object information. However, CLIP's binding function is high-complexity, which likely prevents the image and text encoders from learning a shared binding mechanism that generalizes to unseen concept combinations. We then ask whether this limitation is fundamental. We show that it is not. In controlled transformer models trained from scratch, binding generalization emerges with sufficient data coverage. These models learn low-complexity binding functions characterized by multiplicative interactions between concepts, enabling systematic generalization. Code is publicly available at https://github.com/oshapio/binding-concepts-complexity.
Practical Algebraic Stepping with Scoped Filters
arXiv:2605.31517v1 Announce Type: new Abstract: Algebraic steppers help students learn functional programming by displaying evaluation as a sequence of small-step reductions, but even simple programs produce long traces in which key ideas are buried under mundane reductions. This paper presents the filtered stepper calculus, a formal framework that gives users scoped, pattern-based control over which reduction steps are shown or hidden. Users annotate programs with lightweight filter expressions that match on the structure of redexes. Filters compose via lexical scoping so that inner filters override outer ones. We prove preservation, progress, and a simulation theorem establishing that the filtered stepper agrees with the underlying unfiltered semantics, and mechanize all proofs in Agda. We implement the calculus in the Hazel live programming environment, including its support for stepping programs with holes and type errors. To do so, we reconcile Hazel's internal environment-based evaluator with the substitution-based presentation expected in the classroom. We deploy the system in a university programming languages course. Our evaluation shows that students adopt the stepper organically, though more advanced uses of filters may require further instruction, and that instructors can use filters to craft focused traces for use in lectures.
On the Relationship Between Activation Outliers and Feature Death in Sparse Autoencoders
arXiv:2605.31518v1 Announce Type: new Abstract: Sparse autoencoders (SAEs) decompose neural network activations into interpretable features, but many learned features never activate, a problem called feature death that wastes dictionary capacity and can reintroduce superposition. Death rates vary dramatically between models: near-zero on GPT-2, over 70% on AlphaFold3 with identical configurations. We find that dimension-level activation outliers (dimensions whose mean magnitude is large relative to per-token variation) cause this by shifting pre-activations at initialization based on each feature's alignment with the activation mean. Features anti-aligned with the mean receive permanently negative pre-activations and never fire. We formalize outlier severity as $\gamma = \|\mu\|/\|\sigma\|$; it predicts initial death rates (Spearman $\rho = 0.89$ for dead-by-TopK, $0.82$ for dead-by-ReLU) across 454 model-layer combinations spanning language, vision, protein, and genomic models. Dead features can revive during training, but recovery requires the SAE bias to learn the activation mean, a process that is prohibitively slow at high $\gamma$. Mean-centering (subtracting the activation mean) sidesteps this and eliminates outlier-induced death across all tested models, confirming the mechanism and providing a principled basis for when and why this preprocessing step is necessary.
BERS: Locally Optimal Continuous Algorithm for Maritime Weather Routing with Just-in-Time Arrival
arXiv:2605.31533v1 Announce Type: new Abstract: Maritime weather routing must optimize route geometry under dynamic wind-wave conditions, obstacle constraints, and fixed-arrival requirements. We present B\'ezier Evolve and Refine Strategy (\name{}), a two-stage framework that combines global evolutionary search (CMA-ES) with local variational refinement (FMS). Routes are parametrized as B\'ezier curves and evaluated with dense along-path sampling, enabling smooth trajectories while preserving practical feasibility constraints and accounting for mid-segment effects. We evaluate \name{} on synthetic benchmarks designed to stress seven operational criteria: continuity, obstacle avoidance, dynamic adaptation, flexible objective design, constant-load feasibility, just-in-time arrival, and local optimality. Across these tests, \name{} matches or improves published baselines while maintaining robust convergence under challenging flow fields and land geometries. We then validate the method on real ocean data using hourly ERA5 forcing over 366 daily departures in 2024 for two trans-oceanic corridors (Atlantic and Pacific), with a physics-based model of an 88~m cargo vessel with optional rigid wingsails. In real-ocean experiments, route optimization alone reduces mean propulsive energy by 23--59\% versus great-circle baselines of the same propulsion mode. Combined with wind-assisted propulsion, total savings reach up to 75\%. These results show that \name{} provides a practical and scalable foundation for just-in-time, energy-efficient weather routing in maritime decarbonization workflows.
HUNT: High-Speed UAV Navigation and Tracking in Unstructured Environments via Instantaneous Relative Frames
arXiv:2509.19452v4 Announce Type: replace Abstract: Search and rescue operations require unmanned aerial vehicles to both traverse unknown unstructured environments at high speed and track targets once detected. Achieving both capabilities under degraded sensing and without global localization remains an open challenge. Recent works on relative navigation have shown robust tracking by anchoring planning and control to a visible detected object, but cannot address navigation when no target is in the field of view. We present HUNT (High-speed UAV Navigation and Tracking), a real-time framework that unifies traversal, acquisition, and tracking within a single relative formulation. HUNT defines navigation objectives directly from onboard instantaneous observables such as attitude, altitude, and velocity, enabling reactive high-speed flight during search. Once a target is detected, the same perception-control pipeline transitions seamlessly to tracking. Outdoor experiments in dense forests, container compounds, and search-and-rescue operations with vehicles and mannequins demonstrate robust autonomy where global methods fail.
Towards Atoms of Large Language Models
arXiv:2509.20784v3 Announce Type: replace Abstract: The fundamental representational units (FRUs) of large language models (LLMs) remain undefined, limiting further understanding of their underlying mechanisms. In this paper, we introduce Atom Theory to systematically define, evaluate, and identify such FRUs, which we term atoms. Building on the atomic inner product (AIP), a non-Euclidean metric that captures the underlying geometry of LLM representations, we formally define atoms and propose two key criteria for ideal atoms: faithfulness ($R^2$) and stability ($q^*$). We further prove that atoms are identifiable under threshold-activated sparse autoencoders (TSAEs). Empirically, we uncover a pervasive representation shift in LLMs and demonstrate that the AIP corrects this shift to capture the underlying representational geometry. We find that two widely used units, neurons and features, fail to qualify as ideal atoms: neurons are faithful ($R^2=1$) but unstable ($q^*=0.5\%$), while features are more stable ($q^*=68.2\%$) but unfaithful ($R^2=48.8\%$). To find atoms of LLMs, leveraging atom identifiability under TSAEs, we show via large-scale experiments that reliable atom identification occurs only when the TSAE capacity matches the data scale. Guided by this insight, we identify FRUs with near-perfect faithfulness ($R^2=99.9\%$) and stability ($q^*=99.8\%$) across layers of Gemma2-2B, Gemma2-9B, and Llama3.1-8B, satisfying the criteria of ideal atoms statistically. Further analysis confirms that these atoms align with theoretical expectations and exhibit substantially higher monosemanticity. Overall, we propose and validate Atom Theory as a foundation for understanding the internal representations of LLMs. Code available at https://github.com/ChenhuiHu/towards_atoms.
Performance and Complexity Trade-off Optimization of Speech Models During Training
arXiv:2601.13704v3 Announce Type: replace Abstract: In speech machine learning, neural network models are typically designed by choosing an architecture with fixed layer sizes and structure. These models are then trained to maximize performance on metrics aligned with the task's objective. While the overall architecture is usually guided by prior knowledge of the task, the sizes of individual layers are often chosen heuristically. However, this approach does not guarantee an optimal trade-off between performance and computational complexity; consequently, post hoc methods such as weight quantization or model pruning are typically employed to reduce computational cost. This occurs because stochastic gradient descent (SGD) methods can only optimize differentiable functions, while factors influencing computational complexity, such as layer sizes and floating-point operations per second (FLOP/s), are non-differentiable and require modifying the model structure during training. We propose a reparameterization technique based on feature noise injection that enables joint optimization of performance and computational complexity during training using SGD-based methods. Unlike traditional pruning methods, our approach allows the model size to be dynamically optimized for a target performance-complexity trade-off, without relying on heuristic criteria to select which weights or structures to remove. We demonstrate the effectiveness of our method through three case studies, including a synthetic example and two practical real-world applications: voice activity detection and audio anti-spoofing. The code related to our work is publicly available to encourage further research.