Publications

2026

1. Workshop ECCV EmergingAD

   Pictura: Perspective-View Self-Play at Scale for Driving

   Yuan Yin, Elias Ramzi, Marc Lafon, Valentin Charraut, Victor Bares, Yihong Xu, Éloi Zablocki, Alexandre Boulch, Thibault Buhet, Andrei Bursuc, and Matthieu Cord

   In ECCV 2026 Workshop on Emerging Behaviors in Embodied AI for Achieving Robust Autonomy, Sep 2026

   

   Abstract arXiv Project PDF Code

   Self-play in simulation produces robust driving policies at scale. Demonstrations of such behavior have been made using privileged vectorized observations such as exact poses and velocities, even for occluded agents. This assumes that perception is solved and introduces a representation gap with the partial observation of a deployed agent driving from the perspective view of egocentric cameras. A common fix, distilling the privileged policy into a camera-input student, leaves the student imitating decisions its own view cannot justify. Instead, we establish perspective-view self-play as a practical training regime. We introduce Pictura, a GPU-accelerated multi-agent driving simulator that renders each agent’s egocentric view at every step, mitigating the representation gap at its source. Pictura sustains up to 500K agent-steps/s (2M images/s) on a single H100. Using Pictura, we train Alberti by self-play with plain PPO. It is the first large-scale driving self-play policy trained directly from perspective images, without privileged observations. Training spans 50B agent steps for  35M km of driving. It approaches the driving performance of its privileged vectorized counterpart, and transfers zero-shot to Waymo Open Motion Dataset layouts re-rendered in Pictura, where it outperforms privileged vectorized agents.

2. Preprint

   TOAD: Test-Time Trajectory Optimization for Autonomous Driving

   Yihong Xu, Éloi Zablocki, Yuan Yin, Elias Ramzi, Ellington Kirby, Alexandre Boulch, and Matthieu Cord

   Jun 2026

   

   Abstract arXiv Project PDF Code

   End-to-end planners for autonomous driving typically generate a set of candidate trajectories, score each one, and return the highest-scoring candidate. However, the scorer is applied only after the proposals are generated and cannot influence the set of trajectories: a weak set of candidates limits planning performance regardless of the scorer’s quality. We instead treat the scorer as a learned trajectory-level reward function and search for trajectories that maximize it. Our method, TOAD, runs the Cross-Entropy Method at test time, warm-started from the planner’s proposals. It requires no retraining and is plug-and-play for existing planners. Across six base planners, TOAD improves results on NAVSIM-v1 (94.7 PDMS), NAVSIM-v2 (56.3 EPDMS), and the closed-loop HUGSIM benchmark.

3. Conference CVPR

   Driving on Registers

   Ellington Kirby, Alexandre Boulch, Yihong Xu, Yuan Yin, Gilles Puy, Éloi Zablocki, Andrei Bursuc, Spyros Gidaris, Renaud Marlet, Florent Bartoccioni, Anh-Quan Cao, Nermin Samet, Tuan-Hung VU, and Matthieu Cord

   In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Jun 2026

   

   Abstract arXiv Project PDF Code

   We present DrivoR, a simple and efficient transformer-based architecture for end-to-end autonomous driving. Our approach builds on pretrained Vision Transformers (ViTs) and introduces camera-aware register tokens that compress multi-camera features into a compact scene representation, significantly reducing downstream computation without sacrificing accuracy. These tokens drive two lightweight transformer decoders that generate and then score candidate trajectories. The scoring decoder learns to mimic an oracle and predicts interpretable sub-scores representing aspects such as safety, comfort, and efficiency, enabling behavior-conditioned driving at inference. Despite its minimal design, DrivoR outperforms or matches strong contemporary baselines across NAVSIM-v1, NAVSIM-v2, and the photorealistic closed-loop HUGSIM benchmark. Our results show that a pure-transformer architecture, combined with targeted token compression, is sufficient for accurate, efficient, and adaptive end-to-end driving. Code and checkpoints will be made available.

4. Conference ICRA

   PPT: Pre-Training with Pseudo-Labeled Trajectories for Motion Forecasting

   Yihong Xu^*, Yuan Yin^*, Tuan-Hung Vu, Alexandre Boulch, Éloi Zablocki, and Matthieu Cord

   In The 2026 IEEE International Conference on Robotics and Automation, Jan 2026

   

   Abstract arXiv PDF

   Motion forecasting (MF) for autonomous driving aims at anticipating trajectories of surrounding agents in complex urban scenarios. In this work, we investigate a mixed strategy in MF training that first pre-train motion forecasters on pseudo-labeled data, then fine-tune them on annotated data. To obtain pseudo-labeled trajectories, we propose a simple pipeline that leverages off-the-shelf single-frame 3D object detectors and non-learning trackers. The whole pre-training strategy including pseudo-labeling is coined as PPT. Our extensive experiments demonstrate that: (1) combining PPT with supervised fine-tuning on annotated data achieves superior performance on diverse testbeds, especially under annotation-efficient regimes, (2) scaling up to multiple datasets improves the previous state-of-the-art and (3) PPT helps enhance cross-dataset generalization. Our findings showcase PPT as a promising pre-training solution for robust motion forecasting in diverse autonomous driving contexts.

2025

1. Journal TMLR + Workshop NeurIPS CCFM

   IPA: An Information-Reconstructive Input Projection Framework for Efficient Foundation Model Adaptation

   Yuan Yin, Shashanka Venkataramanan, Tuan-Hung Vu, Andrei Bursuc, and Matthieu Cord

   Transactions on Machine Learning Research, Sep 2025

   

   Best Paper at NeurIPS 2025 CCFM Workshop Abstract arXiv OpenReview PDF Code

   IPA received the Best Paper Award at NeurIPS 2025 CCFM Workshop.

   Parameter-efficient fine-tuning (PEFT) methods, such as LoRA, reduce adaptation cost by injecting low-rank updates into pretrained weights. However, LoRA’s down-projection is randomly initialized and data-agnostic, discarding potentially useful information. Prior analyses show that this projection changes little during training, while the up-projection carries most of the adaptation, making the random input compression a performance bottleneck. We propose IPA, a feature-aware projection framework that explicitly preserves information in the reduced hidden space. In the linear case, we instantiate IPA with algorithms approximating top principal components, enabling efficient projector pretraining with negligible inference overhead. Across language and vision benchmarks, IPA consistently improves over LoRA and DoRA, achieving on average 1.5 points higher accuracy on commonsense reasoning and 2.3 points on VTAB-1k, while matching full LoRA performance with roughly half the trainable parameters when the projection is frozen.

2. Conference ICLR

   Learning a Neural Solver for Parametric PDE to Enhance Physics-Informed Methods

   Lise Le Boudec, Emmanuel de Bézenac, Louis Serrano, Ramón Daniel Regueiro Espiño, Yuan Yin, and Patrick Gallinari

   In The Thirteenth International Conference on Learning Representations, May 2025

   

   Abstract arXiv OpenReview PDF Code

   Physics-informed deep learning often faces optimization challenges due to the complexity of solving partial differential equations (PDEs), which involve exploring large solution spaces, require numerous iterations, and can lead to unstable training. These challenges arise particularly from the ill-conditioning of the optimization problem, caused by the differential terms in the loss function. To address these issues, we propose learning a solver, i.e., solving PDEs using a physics-informed iterative algorithm trained on data. Our method learns to condition a gradient descent algorithm that automatically adapts to each PDE instance, significantly accelerating and stabilizing the optimization process and enabling faster convergence of physics-aware models. Furthermore, while traditional physics-informed methods solve for a single PDE instance, our approach addresses parametric PDEs. Specifically, our method integrates the physical loss gradient with the PDE parameters to solve over a distribution of PDE parameters, including coefficients, initial conditions, or boundary conditions. We demonstrate the effectiveness of our method through empirical experiments on multiple datasets, comparing training and test-time optimization performance.

2024

1. Conference NeurIPS

   GEPS: Boosting Generalization in Parametric PDE Neural Solvers through Adaptive Conditioning

   Armand Kassaï Koupaï, Jorge Mifsut Benet, Yuan Yin, Jean-Noël Vittaut, and Patrick Gallinari

   In The Thirty-eighth Annual Conference on Neural Information Processing Systems, Dec 2024

   

   Abstract arXiv OpenReview Proceedings PDF Code

   Solving parametric partial differential equations (PDEs) presents significant challenges for data-driven methods due to the sensitivity of spatio-temporal dynamics to variations in PDE parameters. Machine learning approaches often struggle to capture this variability. To address this, data-driven approaches learn parametric PDEs by sampling a very large variety of trajectories with varying PDE parameters. We first show that incorporating conditioning mechanisms for learning parametric PDEs is essential and that among them, adaptive conditioning, allows stronger generalization. As existing adaptive conditioning methods do not scale well with respect to the number of parameters to adapt in the neural solver, we propose GEPS, a simple adaptation mechanism to boost GEneralization in Pde Solvers via a first-order optimization and low-rank rapid adaptation of a small set of context parameters. We demonstrate the versatility of our approach for both fully data-driven and for physics-aware neural solvers. Validation performed on a whole range of spatio-temporal forecasting problems demonstrates excellent performance for generalizing to unseen conditions including initial conditions, PDE coefficients, forcing terms and solution domain.

2. Workshop ECCV W-CODA

   ReGentS: Real-World Safety-Critical Driving Scenario Generation Made Stable

   Yuan Yin, Pegah Khayatan, Éloi Zablocki, Alexandre Boulch, and Matthieu Cord

   In ECCV 2024 Workshop on Multimodal Perception and Comprehension of Corner Cases in Autonomous Driving, Sep 2024

   

   Abstract arXiv Proceedings PDF Code

   Machine learning based autonomous driving systems often face challenges with safety-critical scenarios that are rare in real-world data, hindering their large-scale deployment. While increasing real-world training data coverage could address this issue, it is costly and dangerous. This work explores generating safety-critical driving scenarios by modifying complex real-world regular scenarios through trajectory optimization. We propose ReGentS, which stabilizes generated trajectories and introduces heuristics to avoid obvious collisions and optimization problems. Our approach addresses unrealistic diverging trajectories and unavoidable collision scenarios that are not useful for training robust planner. We also extend the scenario generation framework to handle real-world data with up to 32 agents. Additionally, by using a differentiable simulator, our approach simplifies gradient descent-based optimization involving a simulator, paving the way for future advancements.

3. Journal TMLR

   Time Series Continuous Modeling for Imputation and Forecasting with Implicit Neural Representations

   Étienne Le Naour, Louis Serrano, Léon Migus, Yuan Yin, Ghislain Agoua, Nicolas Baskiotis, Patrick Gallinari, and Vincent Guigue

   Transactions on Machine Learning Research, Apr 2024

   

   Abstract arXiv OpenReview PDF Code

   19

   We introduce a novel modeling approach for time series imputation and forecasting, tailored to address the challenges often encountered in real-world data, such as irregular samples, missing data, or unaligned measurements from multiple sensors. Our method relies on a continuous-time-dependent model of the series’ evolution dynamics. It leverages adaptations of conditional, implicit neural representations for sequential data. A modulation mechanism, driven by a meta-learning algorithm, allows adaptation to unseen samples and extrapolation beyond observed time-windows for long-term predictions. The model provides a highly flexible and unified framework for imputation and forecasting tasks across a wide range of challenging scenarios. It achieves state-of-the-art performance on classical benchmarks and outperforms alternative time-continuous models.

2023

1. Thesis PhD

   Physics-Aware Deep Learning and Dynamical Systems: Hybrid Modeling and Generalization. (Apprentissage profond pour la physique et les systèmes dynamiques : modélisation hybride et généralisation)

   Yuan Yin

   Sorbonne University, Paris, France, Jun 2023

   

   Accessit ex æquo to the AFIA IA 2024 Thesis Prize Abstract HAL PDF

   5

   Yuan Yin received the Accessit to the AFIA IA 2024 Thesis Prize for his PhD thesis “Physics-Aware Deep Learning and Dynamical Systems: Hybrid Modeling and Generalization” supervised by Patrick Gallinari. See https://afia.asso.fr/le-prix-de-these-afia/.

   Deep learning has made significant progress in various fields and has emerged as a promising tool for modeling physical dynamical phenomena that exhibit highly nonlinear relationships. However, existing approaches are limited in their ability to make physically sound predictions due to the lack of prior knowledge and to handle real-world scenarios where data comes from multiple dynamics or is irregularly distributed in time and space. This thesis aims to overcome these limitations in the following directions: improving neural network-based dynamics modeling by leveraging physical models through hybrid modeling; extending the generalization power of dynamics models by learning commonalities from data of different dynamics to extrapolate to unseen systems; and handling free-form data and continuously predicting phenomena in time and space through continuous modeling. We highlight the versatility of deep learning techniques, and the proposed directions show promise for improving their accuracy and generalization power, paving the way for future research in new applications.

2. Conference ICLR

   Continuous PDE Dynamics Forecasting with Implicit Neural Representations

   Yuan Yin^*, Matthieu Kirchmeyer^*, Jean-Yves Franceschi, Alain Rakotomamonjy, and Patrick Gallinari

   In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023, May 2023

   

   Abstract arXiv OpenReview PDF Code

   133

   Effective data-driven PDE forecasting methods often rely on fixed spatial and / or temporal discretizations. This raises limitations in real-world applications like weather prediction where flexible extrapolation at arbitrary spatiotemporal locations is required. We address this problem by introducing a new data-driven approach, DINo, that models a PDE’s flow with continuous-time dynamics of spatially continuous functions. This is achieved by embedding spatial observations independently of their discretization via Implicit Neural Representations in a small latent space temporally driven by a learned ODE. This separate and flexible treatment of time and space makes DINo the first data-driven model to combine the following advantages. It extrapolates at arbitrary spatial and temporal locations; it can learn from sparse irregular grids or manifolds; at test time, it generalizes to new grids or resolutions. DINo outperforms alternative neural PDE forecasters in a variety of challenging generalization scenarios on representative PDE systems.

3. Conference NeurIPS

   Operator Learning with Neural Fields: Tackling PDEs on General Geometries

   Louis Serrano, Lise Le Boudec, Armand Kassaï Koupaï, Thomas X. Wang, Yuan Yin, Jean-Noël Vittaut, and Patrick Gallinari

   In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, December 10-16, 2023, New Orleans, Louisiana, USA, Dec 2023

   

   Abstract arXiv OpenReview Proceedings PDF Code

   121

   Machine learning approaches for solving partial differential equations require learning mappings between function spaces. While convolutional or graph neural networks are constrained to discretized functions, neural operators present a promising milestone toward mapping functions directly. Despite impressive results they still face challenges with respect to the domain geometry and typically rely on some form of discretization. In order to alleviate such limitations, we present CORAL, a new method that leverages coordinate-based networks for solving PDEs on general geometries. CORAL is designed to remove constraints on the input mesh, making it applicable to any spatial sampling and geometry. Its ability extends to diverse problem domains, including PDE solving, spatio-temporal forecasting, and inverse problems like geometric design. CORAL demonstrates robust performance across multiple resolutions and performs well in both convex and non-convex domains, surpassing or performing on par with state-of-the-art models.

4. Workshop ICML SynS&ML

   INFINITY: Neural Field Modeling for Reynolds-Averaged Navier-Stokes Equations

   Louis Serrano, Léon Migus, Yuan Yin, Jocelyn Ahmed Mazari, and Patrick Gallinari

   In ICML 2023 Workshop on Synergy of Scientific and Machine Learning Modeling, Jul 2023

   

   Abstract arXiv OpenReview PDF

   For numerical design, the development of efficient and accurate surrogate models is paramount. They allow us to approximate complex physical phenomena, thereby reducing the computational burden of direct numerical simulations. We propose INFINITY, a deep learning model that utilizes implicit neural representations (INRs) to address this challenge. Our framework encodes geometric information and physical fields into compact representations and learns a mapping between them to infer the physical fields. We use an airfoil design optimization problem as an example task and we evaluate our approach on the challenging AirfRANS dataset, which closely resembles real-world industrial use-cases. The experimental results demonstrate that our framework achieves state-of-the-art performance by accurately inferring physical fields throughout the volume and surface. Additionally we demonstrate its applicability in contexts such as design exploration and shape optimization: our model can correctly predict drag and lift coefficients while adhering to the equations.

5. Journal JDSA

   Improving Trust and Confidence in Medical Skin Lesion Diagnosis Through Explainable Deep Learning

   Carlo Metta, Andrea Beretta, Riccardo Guidotti, Yuan Yin, Patrick Gallinari, Salvatore Rinzivillo, and Fosca Giannotti

   International Journal of Data Science and Analytics, Jun 2023

   Abstract Journal

   A key issue in critical contexts such as medical diagnosis is the interpretability of the deep learning models adopted in decision-making systems. Research in eXplainable Artificial Intelligence (XAI) is trying to solve this issue. However, often XAI approaches are only tested on generalist classifier and do not represent realistic problems such as those of medical diagnosis. In this paper, we aim at improving the trust and confidence of users towards automatic AI decision systems in the field of medical skin lesion diagnosis by customizing an existing XAI approach for explaining an AI model able to recognize different types of skin lesions. The explanation is generated through the use of synthetic exemplar and counter-exemplar images of skin lesions and our contribution offers the practitioner a way to highlight the crucial traits responsible for the classification decision. A validation survey with domain experts, beginners, and unskilled people shows that the use of explanations improves trust and confidence in the automatic decision system. Also, an analysis of the latent space adopted by the explainer unveils that some of the most frequent skin lesion classes are distinctly separated. This phenomenon may stem from the intrinsic characteristics of each class and may help resolve common misclassifications made by human experts.

2022

1. Conference ICML

   Generalizing to New Physical Systems via Context-Informed Dynamics Model

   Matthieu Kirchmeyer^*, Yuan Yin^*, Jérémie Donà, Nicolas Baskiotis, Alain Rakotomamonjy, and Patrick Gallinari

   In International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, Jul 2022

   

   Abstract arXiv Proceedings PDF Code

   89

   Data-driven approaches to modeling physical systems fail to generalize to unseen systems that share the same general dynamics with the learning domain, but correspond to different physical contexts. We propose a new framework for this key problem, context-informed dynamics adaptation (CoDA), which takes into account the distributional shift across systems for fast and efficient adaptation to new dynamics. CoDA leverages multiple environments, each associated to a different dynamic, and learns to condition the dynamics model on contextual parameters, specific to each environment. The conditioning is performed via a hypernetwork, learned jointly with a context vector from observed data. The proposed formulation constrains the search hypothesis space to foster fast adaptation and better generalization across environments. We theoretically motivate our approach and show state-of-the-art generalization results on a set of nonlinear dynamics, representative of a variety of application domains. We also show, on these systems, that new system parameters can be inferred from context vectors with minimal supervision.

2. Workshop ICLR WS

   Multi-scale Physical Representations for Approximating PDE Solutions with Graph Neural Operators

   Léon Migus, Yuan Yin, Jocelyn Ahmed Mazari, and Patrick Gallinari

   In Proceedings of Topological, Algebraic, and Geometric Learning Workshops 2022, May 2022

   

   Abstract arXiv OpenReview Proceedings PDF

   Representing physical signals at different scales is among the most challenging problems in engineering. Several multi-scale modeling tools have been developed to describe physical systems governed by \emphPartial Differential Equations (PDEs). These tools are at the crossroad of principled physical models and numerical schema. Recently, data-driven models have been introduced to speed-up the approximation of PDE solutions compared to numerical solvers. Among these recent data-driven methods, neural integral operators are a class that learn a mapping between function spaces. These functions are discretized on graphs (meshes) which are appropriate for modeling interactions in physical phenomena. In this work, we study three multi-resolution schema with integral kernel operators that can be approximated with Message Passing Graph Neural Networks (MPGNNs). To validate our study, we make extensive MPGNNs experiments with well-chosen metrics considering steady and unsteady PDEs.

2021

1. Conference NeurIPS

   LEADS: Learning Dynamical Systems that Generalize Across Environments

   Yuan Yin, Ibrahim Ayed, Emmanuel de Bézenac, Nicolas Baskiotis, and Patrick Gallinari

   In Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual, Dec 2021

   

   Abstract arXiv OpenReview Proceedings PDF Code

   75

   When modeling dynamical systems from real-world data samples, the distribution of data often changes according to the environment in which they are captured, and the dynamics of the system itself vary from one environment to another. Generalizing across environments thus challenges the conventional frameworks. The classical settings suggest either considering data as i.i.d. and learning a single model to cover all situations or learning environment-specific models. Both are sub-optimal: the former disregards the discrepancies between environments leading to biased solutions, while the latter does not exploit their potential commonalities and is prone to scarcity problems. We propose LEADS, a novel framework that leverages the commonalities and discrepancies among known environments to improve model generalization. This is achieved with a tailored training formulation aiming at capturing common dynamics within a shared model while additional terms capture environment-specific dynamics. We ground our approach in theory, exhibiting a decrease in sample complexity with our approach and corroborate these results empirically, instantiating it for linear dynamics. Moreover, we concretize this framework for neural networks and evaluate it experimentally on representative families of nonlinear dynamics. We show that this new setting can exploit knowledge extracted from environment-dependent data and improves generalization for both known and novel environments.

2. Conference ICLR + Journal JSTAT

   Augmenting Physical Models with Deep Networks for Complex Dynamics Forecasting

   Yuan Yin^*, Vincent Le Guen^*, Jérémie Donà^*, Emmanuel de Bézenac^*, Ibrahim Ayed^*, Nicolas Thome, and Patrick Gallinari

   In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021, May 2021

   

   Abstract arXiv OpenReview PDF Code

   264

   Forecasting complex dynamical phenomena in settings where only partial knowledge of their dynamics is available is a prevalent problem across various scientific fields. While purely data-driven approaches are arguably insufficient in this context, standard physical modeling based approaches tend to be over-simplistic, inducing non-negligible errors. In this work, we introduce the APHYNITY framework, a principled approach for augmenting incomplete physical dynamics described by differential equations with deep data-driven models. It consists in decomposing the dynamics into two components: a physical component accounting for the dynamics for which we have some prior knowledge, and a data-driven component accounting for errors of the physical model. The learning problem is carefully formulated such that the physical model explains as much of the data as possible, while the data-driven component only describes information that cannot be captured by the physical model, no more, no less. This not only provides the existence and uniqueness for this decomposition, but also ensures interpretability and benefits generalization. Experiments made on three important use cases, each representative of a different family of phenomena, i.e. reaction-diffusion equations, wave equations and the non-linear damped pendulum, show that APHYNITY can efficiently leverage approximate physical models to accurately forecast the evolution of the system and correctly identify relevant physical parameters.

2020

1. Preprint

   Unsupervised Spatiotemporal Data Inpainting

   Yuan Yin, Arthur Pajot, Emmanuel de Bézenac, and Patrick Gallinari

   Sep 2020

   

   Abstract OpenReview PDF

   We tackle the problem of inpainting occluded area in spatiotemporal sequences, such as cloud occluded satellite observations, in an unsupervised manner. We place ourselves in the setting where there is neither access to paired nor unpaired training data. We consider several cases in which the underlying information of the observed sequence in certain areas is lost through an observation operator. In this case, the only available information is provided by the observation of the sequence, the nature of the measurement process and its associated statistics. We propose an unsupervised-learning framework to retrieve the most probable sequence using a generative adversarial network. We demonstrate the capacity of our model to exhibit strong reconstruction capacity on several video datasets such as satellite sequences or natural videos.

2019

1. Workshop CI

   Unsupervised Inpainting for Occluded Sea Surface Temperature Sequences

   Yuan Yin, Arthur Pajot, Emmanuel de Bézenac, and Patrick Gallinari

   In Proceedings of the 9th International Workshop on Climate Informatics: CI 2019, Dec 2019

   

   Abstract Proceedings Code

   Sea Surface Temperature (SST) data remotely detected from satellites are inevitably occluded by clouds or rains. To learn the prior of SST data and complete the missing part, inpainting approaches in machine learning often require ground truth in the occluded area, which is not applicable for remote SST. In this paper, we propose an inpainting approach based on generative adversarial networks (GANs) for occluded SST. It is capable to complete the occluded area without any supervision on ground truth data of the occluded area. Our framework is evaluated on real SST data with simulated cloud masks.

2017

1. Journal IVC

   Local feature approach to dorsal hand vein recognition by Centroid-based Circular Key-point Grid and fine-grained matching

   Di Huang, Renke Zhang, Yuan Yin, Yiding Wang, and Yunhong Wang

   Image Vision Computing, Feb 2017

   

   Abstract Journal

   50

   Due to the great progress made by local feature matching in both performance and robustness of dorsal hand vein recognition, this paper proposes a novel and effective approach for such an issue by improving two major steps of the SIFT-like framework, i.e. key-point detection and matching. For the former, a new key-point generation pattern, namely Centroid-based Circular Key-point Grid (CCKG), is presented, which efficiently localizes a certain number of points on the dorsal hand for the following SIFT feature extraction, leading to a discriminative description. In contrast to the existing key-point detectors, CCKG comprehensively accounts for the properties of the dorsal hand, including the vein network as well as the surrounding corium region, and hence achieves both good representativeness and low complexity. For the latter, a fine-grained matching process is introduced which makes use of Multi-task Sparse Representation Classifier (MtSRC). Compared with the traditional coarse-grained one that counts the number of associated SIFT features between the gallery and probe dorsal hand images, MtSRC precisely calculates the error of each feature of the probe as reconstructed by the gallery features, and all the errors of the probe features are combined for similarity measurement, reaching a better accuracy in recognition. The proposed approach is evaluated on the NCUT Part A database and shows its effectiveness in both the identification and verification scenarios. Additionally, the experimental results achieved on the NCUT Part B dataset highlight its generality and robustness to low quality images.