The analysis of inks plays a crucial role in the examination process of questioned documents. To address this issue, we propose a new approach for ink mismatch detection in Hyperspectral document (HSD) images based on a new orthogonal and graph regularized Nonnegative Matrix Factorization (NMF) model. Although some previous works have proposed orthogonality constraints to solve clustering problems in different contexts, the application of such constraints is not straightforward due to the sum-to-one assumption related to the physical nature of Hyperspectral images. In this work, we demonstrate that under some acquisition protocols, latent factors in HSD images can be constrained to be orthogonal. We also incorporate a graph regularized term to exploit the geometric information lost by the matricization of HSD images. Furthermore, we propose an efficient alternating direction method of multipliers based algorithm to solve the proposed method. Our empirical validation demonstrates the competitiveness of the proposed algorithm compared to the state-of-the-art methods. It shows a high potential to be used as a reliable tool for quick investigation of questioned documents.