In this work, a method for automatic analysis of natural aggregates using hyperspectral imaging and high-resolution RGB imaging combined with AI algorithms consisting of an intelligent deep-learning-based recognition routine in form of hybrid cascaded recognition routine, and a necessary demonstration setup are demonstrated. Mineral aggregates are an essential raw material for the production of concrete. Petrographic analysis represents an elementary quality assurance measure for the production of high-quality concrete. Petrography is still a manual examination by specially trained experts, and the difficulty of the task lies in a large intra-class variability combined with low inter-class variability. In order to be able to increase the recognition performance, innovative new classification approaches have to be developed. As a solution, this paper presents an innovative cascaded deep-learning-based classification and uses a deep-learning-based data augmentation method to synthetically generate images to optimize the results.