As the specialty of hyperspectral image (HSI), it consists of 2D spatial and 1D spectral information. In the field of deep learning, HSI classification is an appealing research topic. Many existing methods process the HSI in spatial or spectral domain separately, which cannot fully extract the representative features and the most used 3D convolutional neural network (3D-CNN) will suffer from mixing up complex spectral information. In this paper, we propose a spatial-spectral unified method by using recurrent neural networks (RNN) and multi-scanning direction strategy to construct spatial-spectral information sequences for learning the spatial dependencies among the central pixel and neighboring pixels. Meanwhile, residual connections and dense connections are introduced into multi-scanning direction sequences to overcome the memory problem in the RNN. The proposed method is tested on two benchmark datasets: the Pavia University dataset and the Pavia Center dataset. The experimental results demonstrate that the proposed method can achieve better classification rate than other state-of-the-art methods.