Early assessment of liver cancer patients with hepatocellular carcinoma (HCC) is of immense importance to provide the proper treatment plan. In this paper, we have developed a two-stage classification computer-aided diagnostic (CAD) system that has the ability to detect and grade the liver observations from multiphase contrast enhanced magnetic resonance imaging (CE-MRI). The proposed approach consists of three main steps. First, a pre-processing is applied to the CE-MRI scans to delineate the tumor lesions that will be used as an ROI across the four different phases of the CE-MRI, (namely, the pre-contrast, late-arterial, portal-venous, and delayed-contrast). Second, a group of three features are modeled to provide a quantitative discrimination between the tumor lesions; namely: i) the tumor appearance that is modeled using a set of texture features, (namely; the first-order histogram, second-order gray-level co-occurrence matrix, and second-order gray-level run-length matrix), to capture any discrimination that may appear in the lesion texture, ii) the spherical harmonics (SH) based shape features that have the ability to describe the shape complexity of the liver tumors, and iii) the functional features that are based on the calculation of the wash-in/wash-out through that evaluate the intensity changes across the post-contrast phases. Finally, the aforementioned individual features were then integrated together to obtain the combined features to be fed to a machine learning classifier towards getting the final diagnostic decision. The proposed CAD system has been tested using hepatic observations that was obtained from 85 participating patients, 34 patients with benign tumors, 34 patients with intermediate tumors and 34 with malignant tumors. Using a random forests based classifier with a leave-one-subject-out (LOSO) cross-validation, the developed CAD system achieved an 87.1% accuracy in distinguishing the malignant, intermediate and benign tumors. The classification performance is then evaluated using k-fold (5/10-fold) cross-validation approach to examine the robustness of the system. The LR-1 lesions were classified from LR-2 benign lesions with 91.2% accuracy, while 85.3% accuracy was achieved differentiating between LR-4 and LR-5 malignant tumors. The obtained results hold a promise of the proposed framework to be reliably used as a noninvasive diagnostic tool for the early detection and grading of liver cancer tumors.