We develop a novel multiple instance SVM based on maximizing the minimum witness rate (WR) among positive bags. We solve this nonlinear integer programming problem by the multiple instance SVM formulation with a convex loss function, where unknown integer labels of instances in positive bags are relaxed to real numbers between -1 and 1 using the tanh(.) to estimate WR of a positive bag. Our new model, WR-SVM, also can incorporates the multiple instance learning (MIL) problem into a simple deep neural network framework with no additional MIL pooling layers. WR-SVM is robust to input perturbation by eliminating the imbalance between positive instances allocated to positive bags. A better generalization ability is expected by the large margin due to the balanced allocation of positive instances to positive bags. We perform experiments on various video datasets to verify the effectiveness of our method for video classification. The results of WR-SVM outperform the state-of-the-art for MIL-based video classification models.