Video summarization aims to create a succinct representation of videos for efficient browsing and retrieval. We propose an innovative method for the task. It includes two main steps: (i) the first step proposes a Distinct Frame Patch (DFP) index for selecting a set of good candidate frames, and (ii) the second step proposes a novel Appearance based Linear Clustering (ALC) to refine them for distinct ones. While the first step measures the content of frames, the second step considers to what extent one frame is different from another in both the spatial and temporal spaces. The experiments are performed over two publicly accessible datasets. The results show the effectiveness and efficiency of the proposed method when compared with other state-of-the-art techniques.