Document-level Event Factuality Identification (DEFI) predicts the event factuality according to the current document, and mainly depends on event-related tokens and sentences. However, previous studies relied on annotated information and did not filter irrelevant and noisy texts. Therefore, this paper proposes a novel end-to-end model, i.e., Reinforced Multi-Granularity Hierarchical Attention Network (RMHAN), which can learn information at different levels of granularity from tokens and sentences hierarchically. Moreover, with hierarchical reinforcement learning, RMHAN first selects relevant and meaningful tokens, and then selects useful sentences for document-level encoding. Experimental results on DLEF-v2 corpus show that RMHAN model outperforms several state-of-the-art baselines and achieves the best performance.