Several large volatility matrix inference procedures have been developed, based on the latent factor model. They often assumed that there are a few of common factors, which can account for volatility dynamics. However, several studies have demonstrated the presence of local factors. In particular, when analyzing the global stock market, we often observe that nation-specific factors explain their own country’s volatility dynamics. To account for this, we propose the Double Principal Orthogonal complEment Thresholding (Double-POET) method, based on multi-level factor models, and also establish its asymptotic properties. Furthermore, we demonstrate the drawback of using the regular principal orthogonal component thresholding (POET) when the local factor structure exists. We also describe the blessing of dimensionality using Double-POET for local covariance matrix estimation. Finally, we investigate the performance of the Double-POET estimator in an out-of-sample portfolio allocation study using international stocks from 20 financial markets.