Case-based problem-solving systems reason by retrieving relevant prior cases and adapting their solutions to fit new circumstances. The ability of case-based reasoning (CBR) to reason from ungeneralized episodes can benefit knowledge acquisition, but acquiring the needed case adaptation knowledge has proven challenging. This paper presents a method for alleviating this problem with just-in-time gathering of case adaptation knowledge, based on introspective reasoning and mining of Web knowledge sources. The approach combines knowledge planning with introspective reasoning to guide recovery from case adaptation failures and reinforcement learning to guide selection of knowledge sources. The failure recovery and knowledge source selection methods have been tested in three highly different domains with encouraging results. The paper closes with a discussion of limitations and future steps.