As AI assistants promise hyper-personalization through access to private data, developers face a fundamental tension: How do we balance rich contextual memory against ironclad security? We examine the feasibility of end-to-end encryption and local processing for mainstream users, and the technical hurdles in making AI recall truly actionable without becoming a surveillance tool.