Looks like you'll be able to turn the feature off:

> You can turn off memory at any time (Settings > Personalization > Memory). While memory is off, you won't create or use memories.