No, it's not silly to worry about it. Many companies store data in third party systems to which they retain control over access. Once you put data into chatGPT what control do you have over it?