Exactly this. Copyright is restricting the rights of others (specifically speech rights). It is counter to the common-sense stance that whatever you literally put into the domain of the public is public domain.

We've decided that limiting others rights for a temporary period was an acceptable price in order to encourage the creation of new works.

All of this was explicitly and energetically debated when copyright laws were being considered at all in the first place.