Yes, you need to consider authorization at every layer. You can blanket deny a lot of things in a midlayer, but sooner or later you need to start interpreting business logic to do the rest.