Exactly this. It's probably not an accident that one of those numbers is half of 10 (and the number of fingers on one hand) – a common made-up number for everybody – and the other is 2^3 – a common made-up number for programmers.
I've always done planning poker with the Fibonacci sequence - 1, 2, 3, 5, 8. The idea being that the more complicated the task, the harder it is to estimate accurately.
Why is story point estimation tied to fibonacci sequence? Two generations of managers at my previous employment thought this way. It just seems so arbitrary to me.
First, the more difficult a task is, the more inherent difficulty there will be in "accurately" estimating the difficulty of the task. Fibonacci is used to represent the inherent lack of accuracy in more difficult tasks, since the numbers get _very_ far from each other as they go up the scale.
Second, the numbers _are_ arbitrary. Completely, 100% arbitrary. It's a _relative_ difficulty scale. Say you've got 3 tasks - A, B, and C. A and B are approximately as hard as each other, they're 1 story point. C is more difficult than either one - it gets 2 points. That's it. Story points are not, and should not be used as, a unit of measurement. The biggest utility is to identify big, scary tasks with lots of unknown factors.
The fact that they are _numbers_ is what tricks so many teams/PMs/management/etc into thinking that story points are more meaningful than they were ever supposed to be. Incidentally, this is also why some planning poker teams use t-shirt sizing (S,M,L,XL,XXL, etc). No numbers means people are less tempted to punch them into a spreadsheet while deluding themselves into believing that showing numbers going down is the same thing as "showing progress".
The closer the numbers are together the more time is wasted trying to be exact about them. Fib helps reduce the amount of back and forth. If you need to guess how much something weighs and your options are 1,2,3,4,5,6,7,8,9,10,11,12,13,14.....100, you are going to have a lot harder time achieving consensus than if you asks "Is it heavier or lighter than a breadbox?"
