The major problem with this approach is that you go into the analysis with an opinion as to what is the better solution, then you tweak the weights and scores to match that opinion. This is not data-driven or even rational, it's just a way to express "numerically" what your guts tell you.
Clearly someone who disagrees with you will just tell you you got the weights and scores wrong.
The main value of this exercise is to sit down and think about a problem in a slightly deeper way and try to rationalize why we think solution A is better than solution B. That is good in itself. But people shouldn't mistake a mental exercise for a quantitative analysis.
> you go into the analysis with an opinion as to what is the better solution, then you tweak the weights and scores to match that opinion. This is not data-driven or even rational, it's just a way to express "numerically" what your guts tell you
Clearly someone who disagrees with you will just tell you you got the weights and scores wrong.
The main value of this exercise is to sit down and think about a problem in a slightly deeper way and try to rationalize why we think solution A is better than solution B. That is good in itself. But people shouldn't mistake a mental exercise for a quantitative analysis.