Software effort estimation is hard in the same way building anything is hard, but it's harder because:

  - the tools can vary wildly
  - there are few standard components
  - the few standard components have a wide amount of variance in their use and results
  - "there is no single development, in either technology or management technique, which by itself promises even one order of magnitude improvement within a decade in productivity, in reliability, in simplicity"
  - an in-house product is easier to write than a developer product others will use
  - developers have a tendency to use up all available resources (software is a gas) which then makes it harder when the software needs more resources to function (scalability)
  - most people who write software do not do design; thus the software is not designed before they start writing it, which (obviously) leads to an undeterminable amount of time until it is done
  - again because of a lack of design knowledge, developers fall into common traps like the second-system effect and other design flaws
  - the Agile Manifesto emphasizes a lack of planning, rigor, standardization, and documentation, all of which creates uncertainty
  - most projects don't include staff to properly track progress and reorganize efforts to get back on track
  - most people who write software were never properly trained (in methodologies, in software design, in system architecture, etc) and thus their work will be inefficient and unreliable, which adds time and expense
  - most of the tools needed to develop software quickly and with high quality are not available at the beginning of a project, and are built by either the same team building the software, or a different team, which is a bit like trying to build your own scaffolding and power tools while building a building. this adds delay and cost

Things that make estimating building anything hard:

  - the group of people doing the building vary in skill and experience
  - the ability of the group of people to work together well varies
  - the inavailability of materials when needed causes delays
  - coordination of work is difficult
  - humans make mistakes which creates delay and cost overrun, more when the mistakes aren't seen immediately

These things are all extremely well known and understood in our industry, since even 50 years ago. But because there is no standardization in the way our work is done, nor proper training nor roles to handle the difficult work of managing the work, it continues to be a crap-shoot whether you're working with somebody who knows what they're doing.