Soldering temperatures don't produce significant lead fumes. The fumes you see are flux fumes (which are also bad to inhale).
IMO, the most dangerous thing about lead solder is cleaning the iron. Both the common methods (damp sponge and brass wool) create many tiny little balls of solder that are hard to see and bounce about all over the place. Because of the high density of lead they're less affected by air resistance than you might expect, and they roll easily, so they can move surprising distances. They can easily end up caught in clothing, and from there fall into food. This will result in much higher lead ingestion than just touching solder then touching food.
I personally always use lead-free solder. If you have a good temperature controlled soldering iron it's nearly as easy to use as leaded solder.
Lead-free through-hole soldering is easy and practical. Unfortunately, for SMT prototyping (including reflow soldering), lead-free work is no longer that easy. Another problem, even in through-hole devices, is when you have large metal parts - a tough problem for RF/microwave circuits full of SMA and BNC connectors, backed by 1 or even 2 layers of solid ground planes, providing excellent a heatsink and a lot of cursing during work and rework. With lead-free solder, I found the iron needs to be cranked up to 420°C for a usable experience (but a larger iron tip may reduce that to a more reasonable level), and I don't know what temperature does it take to desolder them.
The last time I checked, low-temperature bismuth-tin alloy is only available as solder paste, unfortunately not available as flux-core solder wires (they're not really a good choice for connectors to begin with as the alloy is brittle, but I only need it to survive before the next prototype...)
IMO, the most dangerous thing about lead solder is cleaning the iron. Both the common methods (damp sponge and brass wool) create many tiny little balls of solder that are hard to see and bounce about all over the place. Because of the high density of lead they're less affected by air resistance than you might expect, and they roll easily, so they can move surprising distances. They can easily end up caught in clothing, and from there fall into food. This will result in much higher lead ingestion than just touching solder then touching food.
I personally always use lead-free solder. If you have a good temperature controlled soldering iron it's nearly as easy to use as leaded solder.