HDPE is also what Nalgenes and the like are made of (or used to be anyway), and is very popular for storing chemicals as well. It is very nearly completely inert.
you cunninghammed me: nalgenes are polycarbonate, which is a lot less inert
all the polyethylenes are relatively inert, because polyethylenes are in some sense just heavy paraffins. paraffin is germanized latin for 'relatively inert'