There are several ethnicities (Ashkenazim, Sephardim, etc.) which are historically Jewish, just as there are several ethnicities (Irish, Hispanic, etc.) which are historically Catholic. When someone says they are ethnically Jewish it generally means they belong to one of those ethnicities, and grew up with the associated foods, festivals and other cultural influences.
Sure, but Spanish and Irish atheists don't refer to themselves as atheistic catholics, whereas I have heard jewish friends of mine refer to themselves as atheistic jews.
