Having spent so many years in Japan I have found the same attitude about jaywalking. Though, there are crosswalks on long streets where pedestrians can wait and drivers are taught to stop at if someone is waiting to cross. I haven't seen much reason to jaywalk here in Japan.
It really depends on where you are in Japan (and each individual crosswalk).
I've encountered plenty of places where jaywalking is necessary to save a long detour of a walk. Though those places are pretty clear of traffic. High traffic places where one might want to jaywalk? Pedestrian bridges are built there. So nice.
I similarly find that at most crosswalk, most people stop for you. Also nice.
The only not nice thing? At completely empty intersection on tiny little backroads, pedestrians STILL wait for the light to turn green. Thus making me look like some rebel if I boldly walk or bicycle across the absolutely empty intersection through a red light.