Still don't know what causes it, but even a fairly smooth road causes it, so it's not bumps like I thought. It's mostly 1-30 mph where it will happen. I'll have to have my girlfriend drive around in a parking lot to help figure it out (with me in the back).
They installed a bunch of thin plywood panels on the sides with sheet metal screws. I'm thinking maybe those screws rub against the van's metal sides when the body is flexed or vibrates at low speed and unloaded. It could be anything though.