You would only rebuild the cache if you deleted it, or if there was some kind of conflict that forced a rebuild. It’s generally only an issue if you are testing/building customs or using RB:HP.
I would recommend keeping the customs on the TU4 stick, as otherwise they’ll all show up in your playlist under TU5 (but fail to load).
You are about to leave Rhythm Gaming World and access the external songs DB. While no song or file sold for commercially available videogames is hosted there and only fan generated content is allowed, Rhythm Gaming World has no direct control over the database and no files are hosted here. Alright, we had to made you aware of this, now you can visit the database!