If that span is 20ms, you’re tempo mapping using the wrong references. That looks like a sloppy bass note attack (again, based on waveform and timing you provided), you should use a dry sound, like a snare or a kick (kicks are better because they tend to be on the 1 of the measure but on older recordings they aren’t dry enough). Define small drift: on a normal tempo, not too fast and not too slow, with a low notes density, you can probably get away with as much as 30 ms of delay on a measure or two. If you’re authoring complex prog rock and similar or very slow songs and/or songs with high notes density you may even need to go as precise as beat by beat in some sections. Never, and I mean ever, judge your calibration using customs: make sure your setup works perfectly with official releases, then test customs.

Try not to over think it.

Make sure you’re always using the metronome when you author the tempo map.

If the timing on the drums are a bit sloppy, but the rest of the instrumentation is in time with it, then by all means author beat map to be perfectly in sync with the drums.

If the drums are a bit sloppy, and it’s a little bit out of time with the rest of the instrumentation, then author the beat with the rest of the instrumentation.

(The way I look at it is that for sloppy songs, you want the tempo map to be the click track that the performers were trying to play to. Not sure if I can explain this very well)

You can then author the drum notes that seem a bit too far out of time off of the grid lines, and the rest of the instruments won’t be effected.

As far as where on the envelope (the shape of the notes on the waveform) should you author the beat to, you want the click from the metronome to ‘disappear’ into the sound of the beat (unless the drums are out of time etc). The waveform is more of a guide for the tempo map rather than being exact. The most important thing is how it sounds.