This happens when you stream a podcast with automatically inserted ads of varying lengths. I assume the system swaps out an ad before you finish the stream, so your playback position gets shifted... The solution on any app is to download the full episode before playing.

I wish we could block these ads! I've been thinking about how to build something that pulls multiple copies of the stream, compares the audio to identify the differences, then cuts out the bits that change - the ads. Someday...

Depending on your use case, you might prefer enabling the work profile for your main user rather than switching between 3 different phone users. Just download Insular in Fdroid and you can set it up.