Weather prediction at point locations is extremely challenging to get right because we simply can't observe and make predictions for every single square inch of the earth. Many weather models are run on grids with boxes about the size of a few kilometers at the smallest scale, which means that any physical process in the atmosphere that is the size of that box or smaller won't be represented well by the model.
Specifically on your point about clouds passing over your location, cloud and precipitation formation is even more challenging. Clouds and precipitation form due to atmospheric processes ranging from hundreds of kilometers all the way down to micrometers, which practically means the weather models are making an educated guess (albeit a very good one that is informed by scientific research) about when and where clouds will form. And when a model does predict a cloud, it will cover an entire grid box.
Finally, I saw you made a comment about how machine learning should improve forecasts, and in fact it does! But the weather community is still working on data driven models (as opposed to models that solve physical atmospheric equations), and most of them are run by private companies so their output is not free. As these data driven models get better, it may be possible that they will be able to make predictions at scales less than a kilometer.
