Hi. Good question. I am the admin of this forum. Welcome to our community. Here's an answer to your question that I once learned from a very wise man. It's a little pithy, but read it twice or three times and I hope you'll get it
G-dliness and goodness always shine. Hashem is all good -
all the time - and He wants to give us all the good in the world - always. The problem is twofold. (1) we don't always have the vessel for it. You can't fill a cup that's already full, can you?. (2) More times than not, we don't know what is really good for us.
So here's the deal. When it comes to spiritual matters, Hashem
always answers every true and sincere prayer for spiritual growth (i.e. related to getting "out of self" and closer to G-d/G-dliness). "True prayer" though means we really really feel we need it. At that point, we have a true vessel for it - and the vessel is empty. Once that happens, G-d's light just shines in, as it was always shining in the first place. There is no need for G-d to even answer the prayer. Once we clear the blockage, His light just flows in.
When it comes to physical things however, Hashem will only answer a prayer if he knows it is good for us. And what
we think is good for us is not necessarily good for us at all. So why pray at all for physical things? The point of praying for "stuff" is not to get it, but rather to feel close to Hashem and acknowledge that He gives us everything and that we are completely dependent on Him. Once we feel that way, often circumstances "change" and sometimes what was previously "bad" for us, is now "good for us". In other words, it could be that the whole reason we had the lack in the first place was only so that we get closer and more dependent on G-d. Once we pray and achieve that goal, there is no longer any reason for G-d to withhold it from us. But again - not always. Often, no matter how much we pray for something and think that it is good for us, it may not be at all. And if it isn't, Hashem won't give it to us. Thank G-d for that! :D
Hope this sheds some light...