Crown Vic shift/neutral problem

So since I have owned my 2008 crown Vic, I occasionally experience an issue, where I will be driving, and I normally I will let my foot off the gas because the car does a pretty good job maintaining speed. But sometimes I when I go to accelerate, it’ll act like it’s in neutral. So I then shift the car into second (after letting the rpms drop) drive some more, and then put it back into drive and it is fine. This does not happen at high speeds (never more than 40 mph).

One thing I found is that this one moderately sized hill in my town that curves right (if that matters) I will gun it up, let off the gas so I can slow down some, and then when I hit the gas it acts like it’s in neutral.

My neighbor told me it might be the shift solenoid. I planned on getting my trans fluid and filter changed anyway because I would like to take good care of it, but I was wondering if anybody else had experienced this issue. Thanks in advance.

Edit: when I removed the pan today, there was metal on a magnet in the pan. The metal covered the magnet, but there was not a lot of it, just enough to cover the magnet. Is that normal?