Filling crumbling?

I was chewing chips on the side where I have a filling on my lower molar and felt a tiny ping of pain, and suddenly within the chewed food it felt like there was sand in it (best way I can describe it). I can only imagine it was part of the filling that came out. I brushed my teeth and can't really see or feel much difference- I might be convincing myself that the surface is a little rougher but maybe it's placebo. No pain or sensitivity.

A few questions: Should i make an appointment with my dentist for ASAP? Can fillings just crumble that easily? Would they be able to see if the filling is compromised without drilling out and replacing it? I'm terrified of the filling falling all the way out- I have really small teeth and I think it's a sizable filling.

Thanks!