LDA or Clustering for Research Exploring?

I am building a research area exploring a tool which I collect a list of research papers (>1000), and try to identify the different topics/groups and trends based on their title and abstract. Currently I have built an LDA framework to perform this, but it requires quite a lot of trial and error and fine-tuning to get a sensible result. How I identify the research areas is that I build a TF-IDF, and a word cloud to see what are the possible area names. Now I am exploring using an embedding model like 'sentence-transformers/all-MiniLM-L6-v2' and a clustering algorithm to do this. I have tried using HDBScan, the result was very bad. Now it wonders me, is LDA inherently just better for this task? Please share your insights, it would be extremely helpful, thanks a lot.