What Bohiruci said is valid. You can't deny their presence
simply because you can't see them.
For some, it's difficult to "invite" them out, because of their
strong attachment, could be their family, house, some unfinished
matters or even hatred towards someone, something. The magnititude
to how reluctant they are to leave, is entirely on how deep their
attachments are.
Example. You were staying in your own house, with loads of
memories. Then one day, someone came to you and tell you to leave
because he has bought the house without your knowledge, how would
you feel?
I do know cases whereby the spirits were murdered, and they
lingered in the house because they habour enormous hatred towards
the one who killed them. The later occupants encountered loads of
unexplained incidents, and a few could easy take their lives.
You may view it as a chat of superstition, but you gotta appreciate
that Buddha don't talk about the 6 realm sentient beings for no
reason. These spirits were just part of them.