This is still a bit flaky around when to remove a mention, but it should
work in most cases. Might add a toggle in the future to disable these
though.