I don’t know any specifics about trouble, but would it be good enough to just track the current mode in your own global option/variable whenever you toggle it with your shortcut?

If not I‘d start by taking a look at the trouble source code.