There are two basic kinds of listeners, an active and a passive. You will not get good offline (passive listener) results for a wide variety of speech currently. Most offline listeners are used as ‘passive’ listeners. They usually just listen for the wake word or just a few small commands. They are very accurate but can’t sustain a large vocabulary without CPU issues. Some are also closed source such as Kitt.ai (snowboy), and require that you train each word on their site.
The other kind, aka the “Active listener”; or the one always listening. Would be used to listen once activated by the passive listener, and has a much more broad vocabulary. However, these are online only. Due to the way they work (gathering metadata to increase accuracy), its just not possible currently on a small device. Examples would be standard APIs such as Google, Wit.ai, Ivona/Polly, etc.
There are projects to fix these offline issues, but they are all in the works. See my site for info on some of them. mattcurry.com