The most glaring of issues is that everyone wants to make a purely voice-controlled UI. Don't do that. Make something where voice inputs work with keyboard inputs. Start with something like vi, which has lots of different kinds of inputs (123dd <- dd is the command, 123 alters it to delete 123 lines instead of just one; d/;<RET> <- d is a partial command, /; finishes it to have it delete everything up the next ; ) and make voice an additional source of inputs.
key: /
voice: apple
result: you search for an apple
key: V
voice: block
key: d
result: you delete the language-appropriate block enclosing the cursor
instead of making a inferior voice-only UI, enhance an existing UI with voice