You still need to convert between words and sentence vectors somehow. You could try using a faster model for that, but I suspect that the output quality will suffer.
LLMs somehow need to do this anyway, implicitly or not. Has anyone tried to do it more explicitly?
That is to break out the idea that characters are formed into words, and words into sentences, and a sentence is a sequence of "concepts" for the lack of a better description.
So have one NN which takes a sequence of tokens and predicts an moderately-dimensional "word vector", which is fed into another which predicts a high-dimensional "concept vector".
Then the "thinking layer" would map a sequence of "concept vectors" to "concept vectors", and then you'd have some layers which does the reverse of the input layers to output tokens which can be printed.
Thought being that by splitting it up like this you could swap out the decode and encode layers independently to translate, for example, and so on.
