Sounds like you're using LLMs correctly IMO. The entire appeal of LLMs in dev-tooling is being able to setup hyper-specific, customized workflows and have "extensions" be done by telling the LLM what to fix. For that, the smaller and more minimal the base is, the better.