Depends on the task at hand.. Some tasks like information collection for planning have definitely improved and made easy, but the quality is still not at a level where you can use it without doing an overview.

Overall, it's just the illusion of more productivity or free time. It's just made grunt work easier while making testing/review even more important than before

While I agree with the sentiment of not just letting it run free on the whole codebase and do what it wants, I still have good experience with letting it do small tasks one at a time, guided by me. Coding ability of models has really improved over the last few months itself and I seem to be clearing less and less AI-generated code mess than I was 5 months ago.

It's got a lot to do with problem framing and prompt imo.

With the stochastic nature of LLM’s and the ever moving goal-posts, a framework not based off of knowledge that might already be in its memory is definitely needed