That technique didn’t work when I asked it to create a Python script to return all IAM roles with a certain set of policies attached.

It still missed using a paginator to handle the list_roles call returning more than 50 roles.

Once I pointed it out, it did add pagination support.

I try not to cheat and hint at specific issues (since that relies on prior knowledge)

I'd be surprised if even after the last prompt it wouldn't notice that. Saying "Did we miss anything" leaves it open it to re-evaluate both the implementation and the original considerations

Edit: There's some non-determinism involved, but GPT-4 caught the pagination from planning stage here: https://chat.openai.com/share/3c356d4f-15d4-4f6e-bd29-af6a0b...

Funny enough, even in my original question, it put pagination as a consideration. But it still didn’t include it.

I do short quote-posts on my blog and wrote this one up.

Yes! Excellent approach, capturing.