Since LLMs came out, I’ve mostly been using ChatGPT to write AWS SDK based Python scripts and infrastructure as code.
If it wasn’t trained on a specific newer API or Cloudformation/Terraform/CDK construct, with 4/4o, I just give it the link to the relevant documentation and tell it to use the links to help it create the right code.
I tried giving it a link once to read through a PDF full of cars manufactured back in the 80s to find the cheapest ones. I gave me answers, but I was able to manually find some cheaper ones in the list. So at the end of the day, I couldn’t trust it any more than my own eyes. I think what I was asking was far more basic than writing code. Order the list by price (lowest to highest), and give me the top 5 results.
The PDF had multiple columns, and while ChatGPT seemed to figure that part out, it couldn’t do the logic part. Had it been in an easier Format to deal with, I would have just used to spreadsheet.
ChatGPT struggles mightily with the simple task of ordering the presidents by the year they were born. It got the order wrong, the years wrong and there were duplicates.
I had to explicitly tell it to verify its sources on the web and use Python
Yes, it only works within the context window - ie your session, it isn’t part of its permanent training data.
I purposefully chose something obscure that I knew was a new feature that wouldn’t be in the training data. Even then I had to force it to search the web
If it wasn’t trained on a specific newer API or Cloudformation/Terraform/CDK construct, with 4/4o, I just give it the link to the relevant documentation and tell it to use the links to help it create the right code.