A friend of mine is pretty much the sole technical talent at a start-up. Last month, we started chatting about his use of ChatGPT and other generative AI solutions. He is doing a lot of firmware coding in Rust now, and he has been telling me having the generative AI is like having a bunch of interns or volunteers on hand so far as routine coding tasks go. So much so that he has sprung for an RTX 3090 GPU in order to self-host models and get past arbitrary restrictions on usage.
I only myself got started using ChatGPT via OpenAI’s API this weekend. I’ve asked for, and gotten, several Python code examples, including what purports to be an inverse CDF for the logit function (I still need to test it), and some Python to database functions (which I have tested). I’ve requested some text concerning topics with references, which it does OK. And that got me thinking about long-planned projects and how this tech could assist in those.
One of the tasks I’ve been thinking about a long time is a refresh of Marty Leipzig’s keyword-indexed bibliography hosted on the TalkOrigins Archive. That was put up in the mid-1990s and I think may have had one update from Marty over twenty years ago. Obviously, time marches on. While I’m at it, the ‘Biographica’ material at the TOA could use a major update, too.
So I wrote a program that I can feed a topic name from the bibliography, it builds a prompt, then calls the OpenAI API to get the completion. There’s two prompts built per topic currently, one to write a summary paragraph, and another to find references for the topic. There’s currently some waste/redundancy because the API apparently limits the length of the completion sent back, and I have to send back context and another prompt to resume from where it truncated. There are other things I had to build in, like delays to avoid ‘rate-limit’ error messages. The topic summaries and bibliography entries get saved to a SQLite database.
I have run this to get a sample of topics covered (35 of them), and then checked my OpenAI account, where I’ve used $13 from an $18 credit. Taking off a buck for the incidental exploration, my estimate of cost to complete the exercise for the TOA materials is:
Topic count: 1049, likely cost to complete $347.66
Biography count: 318 likely cost to complete $109.03
The bibliography topic estimate is low, because some more needs to be done to fill out the number of bibliography entries, and that will take both exploration and more prompts with context to get the completions I am after.
If I go forward on this, I’m seriously considering following my friend’s lead on getting a good graphics card capable of self-hosting one or more generative AI models. The cost I’d sink into interactive use is already about a third of my estimate of what it would take to augment my current best desktop, and about a fifth of what I’d estimate a purpose-built new system would take.