Mark Carroll's Journal

nVidia announced the "Digits" a couple of weeks ago. A small form factor PC offering which they claim can run 1 petaflop at FP4 precision; yours for $3000.

I've dabbled with training small numerical models on the desktop and running them on embedded hardware; they were surprisingly good but they were little more than the "hello world" of ML. I'm not entirely sure what level of power is needed to efficiently train an LLM or image/video model but in my book, my powers can only be used for good.

I read the other day that the "major" LLMs - ChatGPT and similar - have already been trained on the entire available corpus of humanity. Everything they could get their virtual hands on, and I'm sure there would have been shady deals to get access to text that was not publicly online. I also fully expect that Github Copilot has been trained on everything that was in Github (including private repos? probably, who knows for sure) as well as everything they could scrape from Bitbucket, Gitlab et al.