I recently submitted an entry for the Inverse Scaling Prize, and while it wasn’t selected, I think it still reveals some interesting properties of model scaling that are worth exploring (and are similar to those analyzed in one winning submission from Cavendish Labs). The goal of the competition is to identify machine learning tasks where …
During a recent email exchange, a reader of the blog brought up the question of technology / AI timelines and what progress in that domain might look like. His question struck a chord, reminding me of the uneasy feeling I often get when reading the optimistic predictions of others (including futurist Ray Kurzweil and philosopher …
About a year and a half ago, OpenAI rolled out GPT-3, a massive text-prediction transformer model which shattered many assumptions about the difficulty of understanding and creating written language. GPT-3’s predecessors (GPT and GPT-2) had shown that generating sensible responses to a variety of input texts was possible with enough data, but GPT-3 took that …