Since I started lifting weights, I’ve found curious the lack of any heuristics to calculate a one rep max based on a particular number of sets and reps. A number of calculations exist for a single set (see below chart from Wikipedia), but as pretty much all programming involves both sets and reps (e.g., bench …
I recently submitted an entry for the Inverse Scaling Prize, and while it wasn’t selected, I think it still reveals some interesting properties of model scaling that are worth exploring (and are similar to those analyzed in one winning submission from Cavendish Labs). The goal of the competition is to identify machine learning tasks where …
Although GPT-3 was released ages ago (in AI time), it continues to generate interesting conversations, particularly with regard to the path toward general artificial intelligence. Building off a discussion of some others in the field (centered around the potential upside of scaling deep learning models), Scott Aaronson (a quantum computing expert who writes Shtetl-Optimized) and …
“Learning” is another one of those abstract concepts which reveals significant complexity upon further examination. In the context of people, learning represents our ability to incorporate experience in a beneficial way; we can learn facts, skills, or social norms (among countless other things) through repeated (or one-time) exposure. The exact mechanics underlying the learning process …
With the massive degree of progress in AI over the last decade or so, it’s natural to wonder about its future – particularly the timeline to achieving human (and superhuman) levels of general intelligence. Ajeya Cotra, a senior researcher at Open Philanthropy, recently (in 2020) put together a comprehensive report seeking to answer this question …
Examining Evolution as an Upper Bound for AGI Timelines Read More »
During a recent email exchange, a reader of the blog brought up the question of technology / AI timelines and what progress in that domain might look like. His question struck a chord, reminding me of the uneasy feeling I often get when reading the optimistic predictions of others (including futurist Ray Kurzweil and philosopher …
About a year and a half ago, OpenAI rolled out GPT-3, a massive text-prediction transformer model which shattered many assumptions about the difficulty of understanding and creating written language. GPT-3’s predecessors (GPT and GPT-2) had shown that generating sensible responses to a variety of input texts was possible with enough data, but GPT-3 took that …
One important concept in data science is that of signal. Signal represents the ability of data to inform accurate predictions in the target domain; a dataset that can tell you more has more signal. For example, say you’re trying to predict the weather in New York for the next day (repeatedly). For the sake of …
In General Intelligence and Context Switching, we examined how the brain brings to bear different operational strategies depending on the requirements of a given situation, and discussed how that flexibility gives rise to general intelligence (vs. the narrow intelligence exhibited by computers today). The argument focused mainly on more academic activities such as physics and …
As previously discussed in this post, deep learning models designed to predict and create natural language have become quite powerful, and are now able to do things like write news articles or compose poetry at a level similar to (or even better than) the average human. This is a significant achievement which will likely have …
Deep Learning Language Models and Exact Procedures Read More »
Intelligence is notoriously difficult to define, in part because of the variety of ways in which we see it manifested in the world. From humans, to mice, to ant colonies, to bacteria, to computational algorithms, we see intelligence all around us, but it’s difficult to put a finger on a single definition which covers all …
General intelligence is a remarkable adaptation. We humans are able to learn a near infinite variety of things, from language to physics to etiquette to chess. Our brains never “fill up”; in fact, increased capability in one domain often enhances our ability to learn others. It doesn’t take much for us to learn, with facts …
The desire for control seems to be deeply rooted in the human psyche. We all seek control over our own lives, and oftentimes reach further and attempt to control the lives of others (generally, of those close to us, or those with a degree of control over us). Therefore, it comes as no surprise that …
Author’s Note: For any French readers, this post has recently been translated by Azurisme! Metals are a unique form of matter, especially with regards to their behavior under heating and cooling. While metals are described by the material they’re made of (e.g. copper or iron), their properties are determined by their arrangement of atoms, and …
A couple hundred thousand years ago, our species reached a point where we could pass knowledge directly (through language and demonstrated action), rather than relying on adaptations to be passed on through genes. With this shift, culture was born – and it allowed us to very rapidly acquire a great deal of knowledge about our …
In his book I am a Strange Loop, Douglas Hofstadter paints a powerful analogy involving the careenium, symms, and symmballs. I highly recommend reading the book (as well as all his other works, if you have the time), but I’ll summarize the analogy briefly to start, as this post involves extending it to help drive …
2040: “Hm, that’s odd,” thought Dr. Newerton. He peered back over his notes, trying again to make sense of the study he had just reviewed on cortex function. Two years ago, his group had thought they were on the cusp of understanding the brain, but now that goal was feeling more and more elusive. The …
In the artificial intelligence literature, it’s common to see researchers examining the question of “what is the right final goal to give an AGI (artificial general intelligence)”. Here, “final goal” means the ultimate task we want the system to achieve – for example, playing Go well (Alpha Go) or classifying images well (VGG19). Researchers frequently …
GPT-X, Paperclip Maximizer? An Analysis of AGI and Final Goals Read More »
Imagine we have a man in a room, and we can ask him to perform any task (we’ll assume he’s motivated to do the tasks). These tasks range across the entire domain of the world, for example: Shoot a bow and arrow at a target (given a bow and arrow and a target) Point to …
The Kernel of Narrow vs. General Intelligence: A Short Thought Experiment Read More »
Humans (and all other brain-having organisms) interact with the world in a hierarchical manner, starting from the top and working down. When we decide to engage in an action, we make the decision at the highest level (for example, deciding to write a blog post), and then break up that action into its lower level …
From its inception, the field of artificial intelligence has separated program specification into two key concepts: “goals” and “intelligence”. The “goal” is the problem a program or agent must solve; “intelligence” is the means by which it solves it. For example, the goal might be to play chess, and the intelligence might be a neural …
A new natural language AI model launched by OpenAI, GPT-3, has been making waves in the artificial intelligence community. GPT-3 is a transformer model (simplifying greatly, a neural network approach with modifications for better performance on text) trained on one specific task: predicting the next word, given all previous words within some text. This simple …
As researchers and philosophers discuss the path towards human-equivalent / superhuman general artificial intelligence, they frequently examine the concept of control. Control of our world has always been of central importance to our species, so it’s no surprise that we’d look to extend our control in the future. However, most of the currently discussed control …
We’ve now touched on the idea of general artificial intelligence (i.e. above human level) in a couple posts, and so it seemed time for a more thorough examination of how we might get there, and of what this type of “superintelligence” might look like. We’ll start out by taking a step back, and looking at …
A number of other posts so far have touched on what it is that brains do – and for the most part, it’s been summarized as “creating a model of the world”. By this, we’ve meant that certain patterns of neural activity can be understood as representing or standing for some observed pattern of material …
Dedicated to Sam Rendall – you and your ideas are greatly missed. What is intelligence? What does it mean to be intelligent? A starting point for a very high level definition might be an ability of an agent to accomplish its goals in its environment. From this definition, we can see three pieces to assessing …