Since I started lifting weights, I’ve found curious the lack of any heuristics to calculate a one rep max based on a particular number of sets and reps. A number of calculations exist for a single set (see below chart from Wikipedia), but as pretty much all programming involves both sets and reps (e.g., bench …
After spending more time with GPT-4, I have to admit I’m surprised at the level of “understanding” possible via simple next token prediction (given massive scale). On a wide variety of tasks the answers it provides are almost uncannily useful, and in domains like test-taking I did not think scale alone would drive the high …
Large language models have been all over the news the last couple months, with the launch of chatGPT (and the subsequent investment in OpenAI by Microsoft) kicking off an arms race of sorts within big tech. The capabilities of these latest models are impressive, and they do seem to represent a step change vs. prior …
With 2022 coming to a close, it seems a good time to reflect on the advances made in AI over this last year. The most high profile ones have come in the domains of image generation (DALL-E 2, Lensa, Imagen) and dialogue (ChatGPT), with the impacts starting to have broad reach. AI-generated pictures are now …
I recently submitted an entry for the Inverse Scaling Prize, and while it wasn’t selected, I think it still reveals some interesting properties of model scaling that are worth exploring (and are similar to those analyzed in one winning submission from Cavendish Labs). The goal of the competition is to identify machine learning tasks where …
Although GPT-3 was released ages ago (in AI time), it continues to generate interesting conversations, particularly with regard to the path toward general artificial intelligence. Building off a discussion of some others in the field (centered around the potential upside of scaling deep learning models), Scott Aaronson (a quantum computing expert who writes Shtetl-Optimized) and …
“Learning” is another one of those abstract concepts which reveals significant complexity upon further examination. In the context of people, learning represents our ability to incorporate experience in a beneficial way; we can learn facts, skills, or social norms (among countless other things) through repeated (or one-time) exposure. The exact mechanics underlying the learning process …
With the massive degree of progress in AI over the last decade or so, it’s natural to wonder about its future – particularly the timeline to achieving human (and superhuman) levels of general intelligence. Ajeya Cotra, a senior researcher at Open Philanthropy, recently (in 2020) put together a comprehensive report seeking to answer this question …
Examining Evolution as an Upper Bound for AGI Timelines Read More »
During a recent email exchange, a reader of the blog brought up the question of technology / AI timelines and what progress in that domain might look like. His question struck a chord, reminding me of the uneasy feeling I often get when reading the optimistic predictions of others (including futurist Ray Kurzweil and philosopher …
About a year and a half ago, OpenAI rolled out GPT-3, a massive text-prediction transformer model which shattered many assumptions about the difficulty of understanding and creating written language. GPT-3’s predecessors (GPT and GPT-2) had shown that generating sensible responses to a variety of input texts was possible with enough data, but GPT-3 took that …
In the comments of https://mybrainsthoughts.com/?p=327, a discussion sprang up on goals and optimization that seems worth diving into further. That post covers some ideas on criminal justice and optimal sentencing approaches, and the idea that came up in the comments was using AI to optimize our laws, so as to take the human emotion out …
One important concept in data science is that of signal. Signal represents the ability of data to inform accurate predictions in the target domain; a dataset that can tell you more has more signal. For example, say you’re trying to predict the weather in New York for the next day (repeatedly). For the sake of …
As previously discussed in this post, deep learning models designed to predict and create natural language have become quite powerful, and are now able to do things like write news articles or compose poetry at a level similar to (or even better than) the average human. This is a significant achievement which will likely have …
Deep Learning Language Models and Exact Procedures Read More »
Intelligence is notoriously difficult to define, in part because of the variety of ways in which we see it manifested in the world. From humans, to mice, to ant colonies, to bacteria, to computational algorithms, we see intelligence all around us, but it’s difficult to put a finger on a single definition which covers all …
The desire for control seems to be deeply rooted in the human psyche. We all seek control over our own lives, and oftentimes reach further and attempt to control the lives of others (generally, of those close to us, or those with a degree of control over us). Therefore, it comes as no surprise that …
The field of machine vision has progressed rapidly over the last decade, with many systems now achieving “better than human” results on standardized image recognition tests. Deep convolutional neural networks have been a main driver of these improvements, and have been enabled by increasing data availability and computing power. ImageNet Competition Best Error Rate Performance, …
A couple hundred thousand years ago, our species reached a point where we could pass knowledge directly (through language and demonstrated action), rather than relying on adaptations to be passed on through genes. With this shift, culture was born – and it allowed us to very rapidly acquire a great deal of knowledge about our …
In his book I am a Strange Loop, Douglas Hofstadter paints a powerful analogy involving the careenium, symms, and symmballs. I highly recommend reading the book (as well as all his other works, if you have the time), but I’ll summarize the analogy briefly to start, as this post involves extending it to help drive …
2040: “Hm, that’s odd,” thought Dr. Newerton. He peered back over his notes, trying again to make sense of the study he had just reviewed on cortex function. Two years ago, his group had thought they were on the cusp of understanding the brain, but now that goal was feeling more and more elusive. The …
In the artificial intelligence literature, it’s common to see researchers examining the question of “what is the right final goal to give an AGI (artificial general intelligence)”. Here, “final goal” means the ultimate task we want the system to achieve – for example, playing Go well (Alpha Go) or classifying images well (VGG19). Researchers frequently …
GPT-X, Paperclip Maximizer? An Analysis of AGI and Final Goals Read More »
Imagine we have a man in a room, and we can ask him to perform any task (we’ll assume he’s motivated to do the tasks). These tasks range across the entire domain of the world, for example: Shoot a bow and arrow at a target (given a bow and arrow and a target) Point to …
The Kernel of Narrow vs. General Intelligence: A Short Thought Experiment Read More »
From its inception, the field of artificial intelligence has separated program specification into two key concepts: “goals” and “intelligence”. The “goal” is the problem a program or agent must solve; “intelligence” is the means by which it solves it. For example, the goal might be to play chess, and the intelligence might be a neural …
A new natural language AI model launched by OpenAI, GPT-3, has been making waves in the artificial intelligence community. GPT-3 is a transformer model (simplifying greatly, a neural network approach with modifications for better performance on text) trained on one specific task: predicting the next word, given all previous words within some text. This simple …
As researchers and philosophers discuss the path towards human-equivalent / superhuman general artificial intelligence, they frequently examine the concept of control. Control of our world has always been of central importance to our species, so it’s no surprise that we’d look to extend our control in the future. However, most of the currently discussed control …
We’ve now touched on the idea of general artificial intelligence (i.e. above human level) in a couple posts, and so it seemed time for a more thorough examination of how we might get there, and of what this type of “superintelligence” might look like. We’ll start out by taking a step back, and looking at …
A number of other posts so far have touched on what it is that brains do – and for the most part, it’s been summarized as “creating a model of the world”. By this, we’ve meant that certain patterns of neural activity can be understood as representing or standing for some observed pattern of material …
Dedicated to Sam Rendall – you and your ideas are greatly missed. What is intelligence? What does it mean to be intelligent? A starting point for a very high level definition might be an ability of an agent to accomplish its goals in its environment. From this definition, we can see three pieces to assessing …
“Open your eyes, Harry.” The noise wakes me up, and I open my eyes. The light from the room floods my eyes, and then I begin to make sense of my bearings, remembering with a jolt where I am. Has the experiment worked? I see Dr. Chet looking at me intently, jotting down notes, with …