AlexNet, the AI model that started it all, released in source code form - for all to download
This breakthrough in artificial intelligence has the potential to revolutionize various industries, from healthcare to finance. Researchers are excited about the possibilities for solving complex problems and creating more advanced technologies. The neural network's success highlights the importance of continued research and development in AI. Its impact could lead to significant advancements in areas like autonomous vehicles and natural language processing.
University of Toronto Professor Geoffrey Hinton and AlexNet
University of Toronto professor Geoffrey Hinton, center, and graduate students Ilya Sutskever, left, and Alex Krizhevsky, right, in 2013. There are many stories of how artificial intelligence came to take over the world, but one of the most important developments is the emergence in 2012 of AlexNet, a neural network that, for the first time, demonstrated a huge jump in a computer's ability to recognize images.
Thursday, the Computer History Museum (CHM), in collaboration with Google, released for the first time the AlexNet source code written by University of Toronto graduate student Alex Krizhevsky, placing it on GitHub for all to peruse and download. "CHM is proud to present the source code to the 2012 version of Alex Krizhevsky, Ilya Sutskever, and Geoffery Hinton's AlexNet, which transformed the field of artificial intelligence," write the Museum organizers in the readme file on GitHub.
The Impact of AlexNet
Krizhevsky's creation would lead to a flood of innovation in the ensuing years, and tons of capital, based on proof that with sufficient data and computing, neural networks could achieve breakthroughs previously viewed as mainly theoretical. The code, which weighs in at a scant 200KB in the source folder, combines Nvidia CUDA code, Python script, and a little bit of C++ to describe how to make a convolutional neural network parse and categorize image files.
The Museum's software historian, Hansen Hsu, spent five years negotiating with Google, which owns the rights to the source, to release the code, as he describes in his essay about the legacy of AI and how AlexNet came to be. Krizhevsky was a graduate student under Nobel Prize-winning AI scientist Geoffrey Hinton at the time. A second grad student, Ilya Sutskever, who later co-founded OpenAI, urged Krizhevsky to pursue the project. As Hsu quotes Hinton, "Ilya thought we should do it, Alex made it work, and I got the Nobel Prize."
Deep Learning Breakthroughs
Google owns the AlexNet intellectual property because it acquired Hinton, Krizhevsky, and Sutskever's startup company, DNNResearch. Until AlexNet, Hinton and others had toiled for years to prove that "deep learning" collections of artificial neurons could learn patterns in data. As Hsu notes, AI had become a backwater because it failed to demonstrate meaningful results.
Hinton and other true believers kept working, refining the design of neural networks, including CNNs, and figuring out in small experiments on Nvidia GPU chips how increasing the number of layers of artificial neurons could theoretically lead to better results. According to Hsu, Sutskever had the insight that the theoretical work could be scaled up to a much larger neural network given enough horsepower and training data.
Scaling Neural Networks
Sutskever knew they were wrong. "It wasn't just an intuition; it was, I would argue, an irrefutable argument, which went like this: If your neural network is deep and large, then it could be configured to solve a hard task." The trio found the training data they needed in ImageNet, which was a new creation by Stanford University professor Fei Fei Li at the time. Li had herself bucked conventional wisdom in enlisting Amazon Mechanical Turk workers to hand-label 14 million images of every kind of object, a data set much larger than any computer vision data set at the time.
When the work was presented at the ImageNet annual competition in September of 2012, AlexNet scored almost 11 points better than the closest competitor, a 15.3% error rate. They described the work in a formal paper. Yann LeCun, chief AI scientist at Meta Platforms, who had earlier studied under Hinton and had pioneered CNN engineering in the 1990s, proclaimed AlexNet at the time to be a turning point.
Legacy of AlexNet
What the trio had done was to make good on all the theoretical work on making "deep" neural networks out of many more layers of neurons, to prove that they could really learn patterns. "AlexNet was just the beginning," writes Hsu. "In the next decade, neural networks would advance to synthesize believable human voices, beat champion Go players, model human language, and generate artwork, culminating with the release of ChatGPT in 2022 by OpenAI, a company co-founded by Sutskever."