In 2020, there were several major breakthroughs in Ai investment research, especially in the fields of natural language processing (NLP) and computational biology. The release of OpenAI’s GPT-3 model and DeepMind’s AlphaFold protein folding algorithm are considered milestones in leveraging Ai technology to push scientific boundaries. These innovations also present new opportunities for Ai investment and commercialization.
On code-sharing and collaboration platform Github, more and more machine learning infrastructure, applications and documentation resources are made free to access, facilitating open research partnerships. Technology giants like Google and Facebook open-sourced many cutting-edge models across computer vision, NLP and reinforcement learning, allowing the broader community to build on top of them.
Apart from research, Ai also played a significant role in the global response to the Covid-19 pandemic, through powering contract tracing apps, predicting infection hotspots, accelerating vaccine and drug discovery and so on. Governments partnered with private Ai firms to launch open datasets and modeling challenges to engage scientists worldwide.
Looking ahead to 2021, Python is expected to become even more popular as the leading programming language for machine learning. Its major rival framework TensorFlow may be surpassed by the fast rising PyTorch. With changing consumer behaviors and explosion of new data sources, the application and business value of data science will continue to grow substantially.
GPT-3 and AlphaFold represent landmark achievements in NLP and biology
In February, Microsoft unveiled MT-NLP model with 17 billion parameters, only to be dwarfed by OpenAI’s GPT-3 months later. As the largest language model to date, GPT-3 has 175 billion parameters and was trained on Common Crawl dataset 116 times bigger than what was used for its predecessor GPT-2.
While not yet as performant as humans, GPT-3 shows strong language comprehension, reasoning and generation abilities across many domains. Its’versatile capabilities allow OpenAI to offer it as an API product to customers. GPT-3’s innovation also lies in its model architecture and pre-training approach. It was selected as one of the best papers at top Ai conference NeurIPS 2020.
In biology, Google DeepMind’s AlphaFold algorithm managed to predict protein folding shapes to a level comparable with experimental results, solving a 50-year grand challenge. This breakthrough is considered as monumental for computational biology as ImageNet moment for computer vision and machine learning. AlphaFold treats protein folding as spatial graph problem and uses attention neural networks for end-to-end training.
Advances spread across computer vision, reinforcement learning and more
Facebook open-sourced BlenderBot, the largest conversational Ai bot to date with 9.4 billion parameters and improved decoding techniques for more human-like conversations. DETR introduced the first object detection model to successfully integrate Transformer for the detection pipeline. FasterSeg brought semantic segmentation to real-time speed while keeping accuracy high. These models were published at top conferences like ICLR and CVPR.
In reinforcement learning, DeepMind’s Agent57 exceeded human-level performance on all 57 Atari games pose challenges. Google also released EfficientDet-D7 which achieved state-of-the-art on COCO dataset with 4-9x fewer parameters and 5-11x speedup compared to other detectors.
On the infrastructure side, Facebook made available Detectron2, a major redesign of its previous detection library with cleaner code, faster training speed and richer model support. The library quickly became one of the most popular projects on Github.
Open datasets and challenges enlisted Ai community against Covid-19
When Covid-19 shocked the world, global health organizations identified nine critical questions for mitigating the pandemic. White House Office of Science and Technology Policy subsequently organized an NLP challenge on Kaggle using a new dataset of 200K academic papers to engage Ai experts worldwide.
The initiative brought together contributions from prominent research groups and companies including Allen Institute for Ai, Chan Zuckerberg Initiative, Georgetown University and others. A separate Kaggle challenge was also launched for predicting key pandemic metrics like infections and fatalities globally based on historical data.
These open challenges not only focused cutting-edge Ai resources against the virus, but also created benchmarks for developing predictive models that could inform dynamic resource planning for future healthcare emergencies.
In 2020 Github became the hub for sharing machine learning resources and research developments, as showcased by breakthrough innovations like GPT-3 and AlphaFold. Ai also stepped up against Covid-19 by powering predictive models and drug discovery with unified efforts from government agencies, health organizations and technology leaders.