OpenAI recently announced their DALL·E 2 system capable of creating images based on a textual description. It’s the second version of the system, and the first one was published nearly a year ago. …

In August 2021 Stanford announced establishing the Center for Research on Foundation Models (CRFM) as part of the Stanford Institute for Human-Centered AI (HAI). The center then published a large 200+ page report called “On the Opportunities and Risks of Foundation Models” in which the authors set the term “Foundation…

This is a part about ASICs from the “Hardware for Deep Learning” series. The content of the series is here. As of beginning 2021, ASICs now is the only real alternative to GPUs for 1) deep learning training (definitely) or 2) inference (less so, because there are some tools to…

JAX by Google Research is getting more and more popular. Deepmind recently announced they are using JAX to accelerate their research and already developed a set of libraries on top of JAX. There are more and more research papers build using JAX, say, the recent Vision Transformer (ViT). JAX demonstrates…

OpenAI just published a paper “Language Models are Few-Shot Learners” presenting a recent upgrade of their well-known GPT-2 model — the GPT-3 family of models, with the largest of them (175B parameters), the “GPT-3” is 100x times larger than the largest (1.5B parameters) GPT-2! Language Models are Few-Shot Learners Recent work has demonstrated substantial gains on many NLP tasks and benchmarks by pre-training on a large corpus of…arxiv.org

There are many floating point formats you can hear about in the context of deep learning. Here is a summary of what are they about and where are they used. FP80 A 80-bit IEEE 754 extended precision binary floating-point format typically known by the x86 implementation started from the Intel 8087…

Part 3: ACT in Transformers Part 1 is here. Part 2 is here. Finally, ACT came into transformers. The Universal Transformer exploits the original idea of ACT applied to a transformer instead of an RNN. The authors say they are adding recurrent inductive bias of RNNs with a dynamic per-position halting mechanism to the transformer…

Part 2: ACT in Residual Networks Part 1 is here. The next step in ACT development was done by Michael Figurnov (then an intern at Google and Ph.D. student in Higher School of Economics, now a researcher in DeepMind). “Spatially Adaptive Computation Time for Residual Networks” by Michael Figurnov, Maxwell D. Collins, Yukun Zhu, Li Zhang…

Part 1: ACT in RNNs There is an interesting little-known topic of Adaptive Computation Time (ACT) in Neural Networks. It is applicable to different kinds of neural networks (RNN, ResNet, Transformer) and you can use this rather general idea somewhere else as well. The general idea is that some complex data might require more computations…