Research

No longer science fiction: Going beyond Go

From a young age, Lang Huang was fascinated by science fiction--namely, time travel as shown in Back to the Future but also technology like artificial intelligence (AI) and advanced robotics that emulated human thought and anatomy. At the time, they seemed only achievable far into the future. However, one chance encounter--long before the rise of ChatGPT and other large language models (LLMs)--would forever shape Huang's vision for AI.

In March 2016, AlphaGo, a computer program developed by Google DeepMind that specializes in the board game Go, faced off against South Korean former professional Go player Lee Sedol in a five-game match, one that ended in a 4-1 victory for AlphaGo and the AI community.

"At the time, I was still a college student majoring in computer science, but I was not very sure what I should do for my next step," Huang recalls. "I think this was the first moment when I realized that a machine could actually outperform humans, and it was one of the decisive moments that led me to take on AI research and join the AI industry to create something that could learn on its own and surpass humans."

Developments in deep learning would later see AI move from human-guided data to deeper networks that figure things out themselves and could handle more data with less human supervision, allowing researchers to eventually develop transformer models capable of supporting LLMs. Driven by his interest in how models improve over time, Huang went on to investigate self-supervised and multimodal representation learning,^［*1］ which require minimal human intervention and combine data like audio and text, respectively, earning his PhD in Information & Communication Engineering at The University of Tokyo and becoming an assistant professor at NII in 2025.

"I was curious about the autoregressive model that LLMs use because it also operates on a time axis. It generates the next thing based on what is already generated--for example, it outputs 'two' because it already generated 'zero' and 'one.' Unlike text or language, which has a specific order--usually from left to right--images and videos do not really work that way," Huang explains. "Generating an image patch by patch from left to right, top to bottom, or generating a video only by looking at the past is odd for humans. Anticipation of the future is something humans always have in mind, and I think autoregressive models should do something similar to better understand what they should generate and what they should remember while generating--so that the future informs the present."

This challenge made the vision modality a "natural entry point" for him to research computer vision, serving as the springboard for Video-Mirai, a real-time video generation model capable of foresight.

Video-Mirai: Foreseeing the future

A part of a larger autoregressive framework called Mirai (meaning "future" in Japanese), Video-Mirai employs both an autoregressive model and a diffusion model--generating frames in an autoregressive manner but adding and removing visual noise during generation based on future frames. Compared to purely autoregressive models, this hybrid model is more stable for long-form videos due to its ability to adapt.

"If a user discovers that the generation quality is decreasing in the middle, they can use that signal as supervision to adapt the model so that it can improve and keep generating. Adaptation is part of the Video-Mirai model we are developing, and we have to think about adapting it over time because it is a causal model with a time axis," Huang says. "In the current era, we have witnessed how big, reactive models cannot adapt to changing environments. They have to adapt as the environment changes."

For Huang, the self-improvement process involves three key steps: architecture modeling, learning through specific loss functions, and adaptation that updates the model when a signal is received. When a model learns a new skill, only a small set of its weights changes, often tens of thousands out of a vastly larger total. That set of changes, or weight deltas, works like a software update based on open-source software--it can be shared, and anyone with the base model can apply it to gain the new ability, without ever seeing the data the skill was learned from.

Another natural development of Huang's research is his foray into humanoid robotics.

"Video is a natural interface for robots to interact with the world, and humans communicate using language. We need multimodality for interaction between humans and machines," Huang says. "Just as Video-Mirai predicts the future to inform current frames, another project of ours uses camera inputs to predict the actions that a humanoid needs to take to complete a task like folding a towel or picking up an object."

Manifesting science fiction

For Video-Mirai, suppressing hallucinations and uncanny movements poses a challenge.

"If we generate a very long video, it might become inconsistent if we do not have any memory of what has already been generated. At the same time, the video may look plausible, but the movements featured could be completely wrong in terms of physics. There should be something connecting the RGB frames--the image frames of the video--to the actual movements featured using physical laws, for instance," Huang says. "At my lab, we are working on adding memory to the model so it can remember what has been generated, what is important, and maybe what should be generated in the future using that memory. We also have a project imposing physically aware constraints on video generation."

Nevertheless, ideas from science fiction continue to fuel Huang's optimism for the years ahead.

"If I could have a personal-assistant robot that did housework for me, bought groceries for me, communicated with me, understood my intentions, and did some positive jobs for me, that would be interesting," he says with a smile. "Honestly though? I would rather have a time machine. I am very curious and excited about what the future has in store because the developments are quite rapid, and I can hardly imagine what the next 10 years will be like. With the barrier to research lowered because of AI, as human researchers, we have to figure out what we should do in the future."