Digital Content and Media Sciences Research Division

ZHENG Yinqiang
Digital Content and Media Sciences Research Division Associate Professor
Doctoral Degrees: Dr. Eng. 2013, Tokyo Institute of Technology
Research Fields: Pattern Media

Introduction of research by science writer

Computer Vision for Perfect Shoes Fitting

With digital camera and smartphones so popular now, the world is overflowing with image data. At this point, these images are just images, but if we import them into a computer to extract information, we can identify new value. Computer vision is the name of the technology required to do this. My research in this field focuses on 3D reconstruction and photometric recovery, in which shape and color of objects are reconstructed from two-dimensional images. This type of research is being done all over the world, but the objective of most of this research is to recreate buildings and other cultural heritage structures. My own objective is to generate new businesses by making this technology useful in everyday life.

Selecting Shoes Based on 3D Reconstructions of Your Feet

Online shopping is flourishing. It's convenient now to purchase anything on the Net. Did you know, though, that shoes don't sell very well online compared to other products? That's because it's especially hard to find shoes that fit. Knowing this, I'm working on 3D reconstructions of foot shapes.

First, I use a smartphone or other similar devices to take a video of the foot for several dozens of seconds. What you have to pay attention to is making sure you shoot from the front, from the back, and from the sides. Then, based on the video recorded in this way, I select several dozen still images that capture the foot from each direction. Next, I reconstruct the foot in three dimensions based on these images.

Once this becomes relatively easy to do, you could send a video of your feet to the shoe seller by email when you order a pair of shoes. The shoe seller could then reconstruct your feet in 3D from the video you send, fit your feet to various shoes, and send a pair that fits perfectly, eliminating situations where you buy a pair of shoes and then have to return them because they don't fit.

Error Tolerances of Less than Several Millimeters

I'm currently creating the software needed to reconstruct a foot in 3D on a computer. I've done 3D reconstructions of buildings and sculptures, for example, but 3D reconstruction software for feet requires extremely high precision. For example, in reconstructing something big like a building, even if the reconstruction differs from the actual building by several dozen centimeters, it's not really a problem. With feet, a margin of error of even several millimeters is unacceptable. One reason a margin of error arises during reconstruction is that lines that appear straight to the naked eye when captured by camera can appear in video images as slightly curved. So, the software has to be able to correct for these curves and optimize the model in the case of varying focal length. That's the new challenges I'm working on now.

Catching up with the Human Eye and Brain

I'm also involved in research on image color. In biology research, there's a need to trace the movement of cells. I've developed software that assigns different colors to cells, depending on type, and traces their movement depending on color. Likewise, endangered coral reefs indicate their relative state through color. To protect the ocean's coral reefs, there's a project underway to automatically identify the status of coral reefs based on image colors and to make use of these data. I'm one of the participants in this effort.

In this way, computer vision is beginning to help us in our everyday lives, and this wish of mine is starting to come true. At the same time, my ultimate goal is to get cameras and computers, the basic tools of computer vision, to function more like the human eye and brain. That's going to take some more time, but through my current research I plan to overcome every challenge that comes up.

PDF Download

Article researched and organized by Akiko Ikeda