EVENT

Talk on "Self-Attention as Graph Filters: Lightweight Transformer via Unrolling of Graph Smoothness Algorithms" by Prof. Gene Cheung

We are pleased to inform you about the upcoming seminar by Prof. Gene Cheung titled:"Self-Attention as Graph Filters: Lightweight Transformer via Unrolling of Graph Smoothness Algorithms" Everyone interested is cordially invited to attend!

Title:

Self-Attention as Graph Filters: Lightweight Transformer via Unrolling of Graph Smoothness Algorithms

Abstract:

We build interpretable and lightweight transformer-like neural nets by unrolling iterative algorithms that minimize graph smoothness priors. The crucial insight is that a normalized signal-dependent graph learning module amounts to a variant of the basic self-attention mechanism in conventional transformers. Unlike "black-box" transformers that require large key, query, and value matrices to compute transformed dot products as affinities and subsequent output embeddings, resulting in huge parameter sets, our unrolled networks employ shallow CNNs to learn low-dimensional features per node to establish pairwise distances and construct sparse similarity graphs. At each layer, given a learned graph, the target interpolated signal is a low-pass filtered output derived from an assumed graph smoothness prior, leading to a dramatic reduction in parameter count. Experiments for image interpolation, denoising, traffic prediction, and EEG classification verify SOTA performance and parameter efficiency of our graph-based unrolled networks compared to conventional transformers.

Speaker:

Gene Cheung, professor, department of electrical engineering & computer science, York University, Toronto, Canada

Gene Cheung received the M.S. and Ph.D. degrees in electrical engineering and computer science from the University of California, Berkeley, in 1998 and 2000, respectively. He was a senior researcher in Hewlett-Packard Laboratories Japan, Tokyo, from 2000 till 2009. He was an assistant then associate professor in National Institute of Informatics (NII) in Tokyo, Japan, from 2009 till 2018. He is now a professor and York research chair (YRC) in York University, Toronto, Canada. His research interests include 3D imaging and graph signal processing. He has served as associate editor for multiple journals, including IEEE Transactions on Multimedia (2007--2011), IEEE Transactions on Circuits and Systems for Video Technology (2016--2017) and IEEE Transactions on Image Processing (2015--2019). He currently serves as senior associate editor for IEEE Signal Processing Letters (2021--present). He is a co-author of several paper awards and nominations, including the best student paper finalist in ICASSP 2021, best student paper award in ICIP 2013, ICIP 2017 and IVMSP 2016, best paper runner-up award in ICME 2012, and IEEE Signal Processing Society (SPS) Japan best paper award 2016. He is a recipient of the Canadian NSERC Discovery Accelerator Supplement (DAS) 2019. He is a fellow
of IEEE.

Time/Date:

11:00-12:00 December 16 (Tuesday), 2025

Place:

Room 1512, NII and online

Contact:

If you would like to join, please contact by email.
Email :kazuya[at]nii.ac.jp