ZeroFlow: Overcoming Catastrophic Forgetting is Easier than You Think

Tao Feng, Wei Li, DiDi Zhu, Hangjie Yuan, Wendi Zheng, Dan Zhang, Jie Tang

Project

• We introduce ZeroFlow, the first benchmark designed to evaluate gradient-free optimization algorithms for overcoming forgetting.
• We find that forward passes alone are enough to overcome forgetting. Maybe, once is all it takes!

A Stronger Mixture of Low-Rank Experts for Fine-Tuning Foundation Models

Mengyang Sun, Yihao Wang, Tao Feng^†, Dan Zhang, Yifan Zhu, Jie Tang^†

Project

• We propose a new training strategy for MoE-LoRA, to stabilize and boost its feature learning procedure by multi-space projections.

Rethinking the Stability-Plasticity Trade-off in Continual Learning from an Architectural Perspective

Aojun Lu, Hangjie Yuan, Tao Feng^†, Yanan Sun

Project

• We propose a novel insight for exploring the stability-plasticity tradeoff from an architectural perspective.
• We introduce a novel framework denoted DualArch, which serves as a plug-in component for continual learning.

Task-driven Image Fusion with Learnable Fusion Loss

Haowen Bai, Jiangshe Zhang, Zixiang Zhao, Yichen Wu, Lilun Deng, Yukun Cui, Tao Feng, Shuang Xu

Project

• Our framework includes a dynamically updated, learnable fusion loss generation module.
• TDFusion can be applied to any architecture of fusion and task networks.

Make Continual Learning Stronger via C-Flat

Ang Bian, Wei Li, Hangjie Yuan, Chengrong Yu, Mang Wang, Zixiang Zhao, Aojun Lu, Ji Pengliang, Tao Feng^†

Project

• We propose a Continual Flatness (C-Flat) method featuring a flatter loss landscape tailored for continual learning.
• Just a line of code, Makes Continual Learning Stronger.

InstructVideo: Instructing Video Diffusion Models with Human Feedback

Hangjie Yuan, Shiwei Zhang, Xiang Wang, Yujie Wei, Tao Feng, Yining Pan, Yingya Zhang, Ziwei Liu, Samuel Albanie, Dong Ni

Project

• We propose InstructVideo, a model that efficiently instructs textto-video diffusion models to follow human feedback.
• To mitigate the absence of a dedicated video reward model for human preferences, we repurpose established image reward models.

Revisiting Neural Networks for Continual Learning: An Architectural Perspective

Aojun Lu, Tao Feng, Hangjie Yuan, Xiaotian Song, Yanan Sun

Project

• This work is the pioneering effort to employ neural architecture design to shape a CL-friendly architecture.
• We propose ArchCraft method to recraft AlexNet/ResNet into AlexAC/ResAC to guide a well-designed network architecture for CL.

RLIP v2: Fast Scaling of Relational Language-image Pre-training

Hangjie Yuan, Shiwei Zhang, Xiang Wang, Samuel Albanie, Yining Pan, Tao Feng, Jianwen Jiang, Dong Ni, Yingya Zhang, Deli Zhao

Project

• We propose RLIP v2, a fast converging model that enables the scaling of relational pre-training to largescale pseudo-labelled scene graph data.
• RLIPv2 introduces Asymmetric Language-Image Fusion (ALIF), a mechanism that facilitates earlier and deeper gated cross-modal fusion with sparsified language encoding layers.

RLIP: Relational Language-Image Pre-training for Human-Object Interaction Detection

Hangjie Yuan, Jianwen Jiang, Samuel Albanie, Tao Feng, Ziyuan Huang, Dong Ni, Mingqian Tang

Project

• We propose Relational Language Image Pre-training (RLIP), a strategy for contrastive pre-training that leverages both entity and relation descriptions.

Overcoming Catastrophic Forgetting in Incremental Object Detection via Elastic Response Distillation

Tao Feng, Mang Wang, Hangjie Yuan

Project

• First open-source incremental detection codebase.
• We propose a response-based knowledge distillation method for Incremental Object Detection, dubbed Elastic Response Distillation (ERD).

Towards Mask-robust Face Recognition

Tao Feng, Liangpeng Xu, Hangjie Yuan, Yongfei Zhao, Mingqian Tang, Mang Wang

Project

• We introduce a mask-to-face image blending approach based on UV texture mapping, and a self-learning based cleaning pipeline for processing noisy training datasets.
• Considering the long-tail distribution and hard faces samples, a loss function named Balanced Curricular Loss is introduced.

Visible Light Human Activity Recognition Driven by Generative Language Model

Yanbing Yang, Ziwei Liu, Yongkun Chen, Binyu Yan, Yimao Sun, Tao Feng^†

Project

• We propose the first HAR foundation model that leverages generative large language models (LLMs) to decode visible light feature representations into human activity descriptions.

Unigrad-FS: Unified Gradient Projection with Flatter Sharpness for Continual Learning

Wei Li, Tao Feng^†, Hangjie Yuan, Ang Bian, Guodong Du, Sixin Liang, Jianhong Gan, Ziwei Liu

Project

• We propose a plug-and-play method UniGrad to tackle the inconsistency of conflicting and non-conflicting gradients in gradient updating for CL.
• We find the flat minima region through perturbing the model parameters in CL.

Class-Incremental Player Detection with Refined Response-based Knowledge Distillation

Liang Bai, Hangjie Yuan, Hong Song^†, Tao Feng^†, Jian Yang^†

Project

• We propose a refined response-based KD (R²KD) strategy specifically designed for efficient knowledge transfer in incremental detection.
• We are the first to comprehensively study class-incremental detection in real-world sports broadcast scenarios.

Identifying Players in Broadcast Videos using Graph Convolutional Network

Tao Feng, Kaifan Ji, Ang Bian, Chang Liu, Jianzhou Zhang

Project

• A novel person representation method using graph convolu- tional network at the level of relational induction bias is presented.
• We innovatively embed implicit pose structure information into the deep features to form high-level features.