Filter, then reweight: Rethinking optimization granularity in on-policy distillation

Yuying Li, Leqi Zheng, Yongzi Yu, Wenrui Zhou, Xuchang Zhong, Xing Hu, Jing Jin, Hangjie Yuan, Tao Feng^†

Project

• FiRe-OPD leverages a soft-weighting mechanism to effectively mitigate information loss and enhance optimization stability, thereby achieving finer-grained OPD optimization
• We validate the effectiveness of FiRe-OPD across strong-to-weak, single-teacher, and multi-teacher settings, and demonstrate its superiority over recent token-level OPD methods.

Continual GUI Agents

Ziwei Liu, Borui Kang, Hangjie Yuan, Zixiang Zhao, Wei Li, Yifan Zhu, Tao Feng^†

Project

• We introduce Continual GUI Agents, a new task that requires GUI agents to perform continual learning under shifted domains and resolutions.
• We introduce GUI-AiF, a Continual GUI Agents framework with the ability to perform Anchoring in Flux.

A Faster Path to Continual Learning

Wei Li, Hangjie Yuan, Zixiang Zhao, Borui Kang, Ziwei Liu, Tao Feng^†

Project

• C-Flat requires three additional gradient computations per iteration, imposing substantial overhead on the optimization process
• We propose C-Flat Turbo, a faster yet stronger optimizer that significantly reduces the training cost.

Lumos-1: On Autoregressive Video Generation from a Unified Model Perspective

Hangjie Yuan, Weihua Chen, Jun Cen, Hu Yu, Jingyun Liang, Shuning Chang, Zhihui Lin, Tao Feng, Pengwei Liu, Jiazheng Xing, Hao Luo, Jiasheng Tang, Fan Wang, Yi Yang

Project

• Existing AR video generators either diverge from standard LLM architectures, depend on bulky external text encoders, or incur prohibitive latency due to next-token decoding.
• We introduce Lumos-1, an LLM-based unified model for AR video generation with efficient discrete diffusion.

CogFlow: Bridging Perception and Reasoning through Knowledge Internalization for Visual Mathematical Problem Solving

Shuhang Chen, Yunqiu Xu, Junjie Xie, Aojun Lu, Tao Feng, Zeying Huang, Ning Zhang, Yi Sun, Yi Yang, Hangjie Yuan

Project

• We present CogFlow, a novel cognitive-inspired three-stage framework that incorporates a knowledge internalization stage, explicitly simulating the hierarchical flow of human reasoning.

Why Does RL Generalize Better Than SFT? A Data-Centric Perspective on VLM Post-Training

Aojun Lu, Tao Feng, Hangjie Yuan, Wei Li, Yanan Sun

Project

• We posits a data-centric explanation for this phenomenon, contending that RL’s generalization advantage arises from an implicit data filtering mechanism that inherently prioritizes medium-difficulty training samples.
• we introduce Difficulty-Curated SFT (DC-SFT), a straightforward method that explicitly filters the training set based on sample difficulty.

ZeroFlow: Overcoming Catastrophic Forgetting is Easier than You Think

Tao Feng, Wei Li, DiDi Zhu, Hangjie Yuan, Wendi Zheng, Dan Zhang, Jie Tang

Project

• We introduce ZeroFlow, the first benchmark designed to evaluate gradient-free optimization algorithms for overcoming forgetting.
• We find that forward passes alone are enough to overcome forgetting. Maybe, once is all it takes!

A Stronger Mixture of Low-Rank Experts for Fine-Tuning Foundation Models

Mengyang Sun, Yihao Wang, Tao Feng^†, Dan Zhang, Yifan Zhu, Jie Tang^†

Project

• We propose a new training strategy for MoE-LoRA, to stabilize and boost its feature learning procedure by multi-space projections.

Rethinking the Stability-Plasticity Trade-off in Continual Learning from an Architectural Perspective

Aojun Lu, Hangjie Yuan, Tao Feng^†, Yanan Sun

Project

• We propose a novel insight for exploring the stability-plasticity tradeoff from an architectural perspective.
• We introduce a novel framework denoted DualArch, which serves as a plug-in component for continual learning.

Task-driven Image Fusion with Learnable Fusion Loss

Haowen Bai, Jiangshe Zhang, Zixiang Zhao, Yichen Wu, Lilun Deng, Yukun Cui, Tao Feng, Shuang Xu

Project

• Our framework includes a dynamically updated, learnable fusion loss generation module.
• TDFusion can be applied to any architecture of fusion and task networks.

Make Continual Learning Stronger via C-Flat

Ang Bian, Wei Li, Hangjie Yuan, Chengrong Yu, Mang Wang, Zixiang Zhao, Aojun Lu, Ji Pengliang, Tao Feng^†

Project

• We propose a Continual Flatness (C-Flat) method featuring a flatter loss landscape tailored for continual learning.
• Just a line of code, Makes Continual Learning Stronger.

InstructVideo: Instructing Video Diffusion Models with Human Feedback

Hangjie Yuan, Shiwei Zhang, Xiang Wang, Yujie Wei, Tao Feng, Yining Pan, Yingya Zhang, Ziwei Liu, Samuel Albanie, Dong Ni

Project

• We propose InstructVideo, a model that efficiently instructs textto-video diffusion models to follow human feedback.
• To mitigate the absence of a dedicated video reward model for human preferences, we repurpose established image reward models.

Revisiting Neural Networks for Continual Learning: An Architectural Perspective

Aojun Lu, Tao Feng, Hangjie Yuan, Xiaotian Song, Yanan Sun

Project

• This work is the pioneering effort to employ neural architecture design to shape a CL-friendly architecture.
• We propose ArchCraft method to recraft AlexNet/ResNet into AlexAC/ResAC to guide a well-designed network architecture for CL.

RLIP v2: Fast Scaling of Relational Language-image Pre-training

Hangjie Yuan, Shiwei Zhang, Xiang Wang, Samuel Albanie, Yining Pan, Tao Feng, Jianwen Jiang, Dong Ni, Yingya Zhang, Deli Zhao

Project

• We propose RLIP v2, a fast converging model that enables the scaling of relational pre-training to largescale pseudo-labelled scene graph data.
• RLIPv2 introduces Asymmetric Language-Image Fusion (ALIF), a mechanism that facilitates earlier and deeper gated cross-modal fusion with sparsified language encoding layers.

RLIP: Relational Language-Image Pre-training for Human-Object Interaction Detection

Hangjie Yuan, Jianwen Jiang, Samuel Albanie, Tao Feng, Ziyuan Huang, Dong Ni, Mingqian Tang

Project

• We propose Relational Language Image Pre-training (RLIP), a strategy for contrastive pre-training that leverages both entity and relation descriptions.

Overcoming Catastrophic Forgetting in Incremental Object Detection via Elastic Response Distillation

Tao Feng, Mang Wang, Hangjie Yuan

Project

• First open-source incremental detection codebase.
• We propose a response-based knowledge distillation method for Incremental Object Detection, dubbed Elastic Response Distillation (ERD).

Towards Mask-robust Face Recognition

Tao Feng, Liangpeng Xu, Hangjie Yuan, Yongfei Zhao, Mingqian Tang, Mang Wang

Project

• We introduce a mask-to-face image blending approach based on UV texture mapping, and a self-learning based cleaning pipeline for processing noisy training datasets.
• Considering the long-tail distribution and hard faces samples, a loss function named Balanced Curricular Loss is introduced.

Visible Light Human Activity Recognition Driven by Generative Language Model

Yanbing Yang, Ziwei Liu, Yongkun Chen, Binyu Yan, Yimao Sun, Tao Feng^†

Project

• We propose the first HAR foundation model that leverages generative large language models (LLMs) to decode visible light feature representations into human activity descriptions.

Unigrad-FS: Unified Gradient Projection with Flatter Sharpness for Continual Learning

Wei Li, Tao Feng^†, Hangjie Yuan, Ang Bian, Guodong Du, Sixin Liang, Jianhong Gan, Ziwei Liu

Project

• We propose a plug-and-play method UniGrad to tackle the inconsistency of conflicting and non-conflicting gradients in gradient updating for CL.
• We find the flat minima region through perturbing the model parameters in CL.

Class-Incremental Player Detection with Refined Response-based Knowledge Distillation

Liang Bai, Hangjie Yuan, Hong Song^†, Tao Feng^†, Jian Yang^†

Project

• We propose a refined response-based KD (R²KD) strategy specifically designed for efficient knowledge transfer in incremental detection.
• We are the first to comprehensively study class-incremental detection in real-world sports broadcast scenarios.

Identifying Players in Broadcast Videos using Graph Convolutional Network

Tao Feng, Kaifan Ji, Ang Bian, Chang Liu, Jianzhou Zhang

Project

• A novel person representation method using graph convolu- tional network at the level of relational induction bias is presented.
• We innovatively embed implicit pose structure information into the deep features to form high-level features.