Kun Zhou (周昆)
Kun Zhou is a postdoctoral researcher at UC San Diego, working with Zhiting Hu and Biwei Huang. His research interests are in world model, with focuses on large language models, multimodal learning, and their applications in complex scenarios. He obtained Ph.D at School of Information, Renmin University of China, under the supervision of Wayne Xin Zhao and Ji-Rong Wen, master degree from Peking University, and bachelor from Hohai University.
Research Opportunities: I am consistently seeking highly motivated students, to work with me in various research projects. If you are interested in my current research topics about improving the advanced capabilities of LLMs and World Models (Complex Reasoning, Multimodal Understanding and Generation), feel free to email me.
News
We release Causal-Copilot, an LLM-oriented toolkit for Automatic Causal Analysis!
We release De-Arena, a Decentralized Arena via Collective LLM Intelligence for Building Automated, Robust, and Transparent LLM Evaluation for Numerous Dimensions!
Our JiuZhang3.0 is accepted by NeurIPS 2024. Thanks for my comrades!
Research Interests
Currently, my primary research lies in large language models (LLMs). I am devoting to answer the following research problems:
- How to Build a Strong World Model?
- Background: A Comprehensive Survey about LLMs Arxiv
- Training: Curriculum Pre-training YuLan-Chat
- What is strong? Studying Benchmark Leakage for Fair Evaluation Arxiv, Decentralized Arena with Collective Intelligence Blog
- How to Enhance the Reasoning Capability? (Knowledge/Mathematical Reasoning)
- Data Refinement and Synthesis: Distilling Data Synthesis Ability NeurIPS 2024
- Training Algorithm: RL with Fine-grained Rewards ACL 2024-Findings, Low-Redundant Optimization EMNLP 2024
- Tool-Augmented Agent: Tool-Augmented Reasoning Agent EMNLP 2023, Iterative Deliberation for Error Check NeurIPS 2023
- Reasoning Agent over Structured Data: Toolization with Prompting Method EMNLP 2023, Autonomous Agent for Reasoning over KG KG-Agent
- How to Enhance the Multimodal Abilities? (Image/Video Understanding/Generation )
- Training Data Management: Complex Visual Instruction Synthesis ComVint, Visual Instruction Value Estimation TIVE
- Multimodal Alignment: Object Hallucination Evaluation EMNLP 2023, Visual Vulnerabilities for Jailbreaking ECCV 2024, Event-oriented Long Video Understanding VIM
Before the era of large language models, I have investigated pre-trained language models (PLMs), and their application on sentence representation learning and complex reasoning.
- Effective Fine-tuning PLMs
- General Fine-tuning Strategy: Visual-Augmentation ACL 2023, Over-Parameterization ACL 2023
- PLM as Diffusion Model Masked Language Model likes Diffusion Model EACL 2024 Evaluation and Model Insight Award
- PLM-based Sentence Representations
- Unsupervised Sentence Representations: Debiased Contrastive Learning ACL 2022
- Dense Text Retrieval: Ambiguous Negatives Sampling EMNLP 2022, Multi-task Bottlenecked MAE ECML-PKDD 2023
- PLM-based Complex Reasoning
- Math-specific PLMs: Curriculum Pre-training KDD 2022, Unified MoE Pre-training KDD 2023
- PLM+KG for Knowledge Reasoning: Simple Rule-based KG Encoder NAACL 2022, Unified Relation Learning ICLR 2023
I also have done interesting research works about conversational recommender systems, sequential recommendation and dialog system. My whole publication list is in Publication
Open-Source Projects
Most of my research work are open-source. Here are some my preferable projects!
- JiuZhang3.0
- We publish JiuZhang3.0-7B/8B/8X7B, a series of LLMs with new SOTA performance on mathematical reasoning tasks, with only 1/4 cost for training and data synthesis!
- YuLan-Chat
- YuLan-Chat-3-12B is a LLM trained from scratch. It is pre-trained on over 1.6TB tokens of English, Chinese, and multilingual data, and then supervised fine-tuned via curriculum learning with high-quality English and Chinese instructions and human preference data.
- YuLan-Chat-2 is developed by continually-pretraining and instruction-tuning LLaMA-2 with high-quality English and Chinese data. It can support 8k maximum length now, and outperforms other llama-2-based bilingual LLM on MMLU, C-Eval and AGIEval-Gaokao.
- YuLan-Chat is a chat-based large language model developed based on fine-tuning LLaMA. YuLan-Chat can chat with users to well follow English or Chinese instructions, and can be deployed on an GPU (A800-80G or RTX3090) after quantization. YuLan (Simplified Chinese 玉兰), is the campus flower of Renmin University of China.
- LLMBook
- A Chinese book for everyone to master the knowledge about large language models.
- LLMSurvey
- A collection of papers and resources related to Large Language Models. The organization of papers refers to our survey “A Survey of Large Language Models”.
Experiences
2022.4 - 2023.6, Research Intern, NLC Group, MSRA.
Mentor: Yeyun Gong, Nan Duan
2021.9 - 2022.4, Research Intern, iFLYTEK Research.
Mentor: Jing Sha, Shijin Wang
2019.12 - 2021.5, Research Intern, NLP center, Meituan Dianping.
Mentor: Sirui Wang, Fuzheng Zhang
2018.8 - 2019.6, Research Intern, XiaoIce, Microsoft Asia.
Mentor: Kai Zhang, Yu Wu
Rewards and Honors
- EACL 2024 Evaluation and Model Insight Award
- Five Highly-cited Papers are selected as the most influential KDD/WWW/CIKM papers by PaperDigest:
- 2022 Baidu Scholarship (10 PhD Students) Link
- 2022 Bytedance Scholarship (10 PhD Students in China Mainland) Link
- 2022 MSRA Fellowship (12 PhD Students in Asia-Pacific-region) Link
- 2022 Baosteel Scholarship (12 Students in RUC).
- 2022 National Scholarship.
- LIC2021 Multi-Skill Dialog Challenge Link
- Ranked 1st in Automatic Metrics Track, 3rd in Human Evaluation Track
- LIC2020 Conversational Recommendation Challenge Link
- Ranked 2st in Automatic Metrics Track, 4rd in Human Evaluation Track
- 2018 Jane Street Electronic Trading Challenges
- Ranked 1st
- 2018 The Data Open Challenges Citadel Datathen
- Ranked 2nd
- 2016 American Mathematical Contest in Modeling
- Honorable Mention
- 2015 China Undergraduate Mathematical Contest in Modeling
- National Second Prize
- 2015 National Zhou Peiyuan college student mechanics competition
- National Third Prize
- 2015 Jiangsu Province Undergraduate Mechanical Competition
- The First Prize
Service
- IJCAI 2021
- Senior PC Reviewer
- AAAI, IJCAI, KDD, SIGIR, WWW, WSDM, ACL, EMNLP, COLING, TOIS, TORS
- PC Reviewer
Zhihu Posts
I love writing blogs in my free time. Here are some my favorite posts!