New Papers

https://www.promptingguide.ai/

https://github.com/rxlqn/awesome-llm-self-reflection

UPRISE: Universal Prompt Retrieval for Improving Zero-Shot Evaluation



# New Papers




## Tool Use


### Tool for other tasks

TORL: Scaling Tool-Integrated RL

ReTool: Reinforcement Learning for Strategic Tool Use in LLMs (see related work of this paper)


### Survey

**Tool learning with large language models: A survey (2405.17935)** -> see Fig. 1-2 to papers before 2024Q2.

**Tool Learning with Foundation Models (2304.08354)**

### Datasets

**ToolACE: Winning the Points of LLM Function Calling (2409.00920)**

Beyond collecting real API data, we developed a Tool Self-Evolution
Synthesis (TSS) module that synthesizes API definitions with various data types and constraints.
Specifically, we utilize pretraining data to extract an API context tree
 
**Toolalpaca: Generalized tool learning for language models with 3000 simulated cases.(2306.05301)**

**Toolllm: Facilitating large language models to master 16000+ real-world apis (2307.16789)** (toolbench)

**Gorilla: Large language model connected with massive apis (2305.15334)**

**API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMs (2304.08244)**

**APIGen: Automated PIpeline for Generating Verifiable and Diverse Function-Calling Datasets (2406.18518)**

**TravelPlanner: A Benchmark for Real-World Planning with Language Agents (sandbox) (2402.01622)**

**Toolformer: Language Models Can Teach Themselves to Use Tools (2302.04761)**

**τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains (2406.12045)**

**StableToolBench: Towards Stable Large-Scale Benchmarking on Tool Learning of Large Language Models (2403.07114)**

**On the Tool Manipulation Capability of Open-source Large Language Models (2305.16504)**

**RestGPT: Connecting Large Language Models with Real-World RESTful APIs (2306.06624)**

**Nestools: A dataset for evaluating nested tool learning abilities of large language models (2410.11805)**

**Api-blend: A comprehensive corpora for training and benchmarking api llms. (2402.15491)**

**Seal-tools: Self-instruct tool learning dataset for agent tuning and detailed benchmark (2405.08355)**

**Taskbench: Benchmarking large language models for task automation. (2311.18760)**

### evaluation

**ToolEyes: Fine-Grained Evaluation for Tool Learning Capabilities of Large Language Models in Real-world Scenarios （2401.00741）**

BFCL: https://gorilla.cs.berkeley.edu/blogs/8_berkeley_function_calling_leaderboard.html

**ToolQA: A Dataset for LLM Question Answering with External Tools (2306.13304)**

**MINT: EVALUATING LLMS IN MULTI-TURN INTERACTION WITH TOOLS AND LANGUAGE FEEDBACK (2309.10691)**

**NESTFUL: A Benchmark for Evaluating LLMs on Nested Sequences of API Calls (2409.03797)**

**Nexusraven: a commercially-permissive language model for function calling.**

**Granite-function calling model: Introducing function calling abilities via multi-task learning of granular tasks. (2407.00121)**

**Wtu-eval: A whether-or-not tool usage evaluation benchmark for large language models. (2407.12823)**

**A comprehensive evaluation of tool-assisted generation strategies.(2310.10062)**

**Planning, Creation, Usage: Benchmarking LLMs for Comprehensive Tool Utilization in Real-World Complex Scenarios (2401.17167) (ultratools)**

**T-Eval: Evaluating the Tool Utilization Capability of Large Language Models Step by Step (2312.14033)**

**ToolSword: Unveiling Safety Issues of Large Language Models in Tool Learning Across Three Stages (2402.10753)**

**AppWorld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agents (2407.18901)**

**Metatool benchmark for large language models: Deciding whether to use tools and which to use (2310.03128)**

Rapid. Rapid api, 2023. https://rapidapi.com/

**Identifying the risks of lm agents with an lm-emulated sandbox (2309.15817)**

**m&m’s: A benchmark to evaluate tool-use for multi-step multimodal tasks. (2403.11085)**

**Gta: A benchmark for general tool agents (2407.08713)**

**Sciagent: Tool-augmented language models for scientific reasoning（2402.11451)**

**Ctooleval: A chinese benchmark for llm-powered agent evaluation in realworld api interactions (2024.8)**

**Toolsandbox: A stateful, conversational, interactive evaluation benchmark for llm tool use capabilities (2408.04682).**

**Agentboard: An analytical evaluation board of multi-turn llm agents (2401.13178)**

**Injecagent: Benchmarking indirect prompt injections in toolintegrated large language model agents (2403.02691)**

**Tooltalk: Evaluating tool-usage in a conversational setting (2311.10775)**

**Gaia: a benchmark for general ai assistants (2311.12983)**

**Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments (2404.07972)**

**Ml-bench: Large language models leverage open-source libraries for machine learning tasks. (2311.09835)**

**AGENTBENCH: EVALUATING LLMS AS AGENTS (2308.03688)**

**An LLM compiler for parallel function calling. （2312.04511）**

重要！fc complier可能和我们的CI有关。

### Training

**WebVoyager: Building an End-to-End Web Agent with Large Multimodal Models (2401.13919)**

**Genegpt: Augmenting large language modelswith domain tools for improved access to biomedical information （2304.09667）**

**ADC: Enhancing Function Calling Via Adversarial Datasets and Code Line-Level Feedback (2412.17754)**

CodeAlpaca [3] uses 21 seed tasks and generates a 20k
dataset via self-instruct [4], while Wizardcoder [5], MagicCoder [6], and WaveCoder [7] apply advanced heuristics and
novel data generation processes based on open-source code
snippets and code instruction data to enhance the complexity
of initial code instructions

CodeNet [23], which comprises approximately 14
million code snippets, and POJ104 [24], a smaller dataset
consisting of 52,000 code snippets focused on 104 algorithmic problems

**Efficient and scalable estimation of tool representations in vector space (2409.02141)**

**Look before you leap: Towards decision-aware and generalizable tool-usage for large language models (2402.16696)** (tooldeer)

**Toolverifier: Generalization to new tools via self-verification（2402.14158）**

**Llms in the imaginarium: Tool learning through simulated trial and error. (2403.04746)**

**Mllm-tool: A multimodal large language model for tool agent learning (2401.10727)**

**iTool: Boosting Tool Use of Large Language Models via Iterative Reinforced Fine-Tuning (2501.09766)**

**Let Me Do It For You: Towards LLM Empowered Recommendation via Tool Learning (2405.15114)**

**AnyTool: Self-Reflective, Hierarchical Agents for Large-Scale API Calls (2402.04253)**

**Hammer: Robust function-calling for on-device language models via function masking (2410.04587)**

重要！这个里面的内容可以参考一下。

**Facilitating multiturn function calling for llms via compositional instruction tuning （2410.12952）**

重要！这个里面的内容可以参考一下。

Enhancing tool retrieval with iterative feedback from large language models

Confucius: Iterative Tool Learning from Introspection Feedback by Easy-to-Difficult Curriculum. 

Magicoder: Source code is all you need.

Hugginggpt: Solving ai tasks with chatgpt and its friends in huggingface.

. Executable code actions elicit better llm agents

watt-tool

### Data synthesis

Magnet: Multi-turn Tool-use Data Synthesis and Distillation via Graph Translation

### Prompt Engineering

**Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Models (2303.04671)**

ToolChain\*: Efficient Action Space Navigation in Large Language Models with A\* Search

Critic: Large language models can self-correct with tool-interactive critiquing

Tool documentation enables zero-shot tool-usage with large language models

Measuring and narrowing the compositionality gap in language models

**ToolRerank: Adaptive and Hierarchy-Aware Reranking for Tool Retrieval (2403.06551)**

Tool documentation enables zero-shot tool-usage with large language models

### Tool Retrieval

**Towards completeness-oriented tool retrieval for large language models (2405.16089)**

Conversely, semantic-based methods, such
as ANCE [45], TAS-B [12], coCondensor [7], and Contriever [15] ...

### Tool Creation

CREATOR: Tool Creation for Disentangling Abstract and Concrete Reasoning of Large Language Models

Large Language Models as Tool Makers (2305.17126)

Craft: Customizing llms by creating and retrieving from specialized toolsets.

EASYTOOL: Enhancing LLM-based agents with concise tool instruction

### Early

**Internetaugmented language models through few-shot prompting for open-domain question answering (2203.05115)**

**ViperGPT: Visual Inference via Python Execution for Reasoning (2303.08128)**

**TaskMatrix.AI: Completing Tasks by Connecting Foundation Models with Millions of APIs (2303.16434)**

TALM: Tool Augmented Language Models

ToolkenGPT: Augmenting Frozen Language Models with Massive Tools via Tool Embeddings

Making Language Models Better Tool Learners with Execution Feedback

Chameleon: Plug-and-play compositional reasoning with large language models.

### Uncategorized

Art: Automatic multistep reasoning and tool-use for large language models.

Tool-lmm: A large multimodal model for tool agent learning.

GPT4Tools: Teaching Large Language Model to Use Tools via Self-instruction

Learning to use tools via cooperative and interactive agents.

Openagi: When llm meets domain experts.

A solution-based llm apiusing methodology for academic information seeking.

Advancing tool-augmented large language models: Integrating insights from errors in inference trees

TPTU: Task planning and tool usage of large language modelbased AI agents.


---

Reliable LLM-based user simulator for task-oriented dialogue systems.


**CLIMB: CLustering-based Iterative Data Mixture Bootstrapping for Language Model Pre-training (2504.13161)**

**Reason-RFT: Reinforcement Fine-Tuning for Visual Reasoning (2503.20752)**

**A Minimalist Approach to LLM Reasoning: from Rejection Sampling to Reinforce (2504.11343)**

**TRUST REGION PREFERENCE APPROXIMATION: A SIMPLE AND STABLE REINFORCEMENT LEARNING ALGORITHM FOR LLM REASONING (2504.04524)**

**What’s Behind PPO’s Collapse in Long-CoT? Value Optimization Holds the Secret (2503.01491)**

**The Evolution of LLM Adoption in Industry Data Curation Practices (2503.01491)**

**Demystifying Long Chain-of-Thought Reasoning in LLMs (2405.09798)**

**Magnet: Multi-turn Tool-use Data Synthesis and Distillation via Graph Translation (2503.07826)**

**The N+ Implementation Details of RLHF with PPO: A Case Study on TL;DR Summarization (2403.17031)**

**UNA: UNIFYING ALIGNMENTS OF RLHF/PPO, DPO AND KTO BY A GENERALIZED IMPLICIT REWARD FUNCTION (2408.15339)**

## Many-shot

## Self-train

## Math

## Reflection

**Many-shot In-Context Learning (2404.11018)**

放了最多2048个example到context里面。不同的顺序对不同的子任务performance不一样，但平均起来看没有什么太大区别。

**Buffer-of-Thoughts** 超高的game of 24成功率
基本的想法是生成一些“template”然后在解决问题时取用。

"self-training": model generated

What is the fundamental limits of LLM self-training?


我们要做的这个东西好像带一点many-shot ICL （但是不是example而是template）又带一点reflection（因为要把题目加入训练）和self-train（因为要self-train）

其实或许可以理解为一种active learning，就是说绝大部分情况下可以自己learn，但是少部分情况下需求人工标注













首先从方法上，偏向于finetuning（学校里面太缺计算资源了，希望能做一些学校里做不到的事情）
就现在来说，我比较关心三个问题：第一个是multi-turn LLM，第二个是understanding real numbers，第三个是LLM直接解决传统的mujoco问题。multi-turn LLM这里是以archer这篇文章为代表。
understanding real numbers这篇是以https://arxiv.org/abs/2401.03735这篇文章为代表的。
今天聊天的时候提出了一个LLM建立了“model-based”的概念，那具体是什么意思呢……？

# Papers Summary  

**LDB: Large Model Debugger (2402.16906)**

Script generate control flow graph; analyze in breakpoints between flow graph nodes  

**Don't Trust: Verify (2403.18120)**

Generate formal languages and verify  

**Can LLM Infer Causation from Correlation? (2306.05836)**

new problems of inference, new dataset  

**CREATOR: ... Disentangling Abstract and Concrete Reasoning of LLM (2305.14318)**

let an agent to generate high-level plans as decision; execute and rectify using compiler feedback; create tools to address this  

**DeepSeekMath (2402.03300)**

group relative policy optimization (grpo) instead of PPO  

**MUSR: Multi-Step Soft Reasoning (2310.16049)**

new dataset of building logic trees  

**Let's Verify Step by Step (2305.20050)**

train a good reward model; use "feedback in the middle" / "supervise in the proecess" 

**Solving Math Word Problems (2211.14275)** 

Early attempt of solving math problem. Basically standard RLHF. Tried by-step and overall feedback. It seems that RM-weighted return is much greater than majority or greedy answers. 

**Scaling Relationship with LLM (2308.01825)**

rejection sampling fine-tuning; data augmentation  

**LM Understand Numbers, at Least Partially (2401.03735)**

empirical study on the internal embedding with numbers as input in LM; uses a single linear layer to "decode"  

**AlphaMath Almost Zero (2405.03553)**

MCTS evaluation to get a value function as reward model; do process supervision with reward assigned by the reward model  

**Best Practices and Lessons Learned in Synthetic Data (2404.07503)**  

an introduction to variant synthetic data for LLM papers

**Fine-Tuning LVLM ... Using RL (2405.10292)**  

RL+VLM  

**Controlling LLM Agents with Entropic Activation Steering (2406.00244)**  

to increase diversity of LLM output; train a "steering vector"  

**InterCode: Standardizing and Benchmarking Interactive Code with Execution Feedback (2306.14898)**  

Another paper which uses agent code command as action and interpreter as environment to interact.

**ToolAlpaca (2306.05301)**  

经典的framework，它在自己的语料库上训练，其中包含了大量的tool use。

**ToolFormer (2302.04761)**  

和toolalpaca类似，但是是更早的工作，没有那么成熟。

**In-Context AE for Context Compression in LLM (2307.06945)**  

实际上和activation beacon或者recurrent transformer都很像， 总的来说就是一些特殊的、在上一个chunk结尾继承了大部分信息的token。

**AutoAct (2401.05268)**  

autoact同样是分为几个agent：meta-agent, plan-agent，tool-agent和reflect-agent。meta-agent会负责选择合适的工具，然后还会用react生成一些成功的trajectory作为training source。在这个基础上用lora finetune其他三个agent。比起之前诸如fireact和lumos这样的工作，autoact不需要使用GPT-4。autoact在Tab.1里测试了大量LLM agent，可以参考。

**AutoGen (2308.08155)**

同样是分为几个agent：assistant， proxy和groupchat。user和assistant两个agent会互相对话，proxy会register一些函数，而assistant则会运行这些函数来返回结果。proxy同时也有python interpreter作为assistant的feedback。 group manager会动态地添加agent/维持一组agent的交流。

**LLM can Strategically Deceive their Users (2311.07590)**  

The first work that shows LLM cheats even when not instructed to do so but under high pressure from human instructors

**Executable Code Actions Elicit Better LLM Agents (2402.01030)**  

use code as action, makes trajectory more compact and minimizes the number of steps

**Data-Copilot (2306.07209)**  

data-copilot是一个用来实时处理大规模信息的LLM agent。它会自动生成代码来处理这些消息，然后调用一些之前设计好的处理接口。这些接口本身也是在搜索数据时LLM agent自己建立起来的。

**WavCraft: Audio Editing & Generation with LLMs (2403.09527)**  

这篇paper是整合了声音输入和编辑的multimodal model，他的编辑是利用了tool API（利用提前整合好的instruction template），输入利用了专门的audio analysis module。实际上外接了大量的已有处理声音的模型。

**SAGE: Semantic and Actionable Parts for Generalizable Manipulation of Articulated Objects (2312.01307)**

实际上是一个visual input的agent，它自己可以描述场景。它调用3D segmentation模型以获知在一个物体上哪些部位是可以动的，这样就可以把现在可以做的action补充到描述里。然后LLM作为一个不断接受feedback的agent来输出policy，使用别的经典算法来估测需要移动的幅度作为反馈。输出是一个tuple，包含目标、动作和参数，这些东西会被送进motion planner。

**Simulating Opinion Dynamics with Networks of LLM-based Agents (2311.09618)**

实际上是一个偏向社会学的工作，试图通过LLM的自然语言来描述人类的心态进而考虑其行为，而不是简单地用先验的公式来描述人类行为。LLM population里每个agent的想法写出来实际上就是模拟每个人内心的想法。当然，所有的信息交换也是通过模拟的twitter实现的。

**Towards Unified Alignment Between Agents, Humans and Environment (2402.07744)**

大部分篇幅描述了一种设计LLM agent的原则，即要认识到人类的目的、要对环境dynamics有认知和要满足经济性等约束。他也提出了一种方法，从过去成功的traj里抽取关键动作，然后根据关键动作的走向匹配最接近的成功案例这样。

**Self-Training Language Models in Arithmetic Reasoning (ICLR 2024 workshop)**

proposes calcX, a new dataset; use calculator API and self-training with preference optimization.

**Reinforced Self-Training (REST) for LLM (2308.08998)**

从本质上是一种curriculum learning（逐步提高接受data的bar）+rejection sampling（只有return足够高的data才会被接受进入训练）

**LLM can self-improve (2023 emnlp)**

跟我们的想法有一点像，他通过voting选出来的最好的reasoning path会被加入到之后的training samples里面

**Agents: An Open-Source Framework for Autonomous Language Agents (2309.07870)**

multi-agent，是一种“基础模型”，它整合了tool use、multi-agent、HCI、symbolic等内容，可以被用于后续的训练中。

**LEAGUE++ (ICLR 2024)**

LLM robotic agent，使用形式化的语言描述plan，生成reward，同时调用一个semantic skills library。这个library会被rollout中余弦相似度最像的traj不断更新。

**Agent LUMOS: Unified and Modular Training for Open-Source Language Agents**

LUMOS features a learnable, unified and modular architecture with a planning module that learns highlevel subgoal generation, and a grounding module trained to translate these into the actions using various tools in the execution module.

仍然是同一个套路：一个planning-agent自然语言生成high-level plan，再用grounding agent细化成标准化动作，最后用各种API实现。

**R2E: Turning any Github Repository into a Programming Agent Environment**

对LLM的自动化testbed生成器。对于每个repo，首先找出一些“有意思的函数”（注意并不是整个repo），然后收集它们的context；利用prompted 程序分析生成testing harness，即自动化测试框架。注意到这里并不是简单地生成输入输出，而是包含了一整套外在的config。

**Beyond A\*: Better Planning with Transformers via Search Dynamics Bootstrapping (2402.14083)**

训练一个transformer来预测search trace。测试了两种情况，一种是先输出trace后输出plan，一种是直接输出plan。显然前者的正确率更高。

**FINMEM: LLM trading agent (2311.13743)**

一个llm trading agent，能够处理不同类型、特别是时效性不同的金融数据。它分为profiling，memory和decision-making三个模块。其实profiling大概就是一个prompt，描述了一些过去的信息。然后有不同agent扮演不同风险爱好的决策者。memory模块会结合每天的新闻、股价、公司报告等选出最相关的信息（综合自不同时效），并且根据时效其重要性会指数衰减。同时，agent还会做reflection。decision-making就是简单地从buy/sell/hold里选一个动作。

**Travel Planner: A Benchmark for Real-World Planning with Language Agents**
基本上就是一个新的testbed，没有提出新方法。

**Towards General Computer Control: RDR II (2402.01030)**

整个pipeline大概分为self-reflection，task inference，然后coding得到对应的操作，再根据操作做planning得到最终要做的动作。总的来说就是一种比较高级的以code作为action的react。实际上和elicity一文有点类似。

**Mobile Agent: Autonomous Multi-Modal Mobile Device Agent with Visual Perception**

一个基于GPT-4V和一些辅助工具生成的手机助手task。不过没有测试baseline。

**LLF-Bench: Interactive Learning from Language Feedback(2312.06853)**

一个新的benchmark，在一般agent环境的基础上添加了语言feedback。

**AgentOhana: Design Unified Data and Training Pipeline for Effective Agent Learning**

它把来自不同环境的trajectory综合成一种恒定的格式，从而生成一种新的dataloader。与此同时利用这个dataloader训练了一个新模型xLAM-v0.1。

**Can Large Language Models be Good path planners? (2310.03249)**

测试了一系列LLM在gridworld navigation上的performance。总的来说，LLM的表现并不好——需要situated spatial info和持续不断的反馈。如果是finetune的LLM，则generalizability不太好。相对来说，ReAct效果不错。不过需要注意到这篇文章比较早了，现在可能有更好的方法。

**Human-inspired reading agent with Gist Memory**

基本上就是让llm缩写context之后把缩写结果扔到context里面然后再RAG。需要把一个很长的文章用LLM分段，然后对每一段做缩写。在提问的时候，prompt llm问他需不需要再仔细看某一页的内容。这篇文章里提到了许多context compression的方法。

**The ART of LLM Refinement: Ask, Refine, and Trust**

首先生成一个初始的答案，然后让一个提问题的LLM基于这个问题分解出一系列小问题，如果都能回答上来则直接输出，否则基于这些小问题修正答案。如果修正了答案，那么让一个truster LLM来评价一下是初始答案更好还是后面的答案更好。

**Do LLM Agents have Regret? A case study in online learning and games**

研究了一些简单的bandit环境。有些情况下是无regret的，但是也有很简单的情况会有regret

**If LLM is the Wizard, then code is the wand (2401.00812)**

是一篇survey，提到了code是如何让LLM变得更好的。具体地说，LLM可以直接写code、评价code，可以做program-of-thought，可以辅助CoT做更好的task decomposition，可以建立reasoning graph，辅助vision input，使用工具，从interpreter那里得到反馈，做env的感知和planning，作为action，组织memory或者是自我改进。

**AgentBoard: An analytical evaluation board of Multi-turn LLM  Agents**

A testbed parallel to agentbench

**Is it possible to edit LLM robustly?**

研究是否在finetune改变llm的某个认知之后会被用户几句话给拐回来。事实表明，越是接近基础的认知越不容易被robustly edited，也就是越容易被用户拐回来。


----------------------------------------------------------------------------------------------------------------------------------------

# Chain of Thought

Vanilla: Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. In NeurIPS, 2022.

thought-observation-action: ReAct: Synergizing Reasoning and Acting in Language Models. In ICLR, 2023.

in plan and out-of-plan: AdaPlanner: Adaptive Planning from Feedback with Language Models. In ArXiv, 2023.

long-term memory by self-reflection: Reflexion: Language Agents with Verbal Reinforcement Learning. In ArXiv, 2023.

majority vote: Complexity-Based Prompting for Multi-Step Reasoning. In ICLR, 2023.

Self-consistency: Self-Consistency Improves Chain of Thought Reasoning in Language Models. In ICLR, 2023.

Tree of thought: Tree of Thoughts: Deliberate Problem Solving with Large Language Models. In ArXiv, 2023.

Graph of thought: Graph of Thoughts: Solving Elaborate Problems with Large Language Model. In ArXiv, 2023.

Algorithm-of-Thought: Algorithm of Thoughts: Enhancing Exploration of Ideas in Large Language Models. In ArXiv, 2023.

Skeleton-of-thought: 



Cumulative Reasoning With Large Language Models. In ArXiv, 2023.



# RL for LLM

## Policy Gradient and Actor-Critic

PPO + nucleus sampling to narrow down possible actions: Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimization. In ICLR, 2023.

explore in the vicinity: Semi-Offline Reinforcement Learning for Optimized Text Generation. In ICML, 2023.

## MCTS

Reasoning with Language Model is Planning with World Model. In ArXiv, 2023.

## In-context RL

In-context Reinforcement Learning with Algorithm Distillation. In NeurIPS, 2022.

Transformers as Decision Makers: Provable In-Context Reinforcement Learning via Supervised Pretraining

Emergent agentic transformer from chain of hindsight experience

Emergence of In-Context Reinforcement Learning from Noise Distillation

AMAGO: Scalable In-Context Reinforcement Learning for Adaptive Agents

In-Context Reinforcement Learning for Variable Action Spaces

Supervised Pretraining Can Learn In-Context Reinforcement Learning, in NeurIPS, 2023.

In-context Exploration-Exploitation for Reinforcement Learning