Fix: OpenAI Node SDK Functions Not Working with Streaming

Diagnosis

When using the OpenAI Node SDK with streaming enabled (stream: true), you may encounter a situation where the model correctly decides to call a function (the finish_reason is 'function_call'), but the response chunk contains an empty delta object with no function_call name or arguments. This is a known issue in the openai-node library, particularly in early v4 beta versions. The root cause is that the function call name and arguments are not included in the final chunk where the finish_reason is set; they are delivered in one or more preceding chunks that the developer may not be handling correctly.

What Causes This Error

1. The function call data arrives in earlier chunks, not the final one

The most common cause, confirmed by the GitHub issue discussion, is that the streaming API sends the function name and arguments incrementally across multiple chunks before the chunk that carries the finish_reason: 'function_call'. If your streaming loop only checks the delta object when finish_reason is present, you will miss the function call data entirely. The final chunk with finish_reason contains an empty delta: {}.

2. Using an outdated version of the openai-node library

The issue was originally reported against openai-node 4.0.0-beta.1. Later versions of the library may have improved handling, but the underlying API behavior remains the same: function call data is streamed incrementally. If you are on an old beta, upgrading may resolve some edge cases, but you still need to handle the streaming correctly.

3. Misunderstanding the streaming contract

The OpenAI streaming API for chat completions with functions sends the function name in one chunk and the arguments (as a JSON string) across potentially many chunks. If your code assumes the entire function call arrives in a single chunk, you will see incomplete or missing data.

4. Not accumulating the delta across chunks

If you are not accumulating the delta content across all chunks (including those with empty content but with function_call), you will lose the function name and arguments.

How to Fix It

Solution 1: Accumulate function call data across all chunks (Recommended)

This is the definitive fix, derived from the GitHub issue discussion and the official API behavior. Instead of only looking at the chunk where finish_reason is 'function_call', you must listen for delta.function_call in every chunk and accumulate the name and arguments.

Steps:

1. Initialize variables to hold the function name and arguments before the stream loop.
2. In each chunk, check if delta.function_call exists.
3. If delta.function_call.name is present, set the function name.
4. If delta.function_call.arguments is present, append it to the accumulated arguments string.
5. When finish_reason is 'function_call', use the accumulated name and arguments.

Example code (JavaScript/Node.js):

import OpenAI from 'openai';

const openai = new OpenAI();

async function streamWithFunctions() {
  const stream = await openai.chat.completions.create({
    model: 'gpt-3.5-turbo',
    messages: [
      { role: 'user', content: 'What is the weather in Boston?' }
    ],
    functions: [
      {
        name: 'get_current_weather',
        description: 'Get the current weather in a given location',
        parameters: {
          type: 'object',
          properties: {
            location: {
              type: 'string',
              description: 'The city and state, e.g. San Francisco, CA'
            },
            unit: { type: 'string', enum: ['celsius', 'fahrenheit'] }
          },
          required: ['location']
        }
      }
    ],
    stream: true,
  });

  let functionName = '';
  let functionArguments = '';

  for await (const chunk of stream) {
    const delta = chunk.choices[0]?.delta;

    if (delta?.function_call) {
      if (delta.function_call.name) {
        functionName = delta.function_call.name;
      }
      if (delta.function_call.arguments) {
        functionArguments += delta.function_call.arguments;
      }
    }

    if (chunk.choices[0]?.finish_reason === 'function_call') {
      // Now functionName and functionArguments are complete
      console.log('Function name:', functionName);
      console.log('Function arguments:', functionArguments);
      // Parse arguments and execute the function
      const args = JSON.parse(functionArguments);
      // ... call your function with args
    }
  }
}

streamWithFunctions();

What to expect when it works:

• The first chunk(s) may contain delta.function_call.name with the function name.
• Subsequent chunks will contain delta.function_call.arguments with partial JSON strings (e.g., '{"location":', '"Boston"', '}').
• The final chunk will have finish_reason: 'function_call' and an empty delta: {}.
• After the loop, functionName will be the full function name and functionArguments will be the complete JSON arguments string.

Solution 2: Upgrade to the latest openai-node version

While not a complete fix on its own, upgrading from beta versions (like 4.0.0-beta.1) to the latest stable release may resolve some edge cases where the library itself mishandles the streaming. The official documentation and community reports indicate that the library has matured.

Steps:

1. Check your current version:

   npm list openai
   

2. Upgrade to the latest version:

   npm install openai@latest
   

3. Verify the upgrade:

   npm list openai
   

What to expect:

• The library will still stream function call data incrementally, so you must still implement Solution 1.
• However, newer versions may have better error handling and more consistent chunk delivery.

Solution 3: Use non-streaming as a fallback for function calls

If you cannot get streaming to work reliably with functions, you can fall back to non-streaming requests when you expect a function call. This is a workaround, not a fix, but it is mentioned in the community discussion as a pragmatic approach.

Steps:

1. Make a non-streaming request first to determine if the model wants to call a function.
2. If the response contains a function_call, execute the function and then make a second request (streaming or non-streaming) with the function result.
3. If no function call is needed, proceed with streaming for the content.

Example code (JavaScript/Node.js):

import OpenAI from 'openai';

const openai = new OpenAI();

async function nonStreamingFallback() {
  // First, non-streaming request to check for function call
  const response = await openai.chat.completions.create({
    model: 'gpt-3.5-turbo',
    messages: [
      { role: 'user', content: 'What is the weather in Boston?' }
    ],
    functions: [
      {
        name: 'get_current_weather',
        description: 'Get the current weather in a given location',
        parameters: {
          type: 'object',
          properties: {
            location: { type: 'string' },
            unit: { type: 'string', enum: ['celsius', 'fahrenheit'] }
          },
          required: ['location']
        }
      }
    ],
    stream: false,
  });

  const message = response.choices[0].message;

  if (message.function_call) {
    console.log('Function call detected:', message.function_call);
    // Execute the function and then make a second request
    const functionResult = executeFunction(message.function_call);
    // Now you can stream the final response
    const stream = await openai.chat.completions.create({
      model: 'gpt-3.5-turbo',
      messages: [
        { role: 'user', content: 'What is the weather in Boston?' },
        message,
        { role: 'function', name: message.function_call.name, content: JSON.stringify(functionResult) }
      ],
      stream: true,
    });

    for await (const chunk of stream) {
      process.stdout.write(chunk.choices[0]?.delta?.content || '');
    }
  } else {
    // No function call, proceed with streaming directly
    console.log(message.content);
  }
}

function executeFunction(functionCall) {
  // Your function execution logic here
  return { temperature: 22, unit: 'celsius' };
}

What to expect:

• This approach avoids the streaming function call issue entirely.
• It adds an extra round trip, which increases latency.
• It is best used when you know the model will likely call a function.

Solution 4: Check for function_call in every chunk, not just when finish_reason is set

This is a more defensive version of Solution 1. Some developers have reported that the function_call data may appear in chunks that do not have a finish_reason at all. Your code should always check for delta.function_call regardless of the presence of finish_reason.

Steps:

1. In your streaming loop, always check for delta.function_call.
2. Accumulate the name and arguments as shown in Solution 1.
3. Do not assume that the function call data is complete until you see finish_reason: 'function_call'.

Example code (JavaScript/Node.js):

import OpenAI from 'openai';

const openai = new OpenAI();

async function defensiveStream() {
  const stream = await openai.chat.completions.create({
    model: 'gpt-3.5-turbo',
    messages: [
      { role: 'user', content: 'What is the weather in Boston?' }
    ],
    functions: [
      {
        name: 'get_current_weather',
        description: 'Get the current weather in a given location',
        parameters: {
          type: 'object',
          properties: {
            location: { type: 'string' },
            unit: { type: 'string', enum: ['celsius', 'fahrenheit'] }
          },
          required: ['location']
        }
      }
    ],
    stream: true,
  });

  let functionName = '';
  let functionArguments = '';
  let isFunctionCall = false;

  for await (const chunk of stream) {
    const choice = chunk.choices[0];
    if (!choice) continue;

    const delta = choice.delta;

    // Always check for function_call data
    if (delta?.function_call) {
      isFunctionCall = true;
      if (delta.function_call.name) {
        functionName = delta.function_call.name;
      }
      if (delta.function_call.arguments) {
        functionArguments += delta.function_call.arguments;
      }
    }

    // Handle regular content
    if (delta?.content) {
      process.stdout.write(delta.content);
    }

    // When finish_reason is function_call, we have all data
    if (choice.finish_reason === 'function_call') {
      console.log('\nFunction call complete:');
      console.log('Name:', functionName);
      console.log('Arguments:', functionArguments);
      // Parse and execute
      const args = JSON.parse(functionArguments);
      // ...
    }
  }

  if (isFunctionCall && !functionName) {
    console.error('Warning: finish_reason was function_call but no name was accumulated.');
  }
}

What to expect:

• This code will correctly capture function call data even if it arrives in chunks without finish_reason.
• It also handles the edge case where finish_reason is 'function_call' but no data was accumulated (which would indicate a different bug).