流式传输（Streaming）

一些 LLM 提供流式响应。这意味着您可以在整个响应返回之前开始处理它，而不是等待它完全返回。如果您希望在生成响应时向用户显示响应，或者希望在生成响应时处理响应，这将非常有用。

目前，我们支持对 OpenAI、ChatOpenAI 和 ChatAnthropic 实现的流式传输。要使用流式传输，请使用一个实现了 on_llm_new_token 的 CallbackHandler。在这个示例中，我们使用的是 StreamingStdOutCallbackHandler。

from langchain.llms import OpenAI
from langchain.callbacks.streaming_stdout import StreamingStdOutCallbackHandler


llm = OpenAI(streaming=True, callbacks=[StreamingStdOutCallbackHandler()], temperature=0)
resp = llm("Write me a song about sparkling water.")

    Verse 1
    I'm sippin' on sparkling water,
    It's so refreshing and light,
    It's the perfect way to quench my thirst
    On a hot summer night.
    
    Chorus
    Sparkling water, sparkling water,
    It's the best way to stay hydrated,
    It's so crisp and so clean,
    It's the perfect way to stay refreshed.
    
    Verse 2
    I'm sippin' on sparkling water,
    It's so bubbly and bright,
    It's the perfect way to cool me down
    On a hot summer night.
    
    Chorus
    Sparkling water, sparkling water,
    It's the best way to stay hydrated,
    It's so crisp and so clean,
    It's the perfect way to stay refreshed.
    
    Verse 3
    I'm sippin' on sparkling water,
    It's so light and so clear,
    It's the perfect way to keep me cool
    On a hot summer night.
    
    Chorus
    Sparkling water, sparkling water,
    It's the best way to stay hydrated,
    It's so crisp and so clean,
    It's the perfect way to stay refreshed.

如果使用 generate，我们仍然可以访问最终的 LLMResult。但是，目前不支持对流式传输的 token_usage。

llm.generate(["Tell me a joke."])

    Q: What did the fish say when it hit the wall?
    A: Dam!


    LLMResult(generations=[[Generation(text='\n\nQ: What did the fish say when it hit the wall?\nA: Dam!', generation_info={'finish_reason': 'stop', 'logprobs': None})]], llm_output={'token_usage': {}, 'model_name': 'text-davinci-003'})