使用langchain与你自己的数据对话
输入“/”快速插入内容
使用langchain与你自己的数据对话
使用langchain与你自己的数据对话
飞书用户68482023年9月4日修改
https://blog.csdn.net/weixin_42608414/article/details/131760937?spm=1001.2014.3001.5502
核心点：
Loader加载器：PyPDFLoader，YoutubeAudioLoader，WebBaseLoader，DirectoryLoader
文档切割器： RecursiveCharacterTextSplitter, CharacterTextSplitter，TokenTextSplitter，MarkdownHeaderTextSplitter等
https://blog.csdn.net/weixin_42608414/article/details/131895512?spm=1001.2014.3001.5502
核心点：
Chroma向量数据库的持久化
代码块
#创建向量数据库
vectordb = Chroma.from_documents(
documents=splits,
embedding=embedding,
persist_directory=persist_directory
)
vectordb.persist()
FAISS向量数据库的持久化
代码块
from langchain.vectorstores.faiss import FAISS
import pickle
vectorstore = FAISS.from_documents(documents, embeddings)
with open("vectorstore.pkl", "wb") as f:
pickle.dump(vectorstore, f)
加载向量存储，并将其放入内存中
代码块
with open("vectorstore.pkl", "rb") as f:
vectorstore = pickle.load(f)
https://blog.csdn.net/weixin_42608414/article/details/131937752?spm=1001.2014.3001.5502
核心点：检索(Retrieval) 中我们介绍了基本语义相似度(Basic semantic similarity)，最大边际相关性(Maximum marginal relevance，MMR), 过滤元数据, LLM辅助检索
https://blog.csdn.net/weixin_42608414/article/details/132039304?spm=1001.2014.3001.5502
核心点: 介绍了几种RetrievalQA检索向量数据库的工作方式，也就是chain type方式，其实默认方式是stuff，除此之外还有map_reduce，refine, map_rerank等几种方式
使用提词
代码块
from langchain.prompts import PromptTemplate
# Build prompt
template = """Use the following pieces of context to answer the question at the end. \
If you don't know the answer, just say that you don't know, don't try to make up an answer. \
Use three sentences maximum. Keep the answer as concise as possible. Always say "thanks for asking!" \
at the end of the answer.
{context}
Question: {question}
Helpful Answer:"""
QA_CHAIN_PROMPT = PromptTemplate.from_template(template)
# Run chain
qa_chain = RetrievalQA.from_chain_type(
llm,
retriever=vectordb.as_retriever(),
return_source_documents=True,
chain_type_kwargs={"prompt": QA_CHAIN_PROMPT}
)
question = "What are major topics for this class?"
result = qa_chain({"query": question})
result["result"]