老葛
64b94d79f9
* feat(xhs): 集成xhshow库优化签名生成与请求参数 - 引入xhshow库用于小红书API签名生成 - 替换原有的seccore_signv2_playwright签名校验方式 - 支持GET和POST请求的差异化签名处理 - 增加对b1值从localStorage获取的容错处理 - 更新x-t时间戳为毫秒级精度 - 在获取博主笔记接口中增加xsec_token和xsec_source参数- 支持通过配置传递验证token和渠道来源 - 更新依赖文件引入xhshow库- 调整配置示例适配新的token参数要求 * Delete MindSpider/DeepSentimentCrawling/MediaCrawler/config/xhs_config.py 移除配置文件 * Add xhs_config.py for Xiaohongshu platform settings 恢复错误删除的文件 --------- Co-authored-by: gehongbin <gehongbin@autohome.com.cn> Co-authored-by: Doiiars <doiiars@qq.com>
60 lines
1.0 KiB
Plaintext
60 lines
1.0 KiB
Plaintext
# MindSpider AI爬虫项目依赖包
|
|
# 整合BroadTopicExtraction和DeepSentimentCrawling模块的核心依赖
|
|
|
|
# ===============================
|
|
# 数据库相关
|
|
# ===============================
|
|
pymysql==1.1.0
|
|
aiomysql==0.2.0
|
|
aiosqlite==0.21.0
|
|
asyncpg
|
|
sqlalchemy
|
|
psycopg[binary]
|
|
|
|
# ===============================
|
|
# HTTP请求和网络
|
|
# ===============================
|
|
httpx==0.28.1
|
|
requests==2.32.3
|
|
aiofiles~=23.2.1
|
|
|
|
# ===============================
|
|
# 话题提取模块依赖
|
|
# ===============================
|
|
openai
|
|
pydantic-settings
|
|
numpy
|
|
pandas==2.2.3
|
|
regex
|
|
tqdm
|
|
python-dateutil
|
|
pytz
|
|
|
|
# ===============================
|
|
# MediaCrawler爬虫依赖
|
|
# ===============================
|
|
Pillow>=10.0.0
|
|
playwright==1.45.0
|
|
tenacity==8.2.2
|
|
opencv-python
|
|
redis~=4.6.0
|
|
pydantic==2.5.2
|
|
fastapi==0.110.2
|
|
uvicorn==0.29.0
|
|
python-dotenv==1.0.1
|
|
jieba==0.42.1
|
|
wordcloud==1.9.3
|
|
matplotlib==3.9.0
|
|
parsel==1.9.1
|
|
pyexecjs==1.5.1
|
|
typer>=0.12.3
|
|
pyhumps==3.8.0
|
|
xhshow>=0.1.3
|
|
|
|
# ===============================
|
|
# 工具包
|
|
# ===============================
|
|
beautifulsoup4
|
|
lxml
|
|
loguru
|