Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Help me: 项目启动报错 #22

Closed
tanpenggood opened this issue Jul 30, 2023 · 11 comments
Closed

Help me: 项目启动报错 #22

tanpenggood opened this issue Jul 30, 2023 · 11 comments
Labels
help wanted Extra attention is needed

Comments

@tanpenggood
Copy link
Contributor

software version
OS mac
python 3.11
$ python main.py --platform xhs --lt qrcode
Traceback (most recent call last):
  File "/Users/xxx/tp-code/MediaCrawler/main.py", line 8, in <module>
    from media_platform.douyin import DouYinCrawler
  File "/Users/xxx/tp-code/MediaCrawler/media_platform/douyin/__init__.py", line 1, in <module>
    from .core import DouYinCrawler
  File "/Users/xxx/tp-code/MediaCrawler/media_platform/douyin/core.py", line 17, in <module>
    from .login import DouYinLogin
  File "/Users/xxx/tp-code/MediaCrawler/media_platform/douyin/login.py", line 6, in <module>
    import aioredis
  File "/Users/xxx/miniconda3/lib/python3.11/site-packages/aioredis/__init__.py", line 1, in <module>
    from aioredis.client import Redis, StrictRedis
  File "/Users/xxx/miniconda3/lib/python3.11/site-packages/aioredis/client.py", line 32, in <module>
    from aioredis.connection import (
  File "/Users/xxx/miniconda3/lib/python3.11/site-packages/aioredis/connection.py", line 33, in <module>
    from .exceptions import (
  File "/Users/xxx/miniconda3/lib/python3.11/site-packages/aioredis/exceptions.py", line 14, in <module>
    class TimeoutError(asyncio.TimeoutError, builtins.TimeoutError, RedisError):
TypeError: duplicate base class TimeoutError
@NanmiCoder
Copy link
Owner

看样子是redis连接超时了,可以检查你的redis是否设置密码了以及密码是否正确。

@NanmiCoder NanmiCoder added the help wanted Extra attention is needed label Jul 30, 2023
@tanpenggood
Copy link
Contributor Author

tanpenggood commented Jul 30, 2023

我在本地6379端口启动的redis,没有设置密码,配置如下:

config/db_config.py

# redis config
REDIS_DB_HOST = "redis://127.0.0.1"  # your redis host
REDIS_DB_PWD = ""  # your redis password

# mysql config
RELATION_DB_URL = "mysql://root:youdbpassword@localhost:3306/media_crawler"

# save data to database option
IS_SAVED_DATABASED = False  # if you want to save data to database, set True

@NanmiCoder
Copy link
Owner

浅看了一下aioredis中Client视乎对密码是必填的,意味着你需要先设置下 Redis Server 配置文件的密码,再在db_config.py中配置密码,不能填空字符。

@tanpenggood
Copy link
Contributor Author

我给redis设置了密码 123456,
再次python main.py --platform xhs --lt qrcode
仍然是上面的错误。

@NanmiCoder
Copy link
Owner

找到原因了,这是一个aioredis的bug,对python3.11版本不支持了,稍后我会用redis模块来替换aioredis,aio-libs-abandoned/aioredis-py#1409

@tanpenggood
Copy link
Contributor Author

太棒了! 我拉下来,因为这个问题好几天都没启动😂

@tanpenggood
Copy link
Contributor Author

刚刚还特意用redis-test.py,做连接redis的测试,发现报错同样和上面一样。

import config
import aioredis

if __name__ == '__main__':
    redis_obj = aioredis.from_url(url=config.REDIS_DB_HOST, password=config.REDIS_DB_PWD, decode_responses=True)
    value = redis_obj.get("hello")
    print(value)

这个坎儿算是过去了,我还以为电脑出问题了😆。

NanmiCoder added a commit that referenced this issue Jul 30, 2023
@NanmiCoder
Copy link
Owner

该问题已经修复,已将aioredis替换为官方的redis实现,麻烦你拉下最新代码跑跑看,能否启动起来。

@tanpenggood
Copy link
Contributor Author

OK!

software version
OS mac
python 3.11
$ pip install -r requirements.txt

...

$ python main.py --platform xhs --lt qrcode
2023-07-30  21:03:41 MediaCrawler INFO Begin create browser context ... 
2023-07-30  21:03:49 MediaCrawler INFO Begin create xiaohongshu API client ... 
2023-07-30  21:03:49 MediaCrawler INFO begin to ping xhs... 
2023-07-30  21:03:50 MediaCrawler INFO Begin search xiaohongshu keywords 
2023-07-30  21:03:50 MediaCrawler INFO Current keyword: 健身 
...

@tanpenggood
Copy link
Contributor Author

LOGIN_TYPE = 'cookie'时, COOKIES我应该怎么设置呢?

@NanmiCoder
Copy link
Owner

LOGIN_TYPE = 'cookie'时, COOKIES我应该怎么设置呢?

LOGIN_TYPE = "qrcode" # qrcode or phone or cookie
COOKIES = "" # login by cookie, if login_type is cookie, you must set this value

bananea20 pushed a commit to bananea20/MediaCrawler that referenced this issue Nov 15, 2023
ly1327 pushed a commit to ly1327/MediaCrawler that referenced this issue Mar 14, 2024
commit b48e4d2
Author: Relakkes <[email protected]>
Date:   Tue Mar 12 23:09:45 2024 +0800

    docs: 将python3命令修改为python

commit 9e2d139
Author: Relakkes <[email protected]>
Date:   Sun Mar 10 03:49:58 2024 +0800

    docs: 增加视频入门教程链接

commit 9ca0a2b
Author: Relakkes <[email protected]>
Date:   Sat Mar 9 15:37:51 2024 +0800

    docs: update README.md

commit 41fee4f
Author: Relakkes <[email protected]>
Date:   Thu Mar 7 22:30:44 2024 +0800

    feat:小红书支持获取评论中的图片链接 NanmiCoder#145

commit 8610190
Author: Relakkes <[email protected]>
Date:   Tue Mar 5 22:45:32 2024 +0800

    docs: 增加捐赠者名单

commit 7606c1d
Author: Relakkes <[email protected]>
Date:   Tue Mar 5 00:03:51 2024 +0800

    docs: 增加捐赠者名单

commit 149b6bc
Author: Relakkes <[email protected]>
Date:   Sun Mar 3 19:36:36 2024 +0800

    fix: 修复抖音关键词搜索为中文的情况下,有bug

commit 7002061
Merge: 384c8f9 2b1ad18
Author: relakkes <[email protected]>
Date:   Sat Mar 2 23:30:31 2024 +0800

    Merge pull request NanmiCoder#142 from jayeeliu/main

    小红书支持通过博主ID采集笔记和评论

commit 2b1ad18
Author: [email protected] <jayeeliu>
Date:   Sat Mar 2 23:05:34 2024 +0800

    Revert "feat: add aerich for migrate db"

    This reverts commit 5f7cd71.

commit 5f7cd71
Author: [email protected] <jayeeliu>
Date:   Sat Mar 2 15:58:35 2024 +0800

    feat: add aerich for migrate db

commit 9a5460c
Merge: 61ba8c5 384c8f9
Author: jayeeliu <[email protected]>
Date:   Sat Mar 2 01:51:02 2024 +0800

    Merge branch 'NanmiCoder:main' into main

commit 61ba8c5
Author: [email protected] <jayeeliu>
Date:   Sat Mar 2 01:49:42 2024 +0800

    feat: 小红书支持通过博主ID采集笔记和评论,小红书type=search时支持配置按哪种排序方式获取笔记数据,小红书笔记增加视频地址和标签字段

commit 384c8f9
Author: Relakkes <[email protected]>
Date:   Mon Feb 26 23:47:02 2024 +0800

    fix: issue NanmiCoder#140

commit 67d2b7c
Merge: 03ca445 c09f9fe
Author: relakkes <[email protected]>
Date:   Thu Feb 22 21:32:19 2024 +0800

    Merge pull request NanmiCoder#136 from jayeeliu/main

    增加sqlite配置示例,解决用sqlite保存数据时,抓取结束不退出的问题

commit c09f9fe
Author: ccgo <>
Date:   Thu Feb 22 00:11:41 2024 +0800

    fix: 增加db.close(),解决抓取命令执行完不退出的问题

commit 4f5f83d
Author: ccgo <>
Date:   Thu Feb 22 00:07:40 2024 +0800

    feat: 增加sqlite配置示例

commit 03ca445
Author: Relakkes <[email protected]>
Date:   Sun Feb 18 13:06:12 2024 +0800

    docs: update README.md

commit 192e725
Author: Relakkes <[email protected]>
Date:   Fri Jan 26 23:42:26 2024 +0800

    fix: NanmiCoder#123

commit f1d3bc6
Merge: eb89a6a 79c0f3b
Author: relakkes <[email protected]>
Date:   Thu Jan 25 14:33:09 2024 +0800

    Merge pull request NanmiCoder#124 from aa65535/patch-1

    修复小红书评论重复插入

commit 79c0f3b
Author: Jian Chang <[email protected]>
Date:   Thu Jan 25 13:01:04 2024 +0800

    修复小红书评论重复插入

commit eb89a6a
Author: relakkes <[email protected]>
Date:   Sun Jan 21 16:42:37 2024 +0800

    docs: 添加赞赏者信息

commit e940a41
Author: Relakkes <[email protected]>
Date:   Wed Jan 17 23:02:05 2024 +0800

    refactor: 移除评论中指定数量和过滤特定关键词的逻辑

commit e0f9a48
Author: Relakkes <[email protected]>
Date:   Tue Jan 16 00:40:07 2024 +0800

    refactor: 代码优化

commit e490123
Merge: e31aebb 4dfa0d3
Author: relakkes <[email protected]>
Date:   Sun Jan 14 22:45:00 2024 +0800

    Merge pull request NanmiCoder#112 from NanmiCoder/feature/stored_in_json

    refactor: 重构数据存储代码,新增jSON

commit 4dfa0d3
Author: Relakkes <[email protected]>
Date:   Sun Jan 14 22:40:01 2024 +0800

    feat: 数据保存支持JSON格式

commit 894dabc
Author: Relakkes <[email protected]>
Date:   Sun Jan 14 22:06:31 2024 +0800

    refactor: 数据存储重构,分离不同类型的存储实现

commit e31aebb
Author: Relakkes <[email protected]>
Date:   Sat Jan 13 15:50:02 2024 +0800

    fix: 修复代理Bug

commit 0240de7
Author: Relakkes <[email protected]>
Date:   Wed Jan 10 23:59:14 2024 +0800

    docs: 添加捐赠信息

commit f85bd0c
Author: Relakkes <[email protected]>
Date:   Sun Jan 7 18:33:08 2024 +0800

    docs: 添加捐赠信息

commit 4de14ad
Author: Relakkes <[email protected]>
Date:   Sat Jan 6 19:18:07 2024 +0800

    fix: 修复微博PC端登录后COOKIE在手机端无法使用的bug

commit fe07380
Author: Relakkes <[email protected]>
Date:   Thu Jan 4 00:11:05 2024 +0800

    feat: 小红书增加获取图片地址的方法

commit 80ecc53
Author: Relakkes <[email protected]>
Date:   Wed Jan 3 23:08:30 2024 +0800

    fix: NanmiCoder#107

commit 38d6f10
Author: Relakkes <[email protected]>
Date:   Sat Dec 30 18:54:21 2023 +0800

    feat: 微博二维码登录done

commit 27a2041
Author: Relakkes <[email protected]>
Date:   Sat Dec 30 15:16:33 2023 +0800

    doc: update README.md

commit eee8162
Author: Relakkes <[email protected]>
Date:   Mon Dec 25 00:02:11 2023 +0800

    feat: 微博支持评论 & 指定帖子

commit b1441ab
Author: Relakkes <[email protected]>
Date:   Sun Dec 24 18:19:26 2023 +0800

    feat: 微博帖子支持保存到数据库中

commit c5b64fd
Author: Relakkes <[email protected]>
Date:   Sun Dec 24 17:57:48 2023 +0800

    feat: 微博爬虫帖子搜索完成

commit 9785abb
Author: Relakkes <[email protected]>
Date:   Sat Dec 23 01:41:19 2023 +0800

    docs: update README.md

commit aba9f14
Author: Relakkes <[email protected]>
Date:   Sat Dec 23 01:04:08 2023 +0800

    refactor: 规范日志打印
    feat: B站指定视频ID爬取(bvid)

commit 273c9a3
Author: Relakkes <[email protected]>
Date:   Fri Dec 22 23:10:44 2023 +0800

    fix: 修复日志打印时参数格式错误

commit bced7f6
Author: Relakkes <[email protected]>
Date:   Fri Dec 22 23:04:08 2023 +0800

    doc: update README.md

commit 2e032d0
Author: Relakkes <[email protected]>
Date:   Tue Dec 19 23:05:12 2023 +0800

    refactor: xhs add log

commit e2aaf0e
Author: Relakkes <[email protected]>
Date:   Sun Dec 17 18:57:43 2023 +0800

    doc: update README.md

commit f85fd97
Author: Relakkes <[email protected]>
Date:   Sat Dec 16 00:12:20 2023 +0800

    fix: NanmiCoder#10

commit 40cbd9f
Merge: c4d68f8 5c6a636
Author: relakkes <[email protected]>
Date:   Thu Dec 14 00:57:17 2023 +0800

    Merge pull request NanmiCoder#96 from PeanutSplash/main

    fix-bug:修复抖音评论筛选

commit 5c6a636
Author: peanutsplash <[email protected]>
Date:   Thu Dec 14 00:55:06 2023 +0800

    fix-bug:修复抖音评论筛选

commit c4d68f8
Merge: 5c42076 f17a853
Author: relakkes <[email protected]>
Date:   Thu Dec 14 00:01:44 2023 +0800

    Merge pull request NanmiCoder#94 from PeanutSplash/main

    添加功能:(哔哩哔哩,快手,小红书)每个视频/帖子抓取评论最大条数限制,评论关键词筛选

commit f17a853
Author: peanutsplash <[email protected]>
Date:   Wed Dec 13 23:53:12 2023 +0800

    添加功能:(哔哩哔哩,快手,小红书)每个视频/帖子抓取评论最大条数限制,评论关键词筛选

commit 5c42076
Author: Relakkes <[email protected]>
Date:   Sat Dec 9 23:50:33 2023 +0800

    doc: update README.md

commit 0501630
Author: Relakkes <[email protected]>
Date:   Sat Dec 9 23:49:19 2023 +0800

    doc: update README.md

commit 97d7a0c
Author: Relakkes <[email protected]>
Date:   Sat Dec 9 21:10:01 2023 +0800

    feat: Bilibili comment done

commit a136ea2
Author: Relakkes <[email protected]>
Date:   Sat Dec 9 13:56:18 2023 +0800

    refactor: delete account_config.py

commit 2d0f12f
Author: Relakkes <[email protected]>
Date:   Sat Dec 9 13:31:53 2023 +0800

    doc: fixed pydantic version

commit fd640f2
Author: Relakkes <[email protected]>
Date:   Sat Dec 9 00:30:31 2023 +0800

    doc: update README.md

commit d37d5cc
Author: Relakkes <[email protected]>
Date:   Sat Dec 9 00:28:29 2023 +0800

    doc: add SPONSORED BY text

commit 385662a
Author: Relakkes <[email protected]>
Date:   Sat Dec 9 00:06:57 2023 +0800

    doc: 新增代理IP使用文档

commit a40a91d
Author: Relakkes <[email protected]>
Date:   Fri Dec 8 00:27:59 2023 +0800

    doc: update README.md

commit 72a68fb
Author: Relakkes <[email protected]>
Date:   Fri Dec 8 00:25:06 2023 +0800

    doc:将文档拆出去

commit 1cec23f
Author: Relakkes <[email protected]>
Date:   Fri Dec 8 00:10:04 2023 +0800

    feat: 代理IP功能 Done

commit c530bd4
Author: Relakkes <[email protected]>
Date:   Wed Dec 6 23:49:56 2023 +0800

    feat: 代理IP缓存到redis中

commit f71d086
Author: Relakkes <[email protected]>
Date:   Tue Dec 5 23:32:35 2023 +0800

    fix: B站get_wbi_keys函数类型标注问题

commit a6e877d
Author: Relakkes <[email protected]>
Date:   Tue Dec 5 22:47:36 2023 +0800

    fix: 修复B站搜索Field命名 bug
    refactor: ping接口统一更换为pong

commit 6ae511b
Merge: 8f04943 ab1a10b
Author: relakkes <[email protected]>
Date:   Tue Dec 5 12:42:32 2023 +0800

    Merge pull request NanmiCoder#89 from PeanutSplash/main

    添加功能:抖音每个视频抓取评论最大条数限制,抖音评论关键词筛选

commit ab1a10b
Author: peanutsplash <[email protected]>
Date:   Tue Dec 5 11:21:47 2023 +0800

    添加功能:抖音每个视频抓取评论最大条数限制,抖音评论关键词筛选

commit 8f04943
Author: Relakkes <[email protected]>
Date:   Mon Dec 4 23:16:02 2023 +0800

    feat: B站评论API

commit a790d72
Author: Relakkes <[email protected]>
Date:   Mon Dec 4 21:54:12 2023 +0800

    fix: 修复小红书笔记获取时,不存在的笔记异常问题

commit 519165c
Author: Relakkes <[email protected]>
Date:   Mon Dec 4 00:11:01 2023 +0800

    doc: 修改项目代码结构tree

commit 94b5030
Author: Relakkes <[email protected]>
Date:   Mon Dec 4 00:02:00 2023 +0800

    feat: B站二维码、Cookie登录实现

commit a90b411
Author: Relakkes <[email protected]>
Date:   Sun Dec 3 23:19:02 2023 +0800

    feat: B站爬虫搜索关键词实现

commit 5aeee93
Author: Relakkes <[email protected]>
Date:   Sun Dec 3 00:30:10 2023 +0800

    feat: B站爬虫签名实现

commit 5c920da
Author: Relakkes <[email protected]>
Date:   Sat Dec 2 18:22:55 2023 +0800

    feat: 快手二维码登录

commit 986179b
Author: Relakkes <[email protected]>
Date:   Sat Dec 2 16:14:36 2023 +0800

    feat: 增加 IP 代理的最新实现

commit a8a4d34
Author: Relakkes <[email protected]>
Date:   Sat Dec 2 10:56:13 2023 +0800

    doc: update README.md

commit e7f68dd
Author: Relakkes <[email protected]>
Date:   Fri Dec 1 23:58:35 2023 +0800

    doc: 提供仓库功能列表表格

commit 5affc8a
Author: relakkes <[email protected]>
Date:   Fri Dec 1 09:55:38 2023 +0800

    Update README.md

commit 6d74355
Author: Relakkes <[email protected]>
Date:   Thu Nov 30 23:59:02 2023 +0800

    doc: update README.md

commit 33721e5
Author: Relakkes <[email protected]>
Date:   Mon Nov 27 23:07:04 2023 +0800

    feat: 快手支持指定视频列表爬取

commit 62534d7
Author: Relakkes <[email protected]>
Date:   Sun Nov 26 22:11:06 2023 +0800

    fix: 移出快手 client 多余的代码

commit 609c3f0
Author: Relakkes <[email protected]>
Date:   Sun Nov 26 22:06:26 2023 +0800

    fix: 修复抖音数据库保存错误

commit d591136
Author: Relakkes <[email protected]>
Date:   Sun Nov 26 21:50:08 2023 +0800

    doc: 增加快手爬虫的描述

commit 9faa4f4
Merge: 523c5f3 dfb1788
Author: relakkes <[email protected]>
Date:   Sun Nov 26 21:44:36 2023 +0800

    Merge pull request NanmiCoder#82 from NanmiCoder/feat/kuaishou_support

    快手视频、评论爬取支持

commit dfb1788
Author: Relakkes <[email protected]>
Date:   Sun Nov 26 21:43:39 2023 +0800

    feat: 快手视频评论爬取done;数据保存到DB、CSV done

commit 2f8541a
Merge: bdf36cc 523c5f3
Author: Relakkes <[email protected]>
Date:   Sun Nov 26 15:44:26 2023 +0800

    Merge branch 'main' into feat/kuaishou_support

commit 523c5f3
Author: Relakkes <[email protected]>
Date:   Sun Nov 26 15:38:38 2023 +0800

    doc: add command usage

commit bdf36cc
Author: Relakkes <[email protected]>
Date:   Sun Nov 26 01:05:52 2023 +0800

    feat: 快手关键词搜索存储CSV完成

commit 2538889
Author: Relakkes <[email protected]>
Date:   Sat Nov 25 12:32:43 2023 +0800

    doc: 增加如何更换账号的问题

commit 512192a
Author: Relakkes <[email protected]>
Date:   Sat Nov 25 00:02:33 2023 +0800

    feat: 搜索接口调试完成

commit f08b2ce
Author: Relakkes <[email protected]>
Date:   Fri Nov 24 00:04:33 2023 +0800

    feat: 1、命令行支持快手 2、快手playwright 代码 done

commit 6908da6
Author: Relakkes <[email protected]>
Date:   Thu Nov 23 23:27:35 2023 +0800

    fix: 修复小红书client中get请求中文bug

commit 95ca606
Author: Relakkes <[email protected]>
Date:   Thu Nov 23 23:13:54 2023 +0800

    feat: 快手文件目录建立

commit 3790e80
Author: Relakkes <[email protected]>
Date:   Mon Nov 20 22:07:00 2023 +0800

    doc: update README.md

commit 81bc8b5
Author: Relakkes <[email protected]>
Date:   Sat Nov 18 22:07:30 2023 +0800

    feat: 抖音支持指定视频列表爬去

commit 098923d
Author: Relakkes <[email protected]>
Date:   Sat Nov 18 15:53:10 2023 +0800

    refactor: 优化代码-变量名

commit ecf9a5e
Merge: 700946b 4c85355
Author: relakkes <[email protected]>
Date:   Sat Nov 18 15:05:16 2023 +0800

    Merge pull request NanmiCoder#74 from Akiqqqqqqq/bugfix-null-key

    fix null key

commit 4c85355
Merge: 602e609 700946b
Author: leonardoqiuyu <[email protected]>
Date:   Sat Nov 18 14:59:03 2023 +0800

    Merge branch 'main' into bugfix-null-key

commit 700946b
Author: Relakkes <[email protected]>
Date:   Sat Nov 18 13:38:11 2023 +0800

    feat: 小红书增加指定帖子爬取功能
    fix: 修复程序一些异常 bug
    refactor: 优化部分代码逻辑

commit f24c892
Author: Relakkes <[email protected]>
Date:   Sat Nov 18 11:18:54 2023 +0800

    fix: 修复评论为空导致程序异常退出的 bug

commit 602e609
Author: Akiqqqqqqq <[email protected]>
Date:   Tue Nov 14 21:11:18 2023 +0800

    fix null key

commit cd3a0bd
Author: Relakkes <[email protected]>
Date:   Sat Nov 11 20:49:32 2023 +0800

    doc: update README.md

commit 68c7139
Merge: 7698c41 d0ded66
Author: relakkes <[email protected]>
Date:   Sun Nov 5 15:28:10 2023 +0800

    Merge pull request NanmiCoder#64 from NanmiCoder/feat/article_url

    feat: add article url for issue NanmiCoder#63

commit d0ded66
Author: Relakkes <[email protected]>
Date:   Sun Nov 5 15:27:18 2023 +0800

    feat: add article url for issue NanmiCoder#63

commit 7698c41
Merge: b7fe5b7 ad69e63
Author: relakkes <[email protected]>
Date:   Thu Aug 31 08:09:06 2023 -0500

    Merge pull request NanmiCoder#40 from yangrq1018/pr

    fix: get phone from phone pool when ENABLE_IP_PROXY is False

commit ad69e63
Author: xiaoniu <[email protected]>
Date:   Thu Aug 31 13:57:54 2023 +0800

    fix: get phone from phone pool when ENABLE_IP_PROXY is False

commit b7fe5b7
Author: Relakkes <[email protected]>
Date:   Wed Aug 16 19:56:12 2023 +0800

    style: Change the import order in the code.

commit 9177c38
Author: Relakkes <[email protected]>
Date:   Wed Aug 16 19:49:41 2023 +0800

    feat: 支持数据保存到CSV中

commit c1a3f06
Author: Relakkes <[email protected]>
Date:   Wed Aug 16 13:58:44 2023 +0800

    fix: issue NanmiCoder#32

commit 99812b4
Merge: 3ae4c5f 1556e1b
Author: Relakkes <[email protected]>
Date:   Mon Aug 14 09:32:48 2023 +0800

    Merge pull request NanmiCoder#32 from renaissancezyc/main

    orm添加Pydantic数据验证

commit 1556e1b
Author: zhoubang <[email protected]>
Date:   Sun Aug 13 22:39:50 2023 +0800

    ✨ feat: orm添加Pydantic数据验证

commit 3ae4c5f
Author: Relakkes <[email protected]>
Date:   Thu Aug 3 22:26:31 2023 +0800

    fix: issue NanmiCoder#23

commit defd823
Merge: 1d3c235 925f0a1
Author: Relakkes <[email protected]>
Date:   Tue Aug 1 13:35:00 2023 +0800

    Merge pull request NanmiCoder#24 from jayeeliu/main

    去掉搜索结果列表中的推荐搜索词(mode_type in ['rec_query', 'hot_query'])

commit 925f0a1
Author: JayeeLiu <[email protected]>
Date:   Tue Aug 1 10:29:00 2023 +0800

    去掉搜索结果列表中的推荐搜索词(mode_type in ['rec_query', 'hot_query'])

commit 1d3c235
Author: Relakkes <[email protected]>
Date:   Sun Jul 30 21:30:26 2023 +0800

    fix: 修复部分变量命名语义不明确

commit bf65945
Author: Relakkes <[email protected]>
Date:   Sun Jul 30 20:43:02 2023 +0800

    fix: issue NanmiCoder#22

commit 4ff2cf8
Author: Relakkes <[email protected]>
Date:   Sat Jul 29 15:35:40 2023 +0800

    refactor: 优化代码

commit febbb13
Author: Relakkes <[email protected]>
Date:   Fri Jul 28 21:23:37 2023 +0800

    fix: douyin 缺少一个collected_count字段

commit 03565d6
Author: Relakkes <[email protected]>
Date:   Tue Jul 25 20:22:22 2023 +0800

    fix: issue NanmiCoder#19

commit 7d63c2f
Merge: e757074 b94924e
Author: Relakkes <[email protected]>
Date:   Mon Jul 24 21:00:00 2023 +0800

    Merge branch 'main' of github.com:NanmiCoder/MediaCrawler

commit e757074
Author: Relakkes <[email protected]>
Date:   Mon Jul 24 20:59:43 2023 +0800

    feat: 增加配置项支持自由选择数据是否保存到关系型数据库中

commit b94924e
Merge: 745e59c ae2de2f
Author: Relakkes <[email protected]>
Date:   Sun Jul 23 14:30:19 2023 +0800

    Merge pull request NanmiCoder#18 from tanpenggood-fork/typo-sam-20230723

    typo(field.py): words "normal" typo

commit ae2de2f
Author: tanpenggood <[email protected]>
Date:   Sun Jul 23 13:24:39 2023 +0800

    typo(field.py): words "normal" typo

commit 745e59c
Author: Nanmi <[email protected]>
Date:   Sun Jul 16 17:57:18 2023 +0800

    feat: 完善类型注释,增加 mypy 类型检测

commit e5bdc63
Author: Relakkes <[email protected]>
Date:   Sat Jul 15 22:25:56 2023 +0800

    feat: xhs增加并发控制参数

commit 2398a17
Author: Relakkes <[email protected]>
Date:   Sat Jul 15 21:30:12 2023 +0800

    refactor: 优化抖音Crawler部分代码
    fix: 日志初始化错误修复

commit dad8d56
Author: Relakkes <[email protected]>
Date:   Sat Jul 15 17:11:53 2023 +0800

    feat: issue NanmiCoder#14
    refactor: 优化小红书crawler流程代码

commit e5f4ecd
Author: Relakkes <[email protected]>
Date:   Mon Jul 3 19:34:37 2023 +0800

    fix: issue NanmiCoder#12

commit 0cedd7d
Author: Relakkes <[email protected]>
Date:   Mon Jul 3 12:50:55 2023 +0800

    docs: update readme.md

commit 5743771
Author: Relakkes <[email protected]>
Date:   Thu Jun 29 16:22:39 2023 +0800

    feat: 抖音三种方式登录实现 & 抖音滑块模拟滑动实现

commit 82b7340
Author: Relakkes <[email protected]>
Date:   Wed Jun 28 21:58:36 2023 +0800

    docs:update README.md

commit 66442b0
Author: Relakkes <[email protected]>
Date:   Wed Jun 28 10:16:20 2023 +0800

    fix: 接收短信通知的py应该放到项目根目录下,不然没法导入一些其他Package

commit b8093a2
Author: Relakkes <[email protected]>
Date:   Tue Jun 27 23:38:30 2023 +0800

    refactor:优化部分代码
    feat: 增加IP代理账号池

commit 963d9a1
Author: Relakkes <[email protected]>
Date:   Sun Jun 25 22:01:38 2023 +0800

    doc: add Star History Chart

commit a7c7f95
Author: Relakkes <[email protected]>
Date:   Sun Jun 25 21:05:30 2023 +0800

    feat: 抖音评论done

commit aefdca6
Merge: 1085a2a a34b6c1
Author: Relakkes <[email protected]>
Date:   Thu Jun 22 22:43:51 2023 +0800

    Merge branch 'main' of github.com:NanmiCoder/MediaCrawler

commit 1085a2a
Author: Relakkes <[email protected]>
Date:   Thu Jun 22 22:43:26 2023 +0800

    fix: 增加小红书登录两种形态下弹窗的兼容代码

commit a34b6c1
Author: Nanmi <[email protected]>
Date:   Sun Jun 18 19:24:22 2023 +0800

    docs: update README

commit 88e8ee3
Merge: 8206f83 79a0296
Author: Robt Lenske <[email protected]>
Date:   Sat Jun 17 15:33:05 2023 +0800

    Merge pull request NanmiCoder#4 from wjw136/main

    handby&catch exception

commit 79a0296
Author: ubuntu <[email protected]>
Date:   Sat Jun 17 15:14:58 2023 +0800

    handby&catch exception

commit 8206f83
Author: Relakkes <[email protected]>
Date:   Fri Jun 16 19:35:43 2023 +0800

    feat: 小红书增加手机号自动登录模式

commit c0a783f
Author: Relakkes <[email protected]>
Date:   Tue Jun 13 14:00:04 2023 +0800

    fix: 修复登录二维码扫描不上的bug

commit e82dcae
Author: NanmiCoder <[email protected]>
Date:   Fri Jun 9 20:41:53 2023 +0800

    feat: 小红书笔记搜索,评论获取done

    docs: update docs

    Create .gitattributes

    Update README.md

commit bca6a27
Author: NanmiCoder <[email protected]>
Date:   Fri Jun 9 20:14:34 2023 +0800

    Initial commit
Merge branch 'main' of https://github.com/ly1327/MediaCrawler into brc1

# Conflicts:
#	README.md
#	base/base_crawler.py
#	config/base_config.py
#	main.py
#	media_platform/kuaishou/core.py
#	media_platform/weibo/client.py
#	media_platform/weibo/core.py
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
help wanted Extra attention is needed
Projects
None yet
Development

No branches or pull requests

2 participants