Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: v2.8.7 #8

Merged
merged 3 commits into from
Jan 17, 2024
Merged

feat: v2.8.7 #8

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
8b0be5e
feat: 同步主项目更新
shing-yu Jan 17, 2024
ed529d1
chore: 更改版本号
shing-yu Jan 17, 2024
7abd385
fix: url缺少最后的/导致崩溃
shing-yu Jan 17, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 12 additions & 4 deletions EULA.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,15 +1,23 @@
# 最终用户许可协议（EULA）
版权所有 © 2023 星隅(shingyu)
版权所有 © 2023-2024 星隅(shingyu)
最后更新日期：
2023年11月23日
2024年1月11日

## 1. 引言

本许可协议（以下简称“协议”）是您（以下简称“用户”）与本爬虫程序（以下简称“软件”）之间的法律协议。一旦安装、复制或以其他方式使用本软件，即表示同意接受本协议各项条件的约束。如果用户不同意本协议的条件，用户有权选择不使用本软件。

## 2. 许可授权

本软件是基于GPLv3开源的，用户可以在GPLv3许可证的约束下，对本软件进行复制、分发和修改。如果用户违反GPLv3许可证的的任何条款，将视为违反本协议。
2.1 本软件是基于GPLv3开源的，用户可以在GPLv3许可证的约束下，对本软件进行复制、分发和修改。如果用户违反GPLv3许可证的的任何条款，将视为违反本协议。

2.2 附加条款 (依据 GPLv3 开源许可证第七条)：

2.2.1 代下载服务：如果用户使用本软件提供代下载服务（无论有偿无偿），用户必须在服务开始前明确告知服务接收方本软件的开源地址，以及本软件遵循的 GPLv3 许可证和本附加条款。用户还必须保证服务接收方能够自由地获取、修改和再分发本软件的源代码，不受任何额外的限制或条件。（依据 GPLv3, 7(c)）

2.2.2 版权声明：用户不得移除该程序所显示的及产生的文件内的版权声明. (依据 GPLv3, 7(b))

2.2.3 基于此程序开发的程序，如果公开发布，为保护API稳定性，不得使请求频率超过0.25秒一次。 (依据 GPLv3, 7)

## 3. 责任免除

Expand Down Expand Up @@ -51,4 +59,4 @@

## 9. 最终解释权

本协议的最终解释权归软件的作者或版权所有者所有。
本协议的最终解释权归软件的作者或版权所有者所有。
12 changes: 5 additions & 7 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
**此项目是[fanqie-novel-download](https://github.com/xing-yv/fanqie-novel-download)的变体项目。**
它提供了一个简单的命令行界面，可以输入小说目录页面的URL并选择保存下载内容的编码格式。
本程序灵感即api均来源于ibxff所作用户脚本（MIT），你可以在[此处](https://greasyfork.org/zh-CN/scripts/479460)获取。
用户QQ4群(闲聊):854953755
QQ： 外1群：149050832 外2群：667146297 频道：https://pd.qq.com/s/hd0ciuont
感谢贡献（赞助）者们对本项目的支持，你可以在[此处](https://github.com/xing-yv/7mao-novel-downloader/blob/main/CONTRIBUTORS.md)获取贡献和赞助者名单。

## 特点
Expand Down Expand Up @@ -69,12 +69,6 @@
pip install -r requirements.txt
```

或（全部版本）

```shell
pip install -r requirements-all.txt
```

## 许可证

为保护此程序不被用于不良商业行为，
Expand Down Expand Up @@ -103,6 +97,10 @@ pip install -r requirements-all.txt

如果您遇到问题或有改进建议，请将其提交到此项目的[GitHub issues](https://github.com/xing-yv/7mao-novel-downloader/issues)页面。

## Star趋势

![Stars](https://api.star-history.com/svg?repos=shing-yu/7mao-novel-downloader&type=Date)

## 赞助

如果您想要支持我的开发，欢迎赞助，感谢您的支持！
Expand Down
13 changes: 0 additions & 13 deletions requirements-all.txt
View file
Open in desktop

This file was deleted.

91 changes: 80 additions & 11 deletions src/function.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -66,7 +66,7 @@ def print_usage():
您可以使用此程序提供有偿代下载服务，但在提供服务的同时，必须向服务的接收者提供此程序的获取方式，
以便他们可以自由使用、修改和分发该软件，同时也必须遵守GPLv3协议的所有其他规定。

用户QQ4群(闲聊):854953755
QQ： 外1群：149050832 外2群：667146297
如果想要指定开始下载的章节，请在输入目录页链接时按Ctrl+C。

免责声明：
Expand Down Expand Up @@ -121,7 +121,7 @@ def start():
elif choice == '4':
mode = 1
clear_screen()
print("您已进入Debug模式，将会给出更多调试信息。\n")
print("您已进入Debug模式，将会给出更多选项和调试信息。\n")
break
elif choice == '5':
mode = 4
Expand Down Expand Up @@ -167,7 +167,7 @@ def start():
clear_screen()
contributors_url = 'https://gitee.com/xingyv1024/7mao-novel-downloader/raw/main/CONTRIBUTORS.md'
try:
contributors = requests.get(contributors_url)
contributors = requests.get(contributors_url, timeout=5)

# 检查响应状态码
if contributors.status_code == 200:
Expand Down Expand Up @@ -260,7 +260,7 @@ def get_parameter(retry):
# 不是则让用户输入小说目录页的链接
while True:
try:
page_url = input("请输入目录页链接：\n")
page_url = input("请输入目录页或手机端分享链接（或书籍ID）：\n")

# 预留七猫小说判断
# if "qimao" in page_url:
Expand All @@ -270,12 +270,26 @@ def get_parameter(retry):
# mode = 3
# elif "qimao" in page_url:

# 检查 url 是否是小说目录页面
if "/shuku/" not in page_url:
print("请输入正确的小说目录页面链接")
else:
book_id = re.search(r"/(\d+)", page_url).group(1)
break # 如果是正确的链接，则退出循环
# 检查 url 类型
try:
if page_url.isdigit():
book_id = page_url
page_url = "https://www.qimao.com/shuku/" + book_id + "/"
break

elif "www.qimao.com/shuku/" in page_url:
book_id = re.search(r"www.qimao.com/shuku/(\d+)", page_url).group(1)
page_url = "https://www.qimao.com/shuku/" + book_id + "/"
break # 如果是正确的链接，则退出循环

elif "app-share.wtzw.com" in page_url:
book_id = re.search(r"article-detail/(\d+)", page_url).group(1)
page_url = "https://www.qimao.com/shuku/" + book_id + "/"
break
else:
print(Fore.YELLOW + Style.BRIGHT + "请输入正确的小说目录页面或手机端分享链接（或书籍ID）")
except AttributeError:
print(Fore.YELLOW + Style.BRIGHT + "请输入正确的小说目录页面或手机端分享链接（或书籍ID）")
# 当用户按下Ctrl+C是，可以自定义起始章节id
except KeyboardInterrupt:
while True:
Expand All @@ -295,7 +309,7 @@ def get_parameter(retry):
while True:
if mode == 4:
break
txt_encoding_num = input("请输入保存文件所使用的编码(默认:1)：1 -> utf-8 | 2 -> gb2312\n")
txt_encoding_num = input("请输入保存文件所使用的编码(默认:1)：1 -> utf-8 | 2 -> gb2312 | 3-> 搜索编码\n")

if not txt_encoding_num:
txt_encoding_num = '1'
Expand All @@ -307,6 +321,9 @@ def get_parameter(retry):
elif txt_encoding_num == '2':
txt_encoding = 'gb2312'
break
elif txt_encoding_num == '3':
txt_encoding = get_more_encoding()
break
else:
print("输入无效，请重新输入。")

Expand Down Expand Up @@ -347,6 +364,58 @@ def get_parameter(retry):
perform_user_mode_action()


def get_more_encoding():
encodings = [
"utf-8", "ascii", "latin-1", "iso-8859-1", "utf-16",
"utf-16-le", "utf-16-be", "utf-32", "utf-32-le", "utf-32-be",
"cp1252", "cp437", "cp850", "cp866", "cp932",
"cp949", "cp950", "koi8-r", "koi8-u", "macroman",
"macintosh", "gb2312", "gbk", "gb18030", "big5",
"big5hkscs", "shift_jis", "euc_jp", "euc_kr", "iso2022_jp",
"iso2022_jp_1", "iso2022_jp_2", "iso2022_jp_2004", "iso2022_jp_3", "iso2022_jp_ext",
"shift_jis_2004", "shift_jisx0213", "euc_jis_2004", "euc_jisx0213", "latin_1",
"iso8859_2", "iso8859_3", "iso8859_4", "iso8859_5", "iso8859_6",
"iso8859_7", "iso8859_8", "iso8859_9", "iso8859_10", "iso8859_13",
"iso8859_14", "iso8859_15", "iso8859_16", "cp500", "cp720",
"cp737", "cp775", "cp852", "cp855", "cp857",
"cp858", "cp860", "cp861", "cp862", "cp863",
"cp864", "cp865", "cp869", "cp874", "cp875",
"cp1006", "cp1026", "cp1140", "cp1250", "cp1251",
"cp1253", "cp1254", "cp1255", "cp1256", "cp1257",
"cp1258", "cp65001", "hz", "iso2022_jp_2004", "iso2022_kr",
"iso8859_11", "iso8859_16", "johab", "ptcp154", "utf_7",
"utf_8_sig"
]

while True:
query = input("请输入你想要搜索的编码：")
query = query.lower()
import difflib
# 使用difflib库的get_close_matches方法找到相似的编码
similar_encodings = difflib.get_close_matches(query, encodings)

print("以下是与你的搜索内容相似的编码：")
for i, encoding in enumerate(similar_encodings):
print(f"{i + 1}. {encoding}")

while True:
choice_ = input("请选择一个编码, 输入r以重新搜索：")
if choice_ == "r":
clear_screen()
break
elif choice_.isdigit():
choice = int(choice_)
if choice > len(similar_encodings):
print("输入无效，请重新输入。")
continue
chosen_encoding = similar_encodings[choice - 1]
print(f"你选择的编码是：{chosen_encoding}")
return chosen_encoding
else:
print("输入无效，请重新输入。")
continue


def perform_user_mode_action():
global return_info
# 判断用户处于什么模式
Expand Down
2 changes: 1 addition & 1 deletion src/main.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@
import function as f
from sys import exit

version = "v2.8.6-post.1"
version = "v2.8.7"

# 检查EULA
f.check_eula()
Expand Down
53 changes: 52 additions & 1 deletion src/public.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,11 @@
from Crypto.Util.Padding import unpad
from base64 import b64decode
import requests
from tqdm import tqdm
import hashlib
from colorama import Fore, Style, init

init(autoreset=True)


# 替换非法字符
Expand Down Expand Up @@ -76,7 +81,53 @@ def decrypt_qimao(content):
return fntxt


def get_qimao(book_id, chapter_id, sign):
def get_api(book_id, chapter):
# 获取章节标题
chapter_title = chapter.find("span", {"class": "txt"}).get_text().strip()
chapter_title = rename(chapter_title)

# 获取章节网址
chapter_url = chapter.find("a")["href"]

# 获取章节 id
chapter_id = re.search(r"/(\d+)-(\d+)/", chapter_url).group(2)

# 尝试获取章节内容
chapter_content = None
retry_count = 1
while retry_count < 4: # 设置最大重试次数
try:
param_string = f"chapterId={chapter_id}id={book_id}{sign_key}"
sign = hashlib.md5(param_string.encode()).hexdigest()
encrypted_content = send_request(book_id, chapter_id, sign)
except Exception as e:

tqdm.write(Fore.RED + Style.BRIGHT + f"发生异常: {e}")
if retry_count == 1:
tqdm.write(f"{chapter_title} 获取失败，正在尝试重试...")
tqdm.write(f"第 ({retry_count}/3) 次重试获取章节内容")
retry_count += 1 # 否则重试
continue

if "data" in encrypted_content and "content" in encrypted_content["data"]:
encrypted_content = encrypted_content['data']['content']
chapter_content = decrypt_qimao(encrypted_content)
chapter_content = re.sub('<br>', '\n', chapter_content)
break # 如果成功获取章节内容，跳出重试循环
else:
if retry_count == 1:
tqdm.write(f"{chapter_title} 获取失败，正在尝试重试...")
tqdm.write(f"第 ({retry_count}/3) 次重试获取章节内容")
retry_count += 1 # 否则重试

if retry_count == 4:
tqdm.write(f"无法获取章节内容: {chapter_title}，跳过。")
return # 重试次数过多后，跳过当前章节

return chapter_title, chapter_content, chapter_id


def send_request(book_id, chapter_id, sign):
headers = {
"AUTHORIZATION": "",
"app-version": "51110",
Expand Down
48 changes: 7 additions & 41 deletions src/qimao_batch.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -130,54 +130,20 @@ def download_novels(url, encoding, path_choice, folder_path, data_folder):
# 遍历每个章节链接
for chapter in tqdm(chapters):
time.sleep(1)
# 获取章节标题
chapter_title = chapter.find("span", {"class": "txt"}).get_text().strip()

# 获取章节网址
chapter_url = chapter.find("a")["href"]

# 获取章节 id
chapter_id = re.search(r"/(\d+)-(\d+)/", chapter_url).group(2)

# 尝试获取章节内容
chapter_content = None
retry_count = 1
while retry_count < 4: # 设置最大重试次数
try:
param_string = f"chapterId={chapter_id}id={book_id}{p.sign_key}"
sign = hashlib.md5(param_string.encode()).hexdigest()
encrypted_content = p.get_qimao(book_id, chapter_id, sign)
except Exception as e:

tqdm.write(Fore.RED + Style.BRIGHT + f"发生异常: {e}")
if retry_count == 1:
tqdm.write(f"{chapter_title} 获取失败，正在尝试重试...")
tqdm.write(f"第 ({retry_count}/3) 次重试获取章节内容")
retry_count += 1 # 否则重试
continue

if "data" in encrypted_content and "content" in encrypted_content["data"]:
encrypted_content = encrypted_content['data']['content']
chapter_content = p.decrypt_qimao(encrypted_content)
chapter_content = re.sub('<br>', '\n', chapter_content)
break # 如果成功获取章节内容，跳出重试循环
else:
if retry_count == 1:
tqdm.write(f"{chapter_title} 获取失败，正在尝试重试...")
tqdm.write(f"第 ({retry_count}/3) 次重试获取章节内容")
retry_count += 1 # 否则重试

if retry_count == 4:
tqdm.write(f"无法获取章节内容: {chapter_title}，跳过。")
continue # 重试次数过多后，跳过当前章节
result = p.get_api(book_id, chapter)

if result is None:
continue
else:
chapter_title, chapter_text, chapter_id = result

# 去除其他 html 标签
# chapter_text = re.sub(r"</?\w+>", "", chapter_text)
#
# chapter_text = p.fix_publisher(chapter_text)

# 在小说内容字符串中添加章节标题和内容
content += f"\n\n\n{chapter_title}\n\n{chapter_content}"
content += f"\n\n\n{chapter_title}\n\n{chapter_text}"

# 打印进度信息
tqdm.write(f"已获取 {chapter_title}")
Expand Down
Loading
Loading