We read every piece of feedback, and take your input very seriously.
To see all available qualifiers, see our documentation.
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
在startProxyCrawl进程中,启动时会检测数据库中代理是否可用,代码位置在: spawns.append(gevent.spawn(detect_from_db, self.myip, proxy, self.proxies)) 当代理不可用时会调用更新操作: sqlhelper.update({'ip': proxy[0], 'port': proxy[1]}, {'score': score}) 因为程序使用gevent,所以在不可用代理较多时会存在并发调用update操作的情况,但是所有update操作共用一个SqlHelper的session实例,该session不是线程安全的,会导致系统报如下异常: RuntimeError: reentrant call inside <_io.BufferedReader name=2296>
需要为该session的更新、删除、添加操作增加同步锁
The text was updated successfully, but these errors were encountered:
No branches or pull requests
在startProxyCrawl进程中,启动时会检测数据库中代理是否可用,代码位置在:
spawns.append(gevent.spawn(detect_from_db, self.myip, proxy, self.proxies))
当代理不可用时会调用更新操作:
sqlhelper.update({'ip': proxy[0], 'port': proxy[1]}, {'score': score})
因为程序使用gevent,所以在不可用代理较多时会存在并发调用update操作的情况,但是所有update操作共用一个SqlHelper的session实例,该session不是线程安全的,会导致系统报如下异常:
RuntimeError: reentrant call inside <_io.BufferedReader name=2296>
需要为该session的更新、删除、添加操作增加同步锁
The text was updated successfully, but these errors were encountered: