Discuss / Python / requests 遇到的坑

requests 遇到的坑

Topic source

xian_wen

#1 Created at ... [Delete] [Delete and Lock User]

看了一圈评论区,发现貌似没有人遇到和我相同的问题,特此记录一下。

环境:anaconda 下新建的环境 py39(python 3.9.4),base 环境为 python 3.8.5,我一般都是在新建的环境下学习,运行requests.get报错:

>>> import requests
>>> r = requests.get('https://www.douban.com')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Anaconda3\envs\py39\lib\site-packages\requests\api.py", line 76, in get
    return request('get', url, params=params, **kwargs)
  File "C:\Anaconda3\envs\py39\lib\site-packages\requests\api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "C:\Anaconda3\envs\py39\lib\site-packages\requests\sessions.py", line 542, in request
    resp = self.send(prep, **send_kwargs)
  File "C:\Anaconda3\envs\py39\lib\site-packages\requests\sessions.py", line 655, in send
    r = adapter.send(request, **kwargs)
  File "C:\Anaconda3\envs\py39\lib\site-packages\requests\adapters.py", line 439, in send
    resp = conn.urlopen(
  File "C:\Anaconda3\envs\py39\lib\site-packages\urllib3\connectionpool.py", line 696, in urlopen
    self._prepare_proxy(conn)
  File "C:\Anaconda3\envs\py39\lib\site-packages\urllib3\connectionpool.py", line 964, in _prepare_proxy
    conn.connect()
  File "C:\Anaconda3\envs\py39\lib\site-packages\urllib3\connection.py", line 359, in connect
    conn = self._connect_tls_proxy(hostname, conn)
  File "C:\Anaconda3\envs\py39\lib\site-packages\urllib3\connection.py", line 500, in _connect_tls_proxy
    return ssl_wrap_socket(
  File "C:\Anaconda3\envs\py39\lib\site-packages\urllib3\util\ssl_.py", line 432, in ssl_wrap_socket
    ssl_sock = _ssl_wrap_socket_impl(sock, context, tls_in_tls)
  File "C:\Anaconda3\envs\py39\lib\site-packages\urllib3\util\ssl_.py", line 474, in _ssl_wrap_socket_impl
    return ssl_context.wrap_socket(sock)
  File "C:\Anaconda3\envs\py39\lib\ssl.py", line 500, in wrap_socket
    return self.sslsocket_class._create(
  File "C:\Anaconda3\envs\py39\lib\ssl.py", line 997, in _create
    raise ValueError("check_hostname requires server_hostname")
ValueError: check_hostname requires server_hostname

解决办法:

1. (python 3.8.5 直接跳到第 2 步) 将默认的 urllib3 1.26.4 降级为 urllib3 1.25.11:

conda install urllib3==1.25.11

2. 再次运行requests.get,豆瓣的status_code为 418,百度、谷歌的status_code为 200:

>>> import requests
>>> r = requests.get('https://www.douban.com')
>>> r.status_code
418
>>> r = requests.get('https://www.baidu.com')
>>> r.status_code
200
>>> r = requests.get('https://www.google.com')
>>> r.status_code
200

3. 添加User-Agent有值就行,具体值无所谓

>>> r = requests.get('https://www.douban.com', headers={'User-Agent': 'M'})
>>> r.status_code
200
>>> r = requests.get('https://www.douban.com', headers={'User-Agent': 'A'})
>>> r.status_code
200

我也是状态码是418,反爬虫

Hsissiya

#3 Created at ... [Delete] [Delete and Lock User]

当前加了代理,状态码还是418

han

#4 Created at ... [Delete] [Delete and Lock User]
headers={    'User-Agent':'Mozilla/5.0 (Windows NT 6.1;WOW64) AppleWebKit/537.36 (KHTML,like GeCKO) Chrome/45.0.2454.85 Safari/537.36 115Broswer/6.0.3',    'Referer':'https://movie.douban.com/',    'Connection':'keep-alive'}r = requests.get('https://www.douban.com/search', params={'q': 'python', 'cat': '1001'},headers=headers)

  • 1

Reply