Discuss / Python / 关于python抓取问题
Back

关于python抓取问题

奇葩的师弟

#1 Created at ... [Delete] [Delete and Lock User] 
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
import urllib2

url="http://www.5477dm.com/"


f = urllib2.urlopen('http://www.5477dm.com/').info()

k = f.getparam('charset')

print(k)
up=urllib2.urlopen(url)

cont=up.read()
cont=unicode(cont,"gbk").encode("utf8")
print(cont)

我想获得http://www.5477dm.com 的代码. 运行结果

gbk
Traceback (most recent call last):
  File "zhuaquceshi.py", line 16, in <module>
    cont=unicode(cont,"gbk").encode("utf8")
UnicodeDecodeError: 'gbk' codec can't decode bytes in position 39162-39163: illegal multibyte sequence

出错,于是我查了下网页源码,发现主页

<meta http-equiv="Content-Type" content="text/html; charset=gbk">

但是,js的引用是 utf-8

<script charset="utf-8" src="http://pan.baidu.com/res/static/thirdparty/yunfujian-end/_build/yunfujian.discuz.js" type="text/javascript"></script>

这样我应该怎么处理才能把网页转成utf-8格式呢? 新手.谢谢.

奇葩的师弟

#2 Created at ... [Delete] [Delete and Lock User] 
cont=cont.decode("gbk","ignore").encode("utf8")

刚刚发现,出错原因是发现了非法字符,使用 ignore 把错误忽略了. 不知道有没有其他更好的方法处理. 求大神教.

奇葩的师弟

#3 Created at ... [Delete] [Delete and Lock User]

那段代码在ub上运行正常，但是在win跑起来就有出错了。 修改代码。

#!/usr/bin/env python3
# -*- coding: utf-8 -*-
import urllib.request as urllib2

url="http://www.5477dm.com/"

up=urllib2.urlopen(url)

cont=up.read()

cont=cont.decode('gbk','ignore').encode('utf-8')
file_object = open('thefile.txt', 'w')
file_object.write(str(cont))
file_object.close( )
cont=cont.decode("utf-8")


print(cont)
  • 1

Reply