我们用python写爬虫代码时，如何去爬一个需要登录用户名和密码的页面获取信息呢？这里我们先来了解下登录后的网页是如何继续访问业务以获取相关的权利。
当用户首次登录请求访问服务器时，服务器先在客户端存放包含该客户的相关信息的Cookie，以后客户端每次请求访问服务器时，都会在HTTP请求数据中包含Cookie，服务器解析HTTP请求中的Cookie，就能由此获得关于客户的相关信息。
因此我们使用登录以后的Cookie。自己编辑Cookie访问Web服务器。网络爬虫中Cookie的两种使用方式，代码中网页已经不存在，但是可以自己在其他网页中尝试。我们使用的是登录之后的Cookie。
如何查看登录后的网站的cookies信息呢？登录网站后点击右键，选择“检查元素”，找到network属性，找到一串cookies信息。 如：
获取到cookies后，我们可以用两种方法进行使用：
1、直接将Cookie写在header头部
# coding:utf-8
import requests
from bs4 import BeautifulSoup
cookie = ihelp_portal_index_M002975=init_init_block;usphidden=true;_ha_id..0c9d=F6-36-48-5E-25-4A-5F-E3-DD-80-3A-2E-CE-AA-89-6B;_ha_ses..0c9d=b5680415e57a0736fd6d9655da0228a770707d4d;login_logFlag=in;suid=F6-36-48-5E-25-4A-5F-E3-DD-80-3A-2E-CE-AA-89-6B;login_uid=F6-36-48-5E-25-4A-5F-E3-DD-80-3A-2E-CE-AA-89-6B;_sid_=4F-62-0C-B3-6C-B5-B2-77-51-B7-C6-A8-1F-26-46-6D-E9-76-62-74-D6-5B-F8-0F-42-40-42-BC-E1-B5-0E-93-A6-2E-AF-33-F8-BF-FA-22;login_sip=7E-1A-97-8E-3D-46-19-E3-36-7B-EB-94-78-CF-76-E5-31-DD-35-52-A5-7C-E0-59;login_sid=4F-62-0C-B3-6C-B5-B2-77-51-B7-C6-A8-1F-26-46-6D-E9-76-62-74-D6-5B-F8-0F-42-40-42-BC-E1-B5-0E-93-A6-2E-AF-33-F8-BF-FA-22;LtpaToken=xsOHrCTLIAlUVP2XBzryGj7RmkqVXxsnzNMFLll+GTiC5R9KP/nrD8JUlyREAUAQr9xCLoL4Xdkr9tWP7I9SwuQdY3XpWxSwvlu4b3hdgN44bripPctg/GBhfZr3pv7+StUO281vrlxDFFDJyQ5XPvf4pbOBxZdY54DDnKbQwsn+6WIWSGKdswcR39LyGGBaKfjEc2mi/KOOYn0fGhpovltXu3oxYDl3QmIYNIDSrIui8FA32D2thJfG9YEim/DfeQXqJQ3BDSzCtN6QqXgsnsGelwAUudNbdwe1yY3FUXxnbVZDqIfKnrR1wljTf0r2PwcWKkYAS1PVuzVIVKHZQIo8a8Gpd8bvze0LgRfB4pU6i3oOZebA1w==;lang=zh;__hau=SUPPORT.1563506466.1302445886;v1st=16C22A2EB7FC7CF0;sensorsdata2015jssdkcross={"distinct_id":"16c46ba49521-038a420ed966aa4-2c554758-1294854-16c46ba49536","$device_id":"16c46ba49521-038a420ed966aa4-2c554758-1294854-16c46ba49536","props":{"$latest_traffic_source_type":"直接流量","$latest_referrer":"","$latest_referrer_host":"","$latest_search_keyword":"未取到值_直接打开"}};bishengSID=0000U2DS4iBlZff208JpzEdf1QX=19grdfk5i;SCORESESSIONID=f2aac09c-51f3-49e8-a697-9a39dbb065bb;hwkm_sid=VsF-DJGTydlOWzpYeezRuapEB0qAo-CF;hwkm_sid.sig=04Gie-iwTWMvHceayR5639x3uWQ''' header = {
'User-Agent': 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Geck o) Chrome/53.0.2785.143 Safari/537.36',
'Connection': 'keep-alive',
'accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8', 'Cookie': cookie}
url = 'http://3ms.huawei.com/km/groups/2026461/my?l=zh-cn'
wbdata = requests.get(url,headers=header).text
soup = BeautifulSoup(wbdata,'lxml')
print(soup)
2、使用requests插入Cookie
# coding:utf-8
import requests
from bs4 import BeautifulSoup
cookie = {
'ihelp_portal_index_M002975':'init_init_block',
'usphidden':'true',
'_ha_id..0c9d':'F6-36-48-5E-25-4A-5F-E3-DD-80-3A-2E-CE-AA-89-6B',
'_ha_ses..0c9d':'b5680415e57a0736fd6d9655da0228a770707d4d',
'login_logFlag':'in',
'suid':'F6-36-48-5E-25-4A-5F-E3-DD-80-3A-2E-CE-AA-89-6B',
'login_uid':'F6-36-48-5E-25-4A-5F-E3-DD-80-3A-2E-CE-AA-89-6B',
'_sid_':'4F-62-0C-B3-6C-B5-B2-77-51-B7-C6-A8-1F-26-46-6D-E9-76-62-74-D6-5B-F8-0F-42-40-42-BC-E1-B5-0E-93-A6-2E-AF-33-F8-BF-FA-22',
'login_sip':'7E-1A-97-8E-3D-46-19-E3-36-7B-EB-94-78-CF-76-E5-31-DD-35-52-A5-7C-E0-59',
'login_sid':'4F-62-0C-B3-6C-B5-B2-77-51-B7-C6-A8-1F-26-46-6D-E9-76-62-74-D6-5B-F8-0F-42-40-42-BC-E1-B5-0E-93-A6-2E-AF-33-F8-BF-FA-22',
'LtpaToken':'xsOHrCTLIAlUVP2XBzryGj7RmkqVXxsnzNMFLll+GTiC5R9KP/nrD8JUlyREAUAQr9xCLoL4Xdkr9tWP7I9SwuQdY3XpWxSwvlu4b3hdgN44bripPctg/GBhfZr3pv7+StUO281vrlxDFFDJyQ5XPvf4pbOBxZdY54DDnKbQwsn+6WIWSGKdswcR39LyGGBaKfjEc2mi/KOOYn0fGhpovltXu3oxYDl3QmIYNIDSrIui8FA32D2thJfG9YEim/DfeQXqJQ3BDSzCtN6QqXgsnsGelwAUudNbdwe1yY3FUXxnbVZDqIfKnrR1wljTf0r2PwcWKkYAS1PVuzVIVKHZQIo8a8Gpd8bvze0LgRfB4pU6i3oOZebA1w':'':'',
'lang':'zh',
'__hau':'SUPPORT.1563506466.1302445886',
'v1st':'16C22A2EB7FC7CF0',
'sensorsdata2015jssdkcross':'{"distinct_id":"16c46ba49521-038a420ed966aa4-2c554758-1294854-16c46ba49536","$device_id":"16c46ba49521-038a420ed966aa4-2c554758-1294854-16c46ba49536","props":{"$latest_traffic_source_type":"直接流量","$latest_referrer":"","$latest_referrer_host":"","$latest_search_keyword":"未取到值_直接打开"}}',
'bishengSID':'0000U2DS4iBlZff208JpzEdf1QX:19grdfk5i',
'SCORESESSIONID':'f2aac09c-51f3-49e8-a697-9a39dbb065bb',
'hwkm_sid':'VsF-DJGTydlOWzpYeezRuapEB0qAo-CF',
'hwkm_sid.sig':'04Gie-iwTWMvHceayR5639x3uWQ'}
url = 'http://3ms.huawei.com/km/groups/2026461/my?l=zh-cn'
wbdata = requests.get(url,cookies=cookie).text
soup = BeautifulSoup(wbdata,'lxml')
print(soup)
这样我们就轻松的使用Cookie获取到了需要登录验证后才能浏览的网页和资源了。上面的Cookie是从自己浏览的网页中复制粘贴得到的。
——————————————————————————
BY：HW-CJ
苏公安网备 32011402010933号