基于scrapy框架的百度地图公交站点数据爬取

在python3.6环境下搭建的scrapy框架，爬取百度地图全国各个城市公交站点数据，提供解决这一类问题的思路和反爬虫机制以及实现代码

from scrapy import cmdline

cmdline.execute（‘scrapy crawl baiduBusSpider‘.split（））

 属性            大小     日期    时间   名称
----------- ---------  ---------- -----  ----
     目录           0  2018-10-08 20:03  baiduScrapy\
     文件        6148  2018-10-09 10:59  baiduScrapy\.DS_Store
     目录           0  2018-10-09 13:37  __MACOSX\
     目录           0  2018-10-09 13:37  __MACOSX\baiduScrapy\
     文件         120  2018-10-09 10:59  __MACOSX\baiduScrapy\._.DS_Store
     目录           0  2018-10-09 13:22  baiduScrapy\chongqingBus\
     文件        6148  2018-10-09 10:59  baiduScrapy\chongqingBus\.DS_Store
     目录           0  2018-10-09 13:37  __MACOSX\baiduScrapy\chongqingBus\
     文件         120  2018-10-09 10:59  __MACOSX\baiduScrapy\chongqingBus\._.DS_Store
     目录           0  2018-10-09 13:09  baiduScrapy\chongqingBus\chongqingBus\
     目录           0  2018-10-09 13:09  baiduScrapy\chongqingBus\chongqingBus\spiders\
     文件         161  2018-10-08 13:01  baiduScrapy\chongqingBus\chongqingBus\spiders\__init__.py
     目录           0  2018-10-09 13:09  baiduScrapy\chongqingBus\chongqingBus\spiders\__pycache__\
     文件        1488  2018-10-09 13:09  baiduScrapy\chongqingBus\chongqingBus\spiders\__pycache__\baiduBusSpider.cpython-36.pyc
     文件         161  2018-10-08 16:58  baiduScrapy\chongqingBus\chongqingBus\spiders\__pycache__\__init__.cpython-36.pyc
     文件        1746  2018-10-09 13:09  baiduScrapy\chongqingBus\chongqingBus\spiders\baiduBusSpider.py
     文件           0  2018-10-08 13:01  baiduScrapy\chongqingBus\chongqingBus\__init__.py
     目录           0  2018-10-09 13:09  baiduScrapy\chongqingBus\chongqingBus\__pycache__\
     文件         601  2018-10-09 11:03  baiduScrapy\chongqingBus\chongqingBus\__pycache__\pipelines.cpython-36.pyc
     文件         395  2018-10-09 11:39  baiduScrapy\chongqingBus\chongqingBus\__pycache__\items.cpython-36.pyc
     文件         514  2018-10-09 13:09  baiduScrapy\chongqingBus\chongqingBus\__pycache__\settings.cpython-36.pyc
     文件         153  2018-10-08 16:58  baiduScrapy\chongqingBus\chongqingBus\__pycache__\__init__.cpython-36.pyc
     文件        3609  2018-10-08 13:02  baiduScrapy\chongqingBus\chongqingBus\middlewares.py
     文件        3129  2018-10-09 13:09  baiduScrapy\chongqingBus\chongqingBus\settings.py
     文件         553  2018-10-09 11:36  baiduScrapy\chongqingBus\chongqingBus\items.py
     文件         664  2018-10-09 11:02  baiduScrapy\chongqingBus\chongqingBus\pipelines.py
     文件         267  2018-10-08 13:02  baiduScrapy\chongqingBus\scrapy.cfg
     文件          82  2018-10-08 21:15  baiduScrapy\chongqingBus\main.py
     文件        8828  2018-10-09 13:19  baiduScrapy\chongqingBus\data.txt
     目录           0  2018-10-08 11:10  baiduScrapy\venv\
     目录           0  2018-10-08 13:01  baiduScrapy\venv\bin\
............此处省略3120个文件信息