You can not select more than 25 topics
Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
|
|
|
|
|
# VulCrawler
|
|
|
|
|
|
|
|
|
|
|
|
本项目是使用scrapy-redis框架实现分布式爬虫，由于头歌平台上传不了目录，只能是一个一个文件上传，具体的项目结构见图
|
|
|
|
|
|
|
|
|
|
|
|
VulCrawl
|
|
|
|
|
|
- __init__.py
|
|
|
|
|
|
- items.py
|
|
|
|
|
|
- middlewares.py
|
|
|
|
|
|
- pipelines.py
|
|
|
|
|
|
- settings.py
|
|
|
|
|
|
- Vul_db.py
|
|
|
|
|
|
--spiders
|
|
|
|
|
|
- __init__.py
|
|
|
|
|
|
- vulcrawl.py
|
|
|
|
|
|
- vulcrawl2.py
|
|
|
|
|
|
|
|
|
|
|
|
|
