首页 > Linux系统管理 > 安装Scrapy 发现的问题
2015
09-15

安装Scrapy 发现的问题

以前机器上面pip都有安装的,所以直接

pip install scrapy

报了一大堆的错误,大慨意思是安装过程中setuptools会通过https访问PyPI来拉取某个库依赖的其它包,而通过https访问PyPI源时,它使用的是高版本的openssl,但其加载的certificate证书却是与系统默认的旧版openssl库配套的证书,所以会报出”certificate verify failed”的错误

可以看到,出问题的地方从”Running setup.py install for cryptography”那行(第33行)开始,而setup.py install调用了setuptools,所以可以推断,认证失败的确是由setuptools引起

安装网上的解决方法

pip install --cert /home/slvher/tools/https-ca/https/cacert.pem -U -i https://pypi.python.org/simple six

在执行安装的时候报了ImportError: No module named pkg_resources

而且连pip都用不了直接报错

pip 和python 版本不合。我把python升级到2.7了的,决定重新安装下相对应的版本

setuptool:
 
#wget https://pypi.python.org/packages/2.7/s/setuptools/setuptools-0.6c11-py2.7.egg  --no-check-certificate
#chmod +x setuptools-0.6c11-py2.7.egg
#sh setuptools-0.6c11-py2.7.egg

 

安装的时候还报 zipimport.ZipImportError: can’t decompress data; zlib not available

解决方法:

1、安装依赖zlib、zlib-devel

2、重新编译安装Python

  1. ./configure
  2. 编辑Modules/Setup文件
  3. 找到下面这句,去掉注释
  4. #zlib zlibmodule.c -I$(prefix)/include -L$(exec_prefix)/lib -lz
  5. 重新编译安装:make & make install

pip:

wget --no-check-certificate https://github.com/pypa/pip/archive/1.5.5.tar.gz

注意:wget获取https的时候要加上:–no-check-certificate

tar zvxf 1.5.5.tar.gz    #解压文件
cd pip-1.5.5/
python setup.py install

当执行pip -v 的时候又报了个 ImportError: cannot import name HTTPSHandler
没有安装openssl-devel 至装了openssl
没办法只能装了

yum install -y openssl-devel

然后还要重新安装python 都重装好几回python了
等python ./configure && make && make install
完后 在执行

# pip -v
 
Usage:   
  pip <command> [options]
 
Commands:
  install                     Install packages.
  uninstall                   Uninstall packages.
  freeze                      Output installed packages in requirements format.
  list                        List installed packages.

显示正常了,哪再安装scrapy试试

pip install scrapy
继续有问题,还是依赖关系的问题

ERROR: /bin/sh: xslt-config: command not found
 
    ** make sure the development packages of libxml2 and libxslt are installed **
 File "/usr/local/lib/python2.7/tarfile.py", line 1744, in bz2open
    raise CompressionError("bz2 module is not available")
CompressionError: bz2 module is not available
#yum -y install libxslt-devel bzip2-devel

再编译还是报CompressionError: bz2 module is not available的错误
老外的意思是

sudo yum install bzip2-devel
and then rebuild python

擦 哪就再次rebuild python吧
继续pip install scrapy 老的问题是解决了,报了个新的问题

distutils.errors.DistutilsError: Setup script exited with error: command 'gcc' failed with exit status 1
 
#yum -y install libffi-devel
# pip install scrapy
Requirement already satisfied (use --upgrade to upgrade): scrapy in /usr/local/lib/python2.7/site-packages
Cleaning up...

没有报错了
scrapy 命令也出来了 但是不能用,继续报错

Traceback (most recent call last):
  File "/usr/local/bin/scrapy", line 7, in <module>
    from scrapy.cmdline import execute
  File "/usr/local/lib/python2.7/site-packages/scrapy/__init__.py", line 27, in <module>
    from . import _monkeypatches
  File "/usr/local/lib/python2.7/site-packages/scrapy/_monkeypatches.py", line 20, in <module>
    import twisted.persisted.styles  # NOQA
  File "/usr/local/lib/python2.7/site-packages/twisted/__init__.py", line 53, in <module>
    _checkRequirements()
  File "/usr/local/lib/python2.7/site-packages/twisted/__init__.py", line 37, in _checkRequirements
    raise ImportError(required + ": no module named zope.interface.")
ImportError: Twisted requires zope.interface 3.6.0 or later: no module named zope.interface.

还需要安装zope包,不是说在linux下安装setuptools,再安装twisted,会联网自动安装zope,不管了就手动安装试下

wget http://pypi.python.org/packages/source/z/zope.interface/zope.interface-4.0.1.tar.gz
# tar xvf zope.interface-4.0.1.tar.gz 
# cd zope.interface-4.0.1
# python setup.py install
># scrapy -h
Scrapy 1.0.3 - no active project
 
Usage:
  scrapy <command> [options] [args]
 
Available commands:
  bench         Run quick benchmark test
  commands      
  fetch         Fetch a URL using the Scrapy downloader
  runspider     Run a self-contained spider (without creating a project)
  settings      Get settings values
  shell         Interactive scraping console
  startproject  Create new project
  version       Print Scrapy version
  view          Open URL in browser, as seen by Scrapy
 
  [ more ]      More commands available when run from project directory
 
Use "scrapy <command> -h" to see more info about a command
# scrapy startproject tutorial
2015-09-16 12:03:28 [scrapy] INFO: Scrapy 1.0.3 started (bot: scrapybot)
2015-09-16 12:03:28 [scrapy] INFO: Optional features available: ssl, http11
2015-09-16 12:03:28 [scrapy] INFO: Overridden settings: {}
New Scrapy project 'tutorial' created in:
    /opt/tutorial
 
You can start your first spider with:
    cd tutorial
    scrapy genspider example example.com

貌似可以了,有点小激动哈

最后编辑:
作者:saunix
大型互联网公司linux系统运维攻城狮,专门担当消防员

留下一个回复