Scrapy installation on CentOS and Windows

By Harikrishna Doredla

Scrapy is an application framework supporting development of applications in a given environment. I discuss here the steps of installation of Scrapy both CentOS and Windows environments including installation of the dependencies thereof.

Scrapy Installation on Centos 6.5

Scrapy needs python 2.7 and above to run in CentOS . CentOS 6.5 comes with Python 2.6 .So we need to install python 2.7+ to run Scrapy code. Here are the steps to install Python 2.7.11. Firstly install the Scrapy dependencies, preceding installation of Python 2.7.11.

Installing Python 2.7.11

Install setup tools


Install PIP (a package management system used to install and manage software packages written in Python.)

Install Scrapy

Once Scrapy installation is done check “scrapy shell <URL> ” from command line

Scrapy Installation on Windows

Python installation

Install Python 2.7.11 from

Update the PATH ; open  Command prompt and run:

Close the command prompt window so that changes become effective. Reopen command prompt and run the following command and to check the Python version:

pywin32 installation
Install pywin32 from

Be sure you download the architecture (win32 or amd64) that matches your system [even our system is 64 bit also we need to download 32 bit only (example: pywin32-219.win32-py2.7.exe)]

PIP Installation
Install pip from
Now open a Command prompt to check pip is installed correctly:

LXML package installation
Install LXML package from below

Install Visual C++ Compiler
Install “Microsoft Visual C++ Compiler for Python 2.7″ from
Scrapy installation
Ensure Python 2.7 and pip package manager working, then install Scrapy:
pip install Scrapy
pip install happybase
pip install requests
pip install OrderedDict
pip install html2text

Add PyDev perspective

Install Eclipse Juno or STS then add PyDev perspective to write python code

Once  Scrapy installation is done check “scrapy shell <URL> ” from command line