For years now I've been using a python script from my Windows box that uses BeautifulSoup and requests to do bit of web scraping to return some running race results for my club. A long time ago, I installed Python 3.x and did a pip install of BeautifulSoup4, things just worked. I would double-click the script -- or run from a command prompt with custom arguments-- and all would run well.
This thing has worked perfectly for years.
Unfortunately I seem to have messed up the install lately. Now when I run it from the command line, I get this:
D:\Dev\Srr>srr_new6.py
Traceback (most recent call last):
File "D:\Dev\Srr\srr_new6.py", line 23, in <module>
from bs4 import BeautifulSoup
ImportError: No module named bs4
D:\Dev\Srr>
But if I try to install beautifulsoup4 it tells me it's already installed.
D:\Dev\Srr>pip install beautifulsoup4
Requirement already satisfied: beautifulsoup4 in c:\program
files\python36\lib\site-packages
D:\Dev\Srr>
When I installed python I was sure to put it in the system path. And when I check the path from my command prompt, there it is. If I type "python" from the command line, I do get the interactive environment. And I tried uninstalling and then reinstalling both python and beautifulsoup it did not appear to help.
Googling I've found mention of "virtualenv" so it sounds to me like I need to "activate" that but cannot seem to find or invoke it. Windows doesn't know about it when I type it from the command line.
Is there some simple way that I can fix this? Is there some way to make beautifulsoup4 be a part of the "default" virtualenv on Windows so that windows will always find it?
(I'd love to learn python. I've been saying that for years. But this isn't for work and there always seems to be other things I need to learn first for work)
Related
I'm a complete beginner in Python programming. I have trouble installing/importing the module 'requests' on python. When I use my command terminal to install requests, I get a message that requests is already installed. However, when I try to import requests into the file I'm working on, python tells me there is no such module installed.
Sorry to bother you with this silly and probably easy question, thanks in advance!
If you use PyCharm (which is a great choice in my opinion), go to the tab Run and select Edit Configuration and in the window that just opened make sure the your Python interpreter is the one you used when you pip installed the package you asked about.
You can check your Python version or just see in PyCharm if the requests is actually installed by going to the tab File, select Settings, click on Project: name_of_your_project and finally check in Project Interpreter that the package is installed.
I'm using netbeans to write a simple python program which I need the requests module for, I've downloaded requests through terminal and it all seems to be fine there but netbeans can't seem to find it.
This is the error that it's throwing up:
import requests
ImportError: No module named requests
I've tried installing the requests library directly into the python folder but the folder won't let me paste anything into it.
There do seem to be answers on the netbeans forums but their server is down so won't let me on their website to my annoyance!
EDIT
I've tried to run python setup.py install as per other answers on the website but had no luck.
EDIT
have tried completely uninstalling python and requests to make sure it wasn't an installation error but still no luck.
This clearly looks like an error of installation of the request module to some other place than where your netbeans expects when running the code.
In your console run
which python
Check if this gives the same path as the one set in your netbeans. You can set your path by adding new platform using Tools > Python Platforms > New:
I would suggest that you learn bit more about sandboxed environments such as virtualenv. This article shows how you can use a virtualenv to install packages and use the same virtualenv for netbeans so that whatever packages you install in the virtualenv will be available in the netbeans for you to use. For this case, it could be requests.
In the end I gave up with requests, as I was using requests to get json data from an API I decided just to go back to the drawing board and start over rather than attempt to fix something that I couldn't work out. I am now using the urllib import and whilst this may not be the most efficient way it works which is the most important thing.
I'm having trouble importing tweepy. I've looked through so many previous questions and still can't find a correct solution. I think it has something to do with how tweepy is being downloaded when I install but I'm not sure. I get an import error saying that "tweepy is not a package".
I have tweepy library connected to the interpreter and all that but, it is saved as a compressed EGG file instead of a file folder like the rest of my packages. I think that has something to do with it but I'm not too sure.
Also, tweepy works in my command line but not in eclipse.
If you recently installed the package maybe just reconfigure your pydev (Window->Preferences->PyDev->Interpreters->Python Interpreter: Quick Auto Configure).
I am trying this program ( https://gist.github.com/eknowles/9939273) to work but when I put the code in PyCharm, it underlines json, requests and BeautifulSoup imports and it says "no module named beautifulsoup...".
Then I tried to install with "easy_install requests" or "easy_install json" but it spits this:
PS C:\Users\Ruzgar> easy_install json
Searching for json
Reading https://pypi.python.org/simple/json/
Couldn't find index page for 'json' (maybe misspelled?)
Scanning index of all packages (this may take a while)
Reading https://pypi.python.org/simple/
No local packages or download links found for json
error: Could not find suitable distribution for Requirement.parse('json')
How can I make this code work?
I understand that I have to fix this import problem first. (I use Python 2.5.4 by the way)
Python 2.5.4 I am using.
You need to update your version of Python. Requests and BeautifulSoup require Python version greater than 2.6; and json is also included in Python from version 2.6
I recommend you install the latest stable version of Python 2.7. You can find it at the download page.
I cheanged the project interpreter to Python 3.4. Now it gives me
different modules are not installed.
Use Python 2.7 as not everything is compatible with 3.4. Once you have downloaded and installed it, restart PyCharm and load your project again; then follow the following steps:
Click on File, then Settings
On the left, under 'Project Settings', click on Python Interpreter
On the right hand side, click on 'Configure Interpreters'
On the right, you will see a list of interpreters available on your system. The interpreter for your project will be highlighted. In the bottom half you will see the packages installed for the interpreter. Click on "Install".
You'll see a new window popup. This is a browser for the Python Package Index (PyPI). In the search box, type requests; when you see the results filtered, click on requests, and then click Install Package. Repeat this process for BeautifulSoup. Remember, you don't need to install json since its already included.
Click Apply, then OK to dismiss the window. Give PyCharm a few seconds to rebuild its cache, and everything should work.
download the package of beautiful soup （I think you should download
the Beautiful Soup 3 because your python version < 2.6）and install it;
if you want to know more about this package,see here
I am new to this platform as well as to Python scraping. I hope that my question will still be understandable and somebody can help me. Sorry, in case I make something unclear...
I have already checked other posts on a similar topic but could not manage to overcome my problem.
I am currently getting into web-scraping and wanted to try Scrapy. Therefore, I followed the installation instructions on the website. http://doc.scrapy.org/en/0.16/intro/install.html#intro-install
After I had figured out how it works I decided to run in a virtual environment.
I installed virtualenv and pip.
Then I installed Scrapy.
When I now want to start with the tutorial
scrapy startproject tutorial
I get the following error message:
File "/Users/XXX/environment_trial/bin/scrapy", line 3, in <module>
from scrapy.cmdline import execute
File "/Users/XXX/environment_trial/lib/python2.7/site-packages/scrapy/cmdline.py", line 7, in <module>
from scrapy.crawler import CrawlerProcess
File "/Users/XXX/environment_trial/lib/python2.7/site-packages/scrapy/crawler.py", line 3, in <module>
from twisted.internet import reactor, defer
ImportError: No module named twisted.internet
(environment_trial)XXX-iMac:~ XXX$
I could not find a Twisted.py on my Mac as suggested by other posts.
Can somebody please tell me what to do?
Simply put, you need to install twisted. You can get it from the download page. It looks like you'll need to install from source on a newer Mac, but that's just a case of extracting the tarball and running python setup.py install in the extracted folder.
edit: Since you already have pip installed, you can also grab twisted with it.
pip -E twisted_env install -U twisted
Please be sure that all the binaries that you are installing correspond to the exact same version of python that you have installed (For e.g python 2.7 ).
I did this mistake of installing pyopenSSL for python3.6 and it took me a lomng time to realize that versions did not match.