When I type dir(urllib) in the Python shell I get NameError: Name urllib not defined.
What is causing this?
It is because, you have to import urllib first. Try this one by one:
import urllib
dir(urllib)
Related
I am just getting my feet wet in the art of webscraping and I am following the tutorials from this source. For some reason I cannot import the error module from 'urllib' to handle exceptions. Since this is a built-in library, I am confused as to why this is an issue.
from urllib import urlopen
from urllib.error import HTTPError
from urllib.error import URLError
yields the error
ImportErrorTraceback (most recent call last)
<ipython-input-1-30b72b3bf2ea> in <module>()
1 from urllib import urlopen
----> 2 from urllib.error import HTTPError
3 from urllib.error import URLError
I have tried the same code with another IDE (IntelliJ) and it works as expected leading me to believe that this could be an issue with Google Colab itself. Could someone weight in and possibly help me find a solution to this problem.
I am new to programming, so if this is a juvenile question or if this is not the appropriate place for this question, I apologize in advance.
P.S. I have double checked that the runtime is Python 3
Your problem is in
from urllib import urlopen
Right way to import urlopen is from urllib.request
from urllib.request import urlopen
Docs
Just try this:
from urllib.request import urlopen
Always remember to try to search the docs of a particular library it helps a lot.
You are trying to run this code using Python 2. Use Python 3 and it will work.
Python 2:
>>> from urllib.error import HTTPError
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: No module named error
>>>
Python 3:
>>> from urllib.error import HTTPError
>>>
I am calling a function from functions.py into work.py, which works fine:
from functions import get_ad_page_urls
The get_ad_page_urls function makes use of a.o. the requests module.
Now, wether or not I import the requests module into work.py, when I run the called function in work.py, it gives an error: NameError: name 'requests' is not defined.
I have defined get_ad_page_urls in functions.py including the module, like so,
def get_ad_page_urls():
import requests
<rest of function>
or excluding the module, like so,
import requests
def get_ad_page_urls():
<rest of function>
but it doesn't matter, the NameError persists.
How should I write the function such that when I call the function in work.py everything works fine?
Traceback:
get_ad_page_urls(page_root_url)
Traceback (most recent call last):
File "<ipython-input-253-ac55b8b1e24c>", line 1, in <module>
get_ad_page_urls(page_root_url)
File "/Users/myname/Documents/RentIndicator/Python Code/idealista_functions.py", line 35, in get_ad_page_urls
NameError: name 'requests' is not defined
functions.py
import requests
import bs4
import re
from bs4 import BeautifulSoup
def get_ad_page_urls(page_root_url):
response = requests.get(page_root_url)
soup = bs4.BeautifulSoup(response.text)
container=soup.find("div",{"class":"items-container"})
return [link.get("href") for link in container.findAll("a", href=re.compile("^(/inmueble/)((?!:).)*$"))]
work.py
import requests
import bs4
import re
from bs4 import BeautifulSoup
from functions import get_ad_page_urls
city='Valencia'
lcity=city.lower()
root_url = 'https://www.idealista.com'
house_href='/alquiler-habitacion/'
page_root_url = root_url +house_href +lcity+ '-' + lcity + '/'
get_ad_page_urls(page_root_url)
Mine works perfectly fine running on python 3.4.4
functions.py
import requests
def get_ad_page_urls():
return requests.get("https://www.google.com")
work.py
from functions import get_ad_page_urls
print(get_ad_page_urls())
# outputs <Response [200]>
Make sure they are in the same directory. You might be using two different python versions and one of them doesn't have requests?
I have always used R and now trying to switch to Python.
I'm using Pycharm and found this error when running the following code:
import pandas as pd
from bs4 import BeautifulSoup
example1 = BeautifulSoup(train["review"][0],"lxml")
print (example1.get_text())
When I run it I have:
ImportError: cannot import name 'BeautifulSoup'
But I don't have any problem using the console. The rest of the code works fine both with the Run command/terminal and console.
Thank you for your help
Oh,I think you should check your file name or folder name.If there is a name that is already used in bs4 module,you will got ImportError.
Hope this helps.
I would like to use urllib.quote(). But python (python3) is not finding the module.
Suppose, I have this line of code:
print(urllib.quote("châteu", safe=''))
How do I import urllib.quote?
import urllib or
import urllib.quote both give
AttributeError: 'module' object has no attribute 'quote'
What confuses me is that urllib.request is accessible via import urllib.request
In Python 3.x, you need to import urllib.parse.quote:
>>> import urllib.parse
>>> urllib.parse.quote("châteu", safe='')
'ch%C3%A2teu'
According to Python 2.x urllib module documentation:
NOTE
The urllib module has been split into parts and renamed in Python 3 to
urllib.request, urllib.parse, and urllib.error.
If you need to handle both Python 2.x and 3.x you can catch the exception and load the alternative.
try:
from urllib import quote # Python 2.X
except ImportError:
from urllib.parse import quote # Python 3+
You could also use the python compatibility wrapper six to handle this.
from six.moves.urllib.parse import quote
urllib went through some changes in Python3 and can now be imported from the parse submodule
>>> from urllib.parse import quote
>>> quote('"')
'%22'
This is how I handle this, without using exceptions.
import sys
if sys.version_info.major > 2: # Python 3 or later
from urllib.parse import quote
else: # Python 2
from urllib import quote
Use six:
from six.moves.urllib.parse import quote
six will simplify compatibility problems between Python 2 and Python 3, such as different import paths.
so I have this code:
def crawl(self, url):
data = urllib.request.urlopen(url)
print(data)
but then when I call the function, it returns
data = urllib.request.urlopen(url)
AttributeError: 'module' object has no attribute 'request'
what did I do wrong? I already imported urllib..
using python 3.1.3
In python3, urllib is a package with three modules request, response, and error for its respective purposes.
Whenever you had import urllib or import urllib2 in Python2.
Replace them with
import urllib.request
import urllib.response
import urllib.error
The classs and methods are same.
BTW, use 2to3 tool if you converting from python2 to python3.
urllib.request is a separate module; import it explicitly.