I am trying to run this app:
https://github.com/bmjr/guhTrends
I have python 2.7.x running the following script at command line. I reckon it was written using python3.x. What is deprecated in the code below?
import urllib
import json
import matplotlib.pyplot as plt
dates = urllib.request.urlopen('http://charts.spotify.com/api/tracks/most_streamed/global/weekly/')
dataDates = json.loads(dates.read().decode())
the error:
Traceback (most recent call last):
File "DataMining.py", line 6, in <module>
dates = urllib.request.urlopen('http://charts.spotify.com/api/tracks/most_streamed/global/weekly/')
AttributeError: 'module' object has no attribute 'request'
That script won't work under python2 as urllib of python2 has no request module.
Use urllib2.urlopen instead of urllib.request if you want start running that script under python2 .
To get python script work on bith (python2 and python3) use six module which is Python 2 and 3 Compatibility Library.
from six.moves import urllib
import json
import matplotlib.pyplot as plt
dates = urllib.request.urlopen('http://charts.spotify.com/api/tracks/most_streamed/global/weekly/')
dataDates = json.loads(dates.read().decode())
You're requesting a resource that is not currently available (I'm seeing a 504). Since this could potentially happen any time you request a remote service, always check the status code on the response; it's not that your code is necessarily wrong, in this case it's that you're assuming the response is valid JSON without checking whether the request was successful.
Check the urllib documentation to see how to do this (or, preferably, follow the recommendation at the top of that page and use the requests package instead).
Related
Good-day Folks,
I am writing a small Python script that will be used on a Ubiquiti EdgeRouter 12P to make an API Call, using Digest Authentication, against another router for some JSON data. This is my first attempt at writing a Python script and I have been able to do this using the Requests module, but only in a Python v3.x.y environment. As I was writing my script, I discovered that the Python version on the EdgeRouter is v2.7.13 and the Requests module isn't installed/loaded.
I have attempted to do a pip install requests but it fails with an invalid syntax error message. So with my limited knowledge, I can't figure out what my options are now. Googled around a bit and saw references to using UrlLib or UrlLib2 - but I'm struggling with figuring out how to use either to make my API Call using Digest Authentication.
I would very much like to stick to the Requests module, as it appears to be the simplest and cleanest approach. Below is a snippet of my code, any help would be really appreciated, thanks.
PS. My script is not yet finished, as I'm still learning how to parse the response data received.
HERE'S MY SCRIPT
#PYTHON MODULE IMPORTS
import sys #Import the Python Sys module - used later to detect the Python version
PythonVersion = sys.version_info.major #Needed to put this global variable up here, so I can use it early
import json #Import the JSON module - used to print the API Call response content in JSON format
if PythonVersion == 3: #Import the REQUESTS and HTTPDigestAuth modules - used to make API Calls if we detect Python version 3
import requests
from requests.auth import HTTPDigestAuth
if PythonVersion == 2: #Import the UrlLib module - used to make API Calls if we detect Python version 2
import urllib
#GLOBAL VARIABLES
NodeSerial = '1234'
NodeIP = '192.168.1.100'
BasePath = 'api/v0.1'
apiURL = f"http://{NodeIP}/{BasePath}/MagiNodes"
#Define the credentials to use
Username='admin'
Passwd='mypassword'
#MAIN ROUTINE
print("This is a test script")
if PythonVersion == 2:
print("Python Version 2 detected")
elif PythonVersion == 3:
print("Python Version 3 detected")
print(f"Now querying MagiLink-{NodeSerial}...")
Query1 = requests.get(f"{apiURL}/{NodeSerial}", auth=HTTPDigestAuth(Username, Passwd)) #Using an F string
pretty_Query1 = json.dumps(Query1.json(), indent=3) #Pretty printing the response content
print(pretty_Query1)
I have about a dozen python module imports that are going to be reused on many different scrapers, and I would love to just throw them into a single file (scraper_functions.py) that also contains a bunch of functions, like this:
import smtplib
import requests
import re
from urllib.request import urlopen
from bs4 import BeautifulSoup
import time
def function_name(var1)
# function code here
then in my scraper I would simply do something like:
import scraper_functions
and be done with it. But listing the imports at the top of scraper_functions.py doesn't work, and neither does putting all the imports in a function. In each case I get errors in the scraper that is doing the importing.
Traceback (most recent call last):
File "{actual-scraper-name-here}.py", line 24, in <module>
x = requests.get(main_url)
NameError: name 'requests' is not defined
In addition, in VSCode, under Problems, I get errors like
Undefined variable 'requests' pylint(undefined-variable) [24,5]
None of the modules are recognized. I have made sure that all files are in the same directory.
Is such a thing possible please?
You need to either use the scraper_functions prefix (same way you do this import name) or use the from keyword to import your things from scraper_functions with the * selector.
Using the form keyword (Recommended)
from scraper_functions import * # import everything with *
...
x = requests.get(main_url)
Using the scraper_functions prefix (Not recommended)
import scraper_functions
...
x = scraper_functions.requests.get(main_url)
I just installed caldav 0.5.0 using pip on Windows. I tried to use this code from the documentation:
from datetime import datetime
import caldav
from caldav.elements import dav, cdav
# Caldav url
url = "https://user:pass#hostname/caldav.php/"
client = caldav.DAVClient(url)
But I get this error:
AttributeError: module 'caldav' has no attribute 'DAVClient'
Does someone know what causes this issue?
It is because your file is named calendar.py, which causes some kind of collision somewhere. Renaming your file to something else will do the trick (it did for me).
I am using Pickle in Python2.7. I am getting error while using cPickle.load() method. The code and error is shown below. Can someone guide me through this?
Code:
#! usr/bin/python
import cPickle
fo = open('result','rb')
dict1 = cPickle.load(fo)
Error:
Traceback (most recent call last):
File "C:\Python27\test.py", line 7, in <module>
dicts = cPickle.load(fo)
ImportError: No module named options
It seems like you can not do
import options
but when you or someone else did
cpickle.dump(xxx, open('result', 'rb'))
there was an object with a class or function of a module options that existed at this point in time, in xxx.
Solution
You can open the file binarily and replace options with the module you replaced the old module options with.
You probably created the file in your package like in module package.main by executing the file main.py or something like it, having a module options in the same directory.
Now you do import package.main, try to read the file and options is now called package.options and the module options can not be found.
How did you create this file? How do you load it now? cPickle/pickle does not transfer source code - so if you use a function you need the module when you load it.
so I have this code:
def crawl(self, url):
data = urllib.request.urlopen(url)
print(data)
but then when I call the function, it returns
data = urllib.request.urlopen(url)
AttributeError: 'module' object has no attribute 'request'
what did I do wrong? I already imported urllib..
using python 3.1.3
In python3, urllib is a package with three modules request, response, and error for its respective purposes.
Whenever you had import urllib or import urllib2 in Python2.
Replace them with
import urllib.request
import urllib.response
import urllib.error
The classs and methods are same.
BTW, use 2to3 tool if you converting from python2 to python3.
urllib.request is a separate module; import it explicitly.