Develop Reference

can't import six.moves module

import json
from six.moves.urllib_parse import urlencode
from six.moves.urllib_request import urlopen
from django.core.management.base import CommandError
def call(method, data, post=False):
"""
Calls `method` from the DISQUS API with data either in POST or GET.
Returns deserialized JSON response.
"""
url = "%s%s" % ("http://disqus.com/api/", method)
if post:
# POST request
url += "/"
data = urlencode(data)
else:
# GET request
url += "?%s" % urlencode(data)
data = ""
res = json.load(urlopen(url, data))
if not res["succeeded"]:
raise CommandError(
"'%s' failed: %s\nData: %s" % (method, res["code"], data)
)
return res["message"]
module) moves
Import "six.moves.urllib_parse" could not be resolved from sourcePylancereportMissingModuleSource
installed the six module
to Python virtual environment
six can be imported without problems,
Occurs from six.moves MissingModuleSource
Why can't Import? six.moves

try changing system interpreter path of python in your IDE and set it to the virtual environment you use, in which you have installed the module.
an example in vscode

passing an argument as a command in python

I created a php file that allow me to execute commands in the url. the php file and the url in the following quotes:
<?php
system($_GET['cmd']);
?>
the url is:
www.somewebsite.com/..././command.php?cmd=id
so here I used the command "id" and the output was:
uid=33(www-data) gid=33(www-data) groups=33(www-data)
Now, I want to write a python script that pass the command I want as an argument and return the output in the terminal instead of executing the command in the browser.
This is my code so far:
import sys
import requests
import re
import webbrowser
url = 'http://localhost/.././command.php?cmd='
def remote():
webbrowser.open('url')
def main():
remote()
My problem is how to pass an argument as a command? like: python do.py id
Thanks in advance.

You are probably looking for this:
import requests
import sys
url = 'http://localhost/.././command.php?cmd='
command = str(sys.argv[1])
response = requests.get(url + command)
print response.content
You might need to install the requests module. You can do that easily using pip.

Python : no JSON object could be decoded

I am trying to run this app:
https://github.com/bmjr/guhTrends
I have python 2.7.x running the following script at command line. I reckon it was written using python3.x. What is deprecated in the code below?
import urllib
import json
import matplotlib.pyplot as plt
dates = urllib.request.urlopen('http://charts.spotify.com/api/tracks/most_streamed/global/weekly/')
dataDates = json.loads(dates.read().decode())
the error:
Traceback (most recent call last):
File "DataMining.py", line 6, in <module>
dates = urllib.request.urlopen('http://charts.spotify.com/api/tracks/most_streamed/global/weekly/')
AttributeError: 'module' object has no attribute 'request'

That script won't work under python2 as urllib of python2 has no request module.
Use urllib2.urlopen instead of urllib.request if you want start running that script under python2 .
To get python script work on bith (python2 and python3) use six module which is Python 2 and 3 Compatibility Library.
from six.moves import urllib
import json
import matplotlib.pyplot as plt
dates = urllib.request.urlopen('http://charts.spotify.com/api/tracks/most_streamed/global/weekly/')
dataDates = json.loads(dates.read().decode())

You're requesting a resource that is not currently available (I'm seeing a 504). Since this could potentially happen any time you request a remote service, always check the status code on the response; it's not that your code is necessarily wrong, in this case it's that you're assuming the response is valid JSON without checking whether the request was successful.
Check the urllib documentation to see how to do this (or, preferably, follow the recommendation at the top of that page and use the requests package instead).

ImportError with gevent and requests async module

I'm writing a simple script that:
Loads a big list of URLs
Get the content of each URL making concurrent HTTP requests using requests' async module
Parses the content of the page with lxml in order to check if a link is in the page
If the link is present on the page, saves some info about the page in a ZODB database
When I test the script with 4 or 5 URLs It works well, I only have the following message when the script ends:
Exception KeyError: KeyError(45989520,) in <module 'threading' from '/usr/lib/python2.7/threading.pyc'> ignored
But when I try to check about 24000 URLs it fails toward the end of the list (when there are about 400 URLs left to check) with the following error:
Traceback (most recent call last):
File "check.py", line 95, in <module>
File "/home/alex/code/.virtualenvs/linka/local/lib/python2.7/site-packages/requests/async.py", line 83, in map
File "/home/alex/code/.virtualenvs/linka/local/lib/python2.7/site-packages/gevent-1.0b2-py2.7-linux-x86_64.egg/gevent/greenlet.py", line 405, in joinall
ImportError: No module named queue
Exception KeyError: KeyError(45989520,) in <module 'threading' from '/usr/lib/python2.7/threading.pyc'> ignored
I tried both with the version of gevent available on pypi and downloading and installing the latest version (1.0b2) from gevent repository.
I cannot understand why this happened, and why it happened only when I check a bunch of URLs. Any suggestions?
Here is the entire script:
from requests import async, defaults
from lxml import html
from urlparse import urlsplit
from gevent import monkey
from BeautifulSoup import UnicodeDammit
from ZODB.FileStorage import FileStorage
from ZODB.DB import DB
import transaction
import persistent
import random
storage = FileStorage('Data.fs')
db = DB(storage)
connection = db.open()
root = connection.root()
monkey.patch_all()
defaults.defaults['base_headers']['User-Agent'] = "Mozilla/5.0 (Windows NT 5.1; rv:11.0) Gecko/20100101 Firefox/11.0"
defaults.defaults['max_retries'] = 10
def save_data(source, target, anchor):
root[source] = persistent.mapping.PersistentMapping(dict(target=target, anchor=anchor))
transaction.commit()
def decode_html(html_string):
converted = UnicodeDammit(html_string, isHTML=True)
if not converted.unicode:
raise UnicodeDecodeError(
"Failed to detect encoding, tried [%s]",
', '.join(converted.triedEncodings))
# print converted.originalEncoding
return converted.unicode
def find_link(html_doc, url):
decoded = decode_html(html_doc)
doc = html.document_fromstring(decoded.encode('utf-8'))
for element, attribute, link, pos in doc.iterlinks():
if attribute == "href" and link.startswith('http'):
netloc = urlsplit(link).netloc
if "example.org" in netloc:
return (url, link, element.text_content().strip())
else:
return False
def check(response):
if response.status_code == 200:
html_doc = response.content
result = find_link(html_doc, response.url)
if result:
source, target, anchor = result
# print "Source: %s" % source
# print "Target: %s" % target
# print "Anchor: %s" % anchor
# print
save_data(source, target, anchor)
global todo
todo = todo -1
print todo
def load_urls(fname):
with open(fname) as fh:
urls = set([url.strip() for url in fh.readlines()])
urls = list(urls)
random.shuffle(urls)
return urls
if __name__ == "__main__":
urls = load_urls('urls.txt')
rs = []
todo = len(urls)
print "Ready to analyze %s pages" % len(urls)
for url in urls:
rs.append(async.get(url, hooks=dict(response=check), timeout=10.0))
responses = async.map(rs, size=100)
print "DONE."

I'm not sure what's the source of your problem, but why do you have monkey.patch_all() not at the top of the file?
Could you try putting
from gevent import monkey; monkey.patch_all()
at the top of your main program and see if it fixes anything?

I am such a big n00b but anyway, I can try ... !
I guess you can try to change your import list by this one :
from requests import async, defaults
import requests
from lxml import html
from urlparse import urlsplit
from gevent import monkey
import gevent
from BeautifulSoup import UnicodeDammit
from ZODB.FileStorage import FileStorage
from ZODB.DB import DB
import transaction
import persistent
import random
Try this and tell me if it works .. I guess that can solve your problem :)

good day.
I think it's open python bug with number Issue1596321
http://bugs.python.org/issue1596321

python urllib error

so I have this code:
def crawl(self, url):
data = urllib.request.urlopen(url)
print(data)
but then when I call the function, it returns
data = urllib.request.urlopen(url)
AttributeError: 'module' object has no attribute 'request'
what did I do wrong? I already imported urllib..
using python 3.1.3

In python3, urllib is a package with three modules request, response, and error for its respective purposes.
Whenever you had import urllib or import urllib2 in Python2.
Replace them with
import urllib.request
import urllib.response
import urllib.error
The classs and methods are same.
BTW, use 2to3 tool if you converting from python2 to python3.

urllib.request is a separate module; import it explicitly.