The code reaches out to a message board and indexes/reports the top topics. Using WING IDE, it works fine and reports no errors. However, when ran via command prompt, it will error out saying it can't properly encode a character. This is the first time I've seen this and haven't found a good resource to fix it.
Being that it runs fine in WING, I'm unsure what else to add to the code that would prevent this issue from happening in the command prompt.
import requests
from bs4 import BeautifulSoup
url = raw_input("Enter the board URL: ")
print "\n"
#send the HTTP request
response = requests.get(url)
if response.status_code == 200:
#pull the content
html_content = response.content
#send the page to BeautifulSoup
html_doc = BeautifulSoup(html_content, "html.parser")
#extract topic data
topic_spider = html_doc.find_all("span",{"class":"subject"})
data = []
for topic in topic_spider:
if topic.text!='':
data.append(topic.text)
topiclist = list(dict.fromkeys(data))
topiclist.sort(reverse=False)
for item in topiclist:
print ('[*] ' + item)
WING runs this just fine with no errors. Via CMD, the following result occurs after several successful results:
[*] Parenting (successful result)
Traceback (most recent call last):
File "D:\xxxx\topicindexer.py", line 29, in <module>
print ('[*] ' + item)
File "C:\Python27\lib\encodings\cp437.py", line 12, in encode
return codecs.charmap_encode(input,errors,encoding_map)
UnicodeEncodeError: 'charmap' codec can't encode character u'\u2019' in position 31: character maps to <undefined>
I note two things.
One, you use print statements like this
print ('[*] ' + item)
which indicates you're using python 3.x
Second, however, your cmd output uses python 2.7.
That appears to be your problem. Try python3 filename.py on the commandline instead of python filename.py, as that is what it defaults to when you have both installed.
See if this solves it before anything else.
Make sure the python environment in CMD and Wing is the same.
Set the environment variables that's in Wing IDE in CMD.
Looks like your code is written in python 3 but your default is set to python 2.
When running your code in CMD just add python3 myfile.py rather than just python myfile.py
Related
I'm working with Django and I'd created two database. Everything seems to work fine, but then I had to edit one of the two and add a column.. From that moment the db wouldn't work anymore, so I exported in a text file the first database and thinking "now I recreate the two db and run a python script to refill the first one". The problem is that whene I try to run the script I get errors, because I can't run the command like bash using os.system, and I don't really know any other way... So, here's my code:
import os
def func ():
try:
FILE=open ("./languagedb.txt", "r")
except IOError:
print 'Can\'t open db file'
exit (1)
for line in FILE:
if (line.startswith('INSERT')):
values=line[43:-1]
language=values[1:3]
values=values[6:]
field=""
fieldBool=True
i=0
while fieldBool:
try:
c=values[i]
except:
print ''
if c != '\'':
field=field+str(c)
i=i+1
else:
fieldBool=False
values=values [(i+3):]
text=""
textBool=True
i=0
while textBool:
try:
c=values[i]
except:
print ''
if c != '\'':
text=text+str(c)
i=i+1
else:
textBool=False
comand="Language.objects.create(language=\""+language+"\", text=\""+text+"\", campo=\""+field+"\")"
os.system(comand)
This is the way I call the shell:
python manage.py shell
and the commands I give it:
import django
from languageMods.models import *
import mymigration #The name fo the file containing the above code
mymigration.func()
And I get the following error, for example
sh: -c: line 0: syntax error near unexpected token `language="en",'
Which is shell's error.
Does someone know how to execute a command from a python script in a python shell?
If you start your script the way you describe it you can just call the django DB API directly in your code:
Language.objects.create(language=language, text=text, campo=field)
I am trying to return the output of a command to the IRC channel using Willie bot.
My code seems to work outputting my variable line by line, but for some reason once I utilize Willie bots say command to output to IRC it doesn't output anything.
Here is my code:
from willie import module
import subprocess
import urllib2
import os
#module.commands('splint')
def splint(bot, trigger):
bot.reply('I will process your request now!')
page = urllib2.urlopen(trigger.group(2))
page_content = page.read();
with open('codeToCheck.c', 'w') as code:
code.write(page_content)
command = 'splint "C:\Users\Justin\Desktop\codeToCheck.c"'
output = subprocess.Popen(command,shell=True, stdout=subprocess.PIPE,stderr=subprocess.PIPE).communicate()[0]
bot.say('I have saved the file successfully. Outputting:')
for i in output.splitlines():
bot.say(i)
bot.say(output)
Using my little test code here I have determined it works with print:
import subprocess,os
output = subprocess.Popen(["splint", "C:\cygwin64\home\Justin\codeToCheck.c"], stdout=subprocess.PIPE).communicate()[0]
command = 'splint "C:\Users\Justin\Desktop\codeToCheck.c"'
output = subprocess.Popen(command,shell=True, stdout=subprocess.PIPE,stderr=subprocess.PIPE).communicate()[0]
for i in output.splitlines():
print i
print 'I have saved the file successfully. Outputting:'
This is what the irc output looks like for my code:
<Fogest> .splint http://pastebin.com/raw.php?i=8cB7DdnQ
<Fogbot> Fogest: I will process your request now!
<Fogbot> I have saved the file successfully. Outputting:
There should be output, but there is nothing. Am I doing something wrong here? Running my test file (the test code I show on this post) via the command line I get the following output like I should:
$ python test.py
Splint 3.1.2 --- 25 Aug 2010
Finished checking --- no warnings
I have saved the file successfully. Outputting:
I switched the code to use the following instead:
process = subprocess.Popen(command,shell=True, stderr=subprocess.PIPE)
output = process.stderr.read()
The problem is now resolved.
I am brand new to Python and am trying to write a monitor to determine whether a Java web app (WAR file) running on localhost (hosted by Apache Tomcat) is running or not. I had earlier devised a script that ran:
ps -aef | grep myWebApp
And inspected the results of the grep to see if any process IDs came back in those results.
But it turns out that the host OS only sees the Tomcat process, not any web apps Tomcat is hosting. I next tried to see if Tomcat came with any kind of CLI that I could hit from the terminal, and it looks like the answer is no.
Now, I'm thinking of using wget or maybe even urllib2 to determine if my web app is running by having them hit http://localhost:8080/myWebApp and checking the results. Here is my best attempt with wget:
wgetCmd = "wget http://localhost:8080/myWebApp"
wgetResults = subprocess.check_output([wgetCmd], shell=True, stderr=subprocess.STDOUT)
for line in wgetResults.strip().split('\n'):
if 'failed' in line:
print "\nError: myWebApp is not running."
sys.exit()
My thinking here is that, if the web app isn't running, wget's output should always contain the word "failed" inside of it (at least, from my experience). Unfortunately, when I run this, I get the following error:
Traceback (most recent call last):
File "/home/myUser/mywebapp-mon.py", line 52, in <module>
main()
File "/home/myUser/mywebapp-mon.py", line 21, in main
wgetResults = subprocess.check_output([wgetCmd], shell=True, stderr=subprocess.STDOUT)
File "/usr/lib/python2.7/subprocess.py", line 544, in check_output
raise CalledProcessError(retcode, cmd, output=output)
subprocess.CalledProcessError: Command '['wget http://localhost:8080/myWebApp']' returned non-zero exit status 4
Any thoughts as to what's going on here (what the error is)? Also, and more importantly, am I going about this the wrong way? Thanks in advance!
I suggest you try the Requests module. It is much more user-friendly then wget or urllib. Try something like this:
import requests
r = requests.get('http://localhost:8080/myWebApp')
>>> r.status_code
200
>>> r.text
Some text of your webapp
*EDIT * installation instructions http://docs.python-requests.org/en/latest/user/install/
To check url:
import urllib2
def check(url):
try:
urllib2.urlopen(url).read()
except EnvironmentError:
return False
else:
return True
To investigate what kind of an error occurred you could look at the exception instance.
You can also build on using urllib2/requests and interact with Tomcat's manager service if it's installed. My using the list method you can receive the following information:
OK - Listed applications for virtual host localhost
/webdav:running:0
/examples:running:0
/manager:running:0
/:running:0
Based on Chapter 12 of the OTP in Action book and Cesarini's book I wrote this Erlang code:
Erlang:
p(Param) ->
?DBG("Starting~n", []),
Cmd = "python test.py",
Port = open_port({spawn,Cmd}, [stream,{line, 1024}, exit_status]),
?DBG("Opened the port: ~w~n", [Port]),
Payload = term_to_binary(list_to_binary(integer_to_list(Param))),
erlang:port_command(Port, Payload),
?DBG("Sent command to port: ~w~n", [Payload]),
?DBG("Ready to receive results for command: ~w~n", [Payload]),
receive
{Port, {data, Data}} ->
?DBG("Received data: ~w~n", [Data]),
{result, Text} = binary_to_term(Data),
Blah = binary_to_list(Text),
io:format("~p~n", [Blah]);
Other ->
io:format("Unexpected data: ~p~n", [Other])
end.
Python:
import sys
def main():
while True:
line = sys.stdin.readline().strip()
if line == "stop-good":
return 0
elif line == "stop-bad":
return 1
sys.stdout.write("Python got ")
sys.stdout.write(line)
sys.stdout.write("\n")
sys.stdout.flush()
if __name__ == "__main__":
sys.exit(main())
The Erlang code suspends at the recieve clause - it never gets any message.
I have also checked Python from a regular Linux shell - it prints out every user input (1 - "Python got 1").
Where is the mistake here? Why doesn't my Erlang code get anything back?
There are two points:
make sure that Python does not buffer your output, try running python -u in open_port
using term_to_binary/1 and binary_to_term/1 won't work, since they assume that Python is able to encode/decode Erlang External Term Format, which does not seem to be the case. If you want to go this route, check out ErlPort
Does your Param contain the command limiter for Python? (in this case I assume newline, "\n"). Also, list_to_binary/1 and then a term_to_binary/1 feels kinda wrong. term_to_binary/1 directly (including the newline) should be sufficient.
Is there any way to start up the Python interpreter from within a script , in a manner similar to just using python -i so that the objects/namespace, etc. from the current script are retained? The reason for not using python -i is that the script initializes a connection to an XML-RPC server, and I need to be able to stop the entire program if there's an error. I can't loop until there's valid input because apparently, I can't do something like this:
#!/usr/bin/python -i
# -*- coding: utf-8 -*-
import xmlrpclib
# Create an object to represent our server.
server_url = str(raw_input("Server: "))
while not server = xmlrpclib.Server(server_url):
print 'Unable to connect to server. Please try again'
else:
print 'Xmlrpclib.Server object `__main__.server\' of URL `', server_url, "' created"
break
# Python interpreter starts...
because:
% chmod u+x ./rpcclient.py
% ./rpclient.py
Traceback (most recent call last):
File "./rpcclient.py", line 8
while not server = xmlrpclib.Server(server_url):
^
SyntaxError: invalid syntax
>>>
Unfortunately, python -i starts the interpreter just after it prints out the traceback, so I somehow have to call the interactive interpreter - replacing the execution of the script so it retains the server connection - from within the script
Have you tried reading the error message? :)
= is assignment, you want the comparison operator == instead.
Well, I finally got it to work.
Basically, I put the entire try/except/else clause in a while True: loop, with the else suite being a break statement and the end of the except suite being a continue statement. The result is that it now continually loops if the user puts in an address that doesn't have a fully compliant XML-RPC2 server listening. Here's how it turned out:
#!/usr/bin/python -i
# -*- coding: utf-8 -*-
import xmlrpclib, socket
from sys import exit
# Create an object to represent our server.
#server = xmlrpclib.Server(server_url) and print 'Xmlrpclib.Server object `__main__.server\' of URL `', server_url, "' created"
server_url = str(raw_input("Server: "))
server = xmlrpclib.ServerProxy(server_url)
while True:
try:
server.system.listMethods()
except xmlrpclib.ProtocolError, socket.error:
print 'Unable to connect to server. Please try again'
server_url = str(raw_input("Server: "))
server = xmlrpclib.ServerProxy(server_url)
continue
except EOFError:
exit(1)
else:
break
print 'Xmlrpclib.Server object `__main__.server\' of URL `', server_url, "' created"
# Python interpreter starts...
Thank you very much!
...and I have to wait another day to accept this...