I'm checking out how much of a performance increase I get after compiling a python script. After research looking into this issue I don't think I will actually see an increase in performance with the script I have written because I found out that once the script is loaded, the execution time doesn't increase. I still would like to know why this is failing to run after compiling as this is my first time trying this. Here is my script
#!/bin/python3
from datetime import datetime
start = datetime.now()
import psutil
BYTES_PER_GB = 1024*1024*1024
# Memory
m = psutil.virtual_memory()
#total = m.total/BYTES_PER_GB
#available = m.available/BYTES_PER_GB
#used = m.used/BYTES_PER_GB
m_free= m.free/BYTES_PER_GB
m_percent = m.percent
# Swap
s = psutil.swap_memory()
s_free = s.free/BYTES_PER_GB
s_percent = s.percent
print(' %.1fG (%.1f%%) %.1fG (%.1f%%)' % (m_free, m_percent, s_free, s_percent))
print(' %.1fG (%.1f%%) %.1fG (%.1f%%)' % (m_free, m_percent, s_free, s_percent))
print(datetime.now() - start)
I'm trying to compile with this line
python3 -m py_compile memory
In my print statements I have some special characters from font awesome. Not sure if that would cause a problem but if it doesn't show up correctly in my post then that's what that is.
The output when I try to run the compiled file is
./memorycpython-35.pyc: line 1: $'\026\r\r': command not found
./memorycpython-35.pyc: line 2: �k�W��#s�ddlmZej�ZddlZdZej�Zejeej: command not found
./memorycpython-35.pyc: line 3: syntax error near unexpected token `)'
./memorycpython-35.pyc: line 3: `ej
�Z
e
e je
j Ze�eej�e�dS)�)datetimeNiii#)rZnow�startZpsutilZ
BYTES_PER_GBZvirtual_memory�mZfreeZm_freeZpercentZ m_percentZ
swap_memory�sZs_freeZ s_percent�print�rr�memory<module>s
'
^[[?62;c^[[?62;c
EDIT
To narrow down the problem I wrote the following script
#!/bin/python3
print("Hello World!")
This is the output
./testcpython-35.pyc: line 1: $'\026\r\r': command not found
./testcpython-35.pyc: line 2: syntax error near unexpected token `)'
./testcpython-35.pyc: line 2: `�r�W%�#sed�dS)z
Hello World!N)�print�rr�./test<module>s'
Compiled using
python3 -m py_compile ./test
This creates a file in __pycache__/ called testcpython-35.pyc which I then do chmod +x testcpython-35.pyc and ./testcpython-35.pyc
It appears my issue is the ./testcpython-35.pyc part. When I run python3 testcpython-35.pyc, independent on whether I did chmod +x ./testcpython-35.pyc, the output is scrabbled. As long as I run the compiled program by first specifying what program to run it with, python3, it outputs Hello World! as expected.
Related
I am trying to run a python script within another python script. Which will run 10 times and produce 10 outputs.
I want to run program1.py inside program2.py. Now my program1.py was initially taking a C executable inside it and it takes 1 command line argument.
The program1.py looks like below:
import os
import sys
dataset = sys.argv[1]
os.system(f"/home/Dev/c4.5 -u -f {dataset}")
os.system(f"/home/Dev/c4.5rules -u -f {dataset}")
os.system(f"/home/Dev/c4.5rules -u -f {dataset} > Temp")
f = open('Temp')
# Some code
Where c4.5 and c4.5rules are the name of the executable files. To run this I was using python3 program1.py dataset_name
Now I am trying to put this program1.py inside program2.py and I am trying this below approach:
import os
import subprocess
# Some code
for track in range(0, 10):
with open(f'Train_{track}', 'r') as firstfile, open(f'DF_{track}.data', 'w') as secondfile:
for line in firstfile:
secondfile.write(line)
os.system("/home/Dev/program1.py DF_track")
#subprocess.Popen("/home/Dev/program1.py DF_track", shell=True)
Where I simply want to get the output of program1.py 10 times and want to use DF_track as the command line input for each output generation.
Using above approach I am getting lots of error. Please help.
Edit_1 :
Actually whenever I am trying to run, my cursor is not working, it is freezing, so unable to copy the errors.
Here are some of them :
1. attempt to perform an operation not allowed by security policy.
2. syntax error : word expected (expecting ")")
Imagine I have 2 files, the first file is a.py and the other is b.py and I want to call the a.py from b.py.
The content of a.py is:
print('this is the a.py file')
and the content of b.py is:
import os
stream = os.popen('python3 a.py')
output = stream.read()
print(output)
Now when I call b.py from terminal I get the output I expect which is a.py print statment
user#mos ~ % python3 b.py
this is the a.py file
You can do this with subprocess too instead of os module.
Here is a nice blog I found online where I got the code from: https://janakiev.com/blog/python-shell-commands/
See the example below.
a.py
def do_something():
pass
b.py
from a import do_something
do_something()
I'm working with Django and I'd created two database. Everything seems to work fine, but then I had to edit one of the two and add a column.. From that moment the db wouldn't work anymore, so I exported in a text file the first database and thinking "now I recreate the two db and run a python script to refill the first one". The problem is that whene I try to run the script I get errors, because I can't run the command like bash using os.system, and I don't really know any other way... So, here's my code:
import os
def func ():
try:
FILE=open ("./languagedb.txt", "r")
except IOError:
print 'Can\'t open db file'
exit (1)
for line in FILE:
if (line.startswith('INSERT')):
values=line[43:-1]
language=values[1:3]
values=values[6:]
field=""
fieldBool=True
i=0
while fieldBool:
try:
c=values[i]
except:
print ''
if c != '\'':
field=field+str(c)
i=i+1
else:
fieldBool=False
values=values [(i+3):]
text=""
textBool=True
i=0
while textBool:
try:
c=values[i]
except:
print ''
if c != '\'':
text=text+str(c)
i=i+1
else:
textBool=False
comand="Language.objects.create(language=\""+language+"\", text=\""+text+"\", campo=\""+field+"\")"
os.system(comand)
This is the way I call the shell:
python manage.py shell
and the commands I give it:
import django
from languageMods.models import *
import mymigration #The name fo the file containing the above code
mymigration.func()
And I get the following error, for example
sh: -c: line 0: syntax error near unexpected token `language="en",'
Which is shell's error.
Does someone know how to execute a command from a python script in a python shell?
If you start your script the way you describe it you can just call the django DB API directly in your code:
Language.objects.create(language=language, text=text, campo=field)
I am trying to write a python CLI program using module python cmd. When I try to execute another python script in my CLI program my objective is I have some python script in other folder and CLI program in other folder. I am trying to execute those python script using CLI program.
Below is the os.popen method used to execute other script there is CLI program:
import cmd
import os
import sys
class demo(cmd.Cmd):
def do_shell(self,line,args):
"""hare is function to execute the other script"""
output = os.popen('xterm -hold -e python %s' % args).read()
output(sys.argv[1])
def do_quit(self,line):
return True
if __name__ == '__main__':
demo().cmdloop()
and hare is error:
(Cmd) shell demo-test.py
Traceback (most recent call last):
File "bemo.py", line 18, in <module>
demo().cmdloop()
File "/usr/lib/python2.7/cmd.py", line 142, in cmdloop
stop = self.onecmd(line)
File "/usr/lib/python2.7/cmd.py", line 221, in onecmd
return func(arg)
TypeError: do_shell() takes exactly 3 arguments (2 given)
there is some link to other cmd CLI program
1 = cmd – Create line-oriented command processors
2 = Console built with Cmd object (Python recipe)
and some screen shot's for more information:
Please run above code in your system.
As specified in the doc:
https://pymotw.com/2/cmd/index.html
do_shell is defined as such:
do_shell(self, args):
But you are defining it as
do_shell(self, line, args):
I think the intended use is define it as specified from the documentation.
I ran your code and followed your example. I replicated your error. I then, as specified in the documentation for do_shell, I changed the method to the as expected:
do_shell(self, args):
From there, the sys module was missing, so you need to import that as well (unless you did not copy it from your source). After that, I got an error for index out of range, probably because of the expectation of extra parameters needing to be passed.
Furthermore, because you are talking about Python scripts, I don't see the need for the extra commands you are adding, I simply changed the line to this:
output = os.popen('python %s' % args).read()
However, if there is a particular reason you need the xterm command, then you can probably put that back and it will work for your particular case.
I also, did not see the use case for this:
output(sys.argv[1])
I commented that out. I ran your code, and everything worked. I created a test file that just did a simple print and it ran successfully.
So, the code actually looks like this:
def do_shell(self, args):
"""hare is function to execute the other script"""
output = os.popen('python %s' % args).read()
print output
The full code should look like this:
import cmd
import os
import sys
class demo(cmd.Cmd):
def do_shell(self, args):
"""hare is function to execute the other script"""
output = os.popen('python %s' % args).read()
print output
def do_quit(self,line):
return True
if __name__ == '__main__':
demo().cmdloop()
I have a problem I'd like you to help me to solve.
I am working in Python and I want to do the following:
call an SGE batch script on a server
see if it works correctly
do something
What I do now is approx the following:
import subprocess
try:
tmp = subprocess.call(qsub ....)
if tmp != 0:
error_handler_1()
else:
correct_routine()
except:
error_handler_2()
My problem is that once the script is sent to SGE, my python script interpret it as a success and keeps working as if it finished.
Do you have any suggestion about how could I make the python code wait for the actual processing result of the SGE script ?
Ah, btw I tried using qrsh but I don't have permission to use it on the SGE
Thanks!
From your code you want the program to wait for job to finish and return code, right? If so, the qsub sync option is likely what you want:
http://gridscheduler.sourceforge.net/htmlman/htmlman1/qsub.html
Additional Answer for an easier processing:
By using the python drmaa module : link which allows a more complete processing with SGE.
A functioning code provided in the documentation is here: [provided you put a sleeper.sh script in the same directory]
please notice that the -b n option is needed to execute a .sh script, otherwise it expects a binary by default like explained here
import drmaa
import os
def main():
"""Submit a job.
Note, need file called sleeper.sh in current directory.
"""
s = drmaa.Session()
s.initialize()
print 'Creating job template'
jt = s.createJobTemplate()
jt.remoteCommand = os.getcwd()+'/sleeper.sh'
jt.args = ['42','Simon says:']
jt.joinFiles=False
jt.nativeSpecification ="-m abe -M mymail -q so-el6 -b n"
jobid = s.runJob(jt)
print 'Your job has been submitted with id ' + jobid
retval = s.wait(jobid, drmaa.Session.TIMEOUT_WAIT_FOREVER)
print('Job: {0} finished with status {1}'.format(retval.jobId, retval.hasExited))
print 'Cleaning up'
s.deleteJobTemplate(jt)
s.exit()
if __name__=='__main__':
main()
I want to count how many lines of code I have written.
Here is the Python code:
import os
import sys
EXT = ['.c','.cpp','.java','.py']
def main():
l = []
if os.path.isdir(sys.argv[1]):
for root, dirs, files in os.walk(sys.argv[1]):
l.extend([os.path.join(root, name) for name in files])
else:
l.append(sys.argv[1])
params = ["'"+p+"'" for p in l if os.path.splitext(p)[1] in EXT]
result = os.popen("wc -l %s "%" ".join(params)).read()
print result
if __name__ == '__main__':
main()
Before this, it was running as expected. But today, it give me this error:
sh: 1: Syntax error: Unterminated quoted string
I don't know what happened.
Your Python script is missing a shebang line. Add the following to the top of your file:
#!/usr/bin/env python
Then you should be able to run the following, assuming your script is at /path/to/your_script.py and it has the executable bit set:
/path/to/your_script.py arg1 arg2 [...]
Alternatively:
python /path/to/your_script.py arg1 arg2 [...]
Update following comments
I suspect what has changed is that a source file containing a ' in its name has been added to the directory you are checking and the shell is choking on this.
You could add the following function to your program:
def shellquote(s):
return "'" + s.replace("'", "'\\''") + "'"
[Lifted from Greg Hewgill's answer to How to escape os.system() calls in Python? .]
And call it like this:
params = [shellquote(p) for p in l if os.path.splitext(p)[1] in EXT]
#Johnsyweb's updated answer seems to have the correct diagnostic, but the correct fix is to not use a shell to invoke wc. Try something like this instead:
cmd = ['/bin/wc', '-l'] # Need full path!
[cmd.extend(p) for p in l if os.path.splitext(p)[1] in EXT]
result = os.popen2(cmd).read()
Note that the subprocess module is the recommended solution now. Switching to that requires a less intrusive change to your current code, though; see http://docs.python.org/2/library/subprocess.html#replacing-os-popen-os-popen2-os-popen3
Looks like your Python program was parsed like a shell script. Add something like this at the header to indicate where your Python is:
#!/usr/bin/python
or you just run python a.py.