Develop Reference

scrapy command line in Azure Function App

I cannot get the following to work in a Python Function App (Azure):
subprocess.run(["scrapy"])
Why do I need this to work? I am using advertools (which runs that command, see https://github.com/eliasdabbas/advertools/blob/master/advertools/spider.py)
What are the issues:
First when deploying, the command line does not add the scrapy command line executable to the path
Deploying with oryx, the is an additional issue which oryx ads the wrong python interpreter to the scrapy executable (#!/tmp/orxy/.../python3)
What do I do try to fix this:
add the scrapy exec to my project: lib/advertools/scrapy_path/scrapy (with the correct path the python)
add that file to my path:
os.environ["PATH"] += os.pathsep + str(scrapy_bin_path)
What is the result:
running subprocess.run(["ls", '-la', str(scrapy_bin_path)], capture_output=True, text=True) returns:
CompletedProcess(args=['ls', '-la', '/home/site/wwwroot/lib/advertools/scrapy_path'], returncode=0, stdout='total 0\n-rwxr-xr-x 1 root root 230 Dec 2 10:10 scrapy\n', stderr='')
so file is present and executable
running subprocess.run(["which", "scrapy"], capture_output=True, text=True) returns:
CompletedProcess(args=['which', 'scrapy'], returncode=0, stdout='/home/site/wwwroot/lib/advertools/scrapy_path/scrapy\n', stderr='')
encouraging...
but now finally running subprocess.run(["scrapy"], capture_output=True, text=True) returns:
[Information] Traceback (most recent call last):
File "/home/site/wwwroot/lib/advertools/test.py", line 74, in exec
result_scrapy = subprocess.run(["scrapy"], capture_output=True, text=True)
File "/usr/local/lib/python3.9/subprocess.py", line 505, in run
with Popen(*popenargs, **kwargs) as process:
File "/usr/local/lib/python3.9/subprocess.py", line 951, in __init__
self._execute_child(args, executable, preexec_fn, close_fds,
File "/usr/local/lib/python3.9/subprocess.py", line 1821, in _execute_child
raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'scrapy'
file not found?!
I don't understand why it cannot find 'scrapy' if which scrapy does find it

Please check the below troubleshooting steps to fix the issue:
Try by creating the Virtual Environment > Install Scrapy > Do Next Steps
Scrapy is a part of the Spider framework. Reinstall the scrapy module and check once.
No such file or directory
This error comes when any files or directory are not found or accessible. We need to provide the absolute path to that module or file or directory.
Working directory might be changed sometime when you run the Python Script through Scrapy, try creating the breakpoint and check the present directory in terms of code perspective to find the root cause of the error.
Refer to this SO Thread for more detailed information on No Such File or directory: Scrapy Python issue and AppsLoveWorld article.

Python script executes from terminal, but not when using subprocess?

I'm using subprocess to execute a Python script called trace.py that is located in a different folder. The script trace.py then uses subprocess to run a traceroute command and then prints the output. When I go to the folder that trace.py is located in and type this in the terminal:
python trace.py
or
./trace.py
or from any location:
python /home/.../cgi-bin/trace.py
it works fine and the traceroute is printed to the terminal. However, when I try to execute trace.py from main.py by using subprocess, it doesn't seem to work. I've tested this by creating test.py and using subprocess to execute it from main.py and this works. I do this with the following:
output = subprocess.check_output([sys.executable, script_path])
Where script_path is the absolute path to trace.py.
The full error I get is this (paths are shortened):
Traceback (most recent call last):
File "/home/.../cgi-bin/trace.py", line 11, in <module>
traceroute = subprocess.check_output(["traceroute", "www.google.com"])
File "/usr/lib/python2.7/subprocess.py", line 566, in check_output
process = Popen(stdout=PIPE, *popenargs, **kwargs)
File "/usr/lib/python2.7/subprocess.py", line 710, in __init__
errread, errwrite)
File "/usr/lib/python2.7/subprocess.py", line 1327, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory
Traceback (most recent call last):
File "main.py", line 97, in <module>
serve(args.port, public_html, cgibin)
File "main.py", line 55, in serve
process = subprocess.check_output(["/usr/bin/python", script_path])
File "/usr/lib/python2.7/subprocess.py", line 573, in check_output
raise CalledProcessError(retcode, cmd, output=output)
subprocess.CalledProcessError: Command '['/usr/bin/python', '/home/.../cgi-bin/trace.py']' returned non-zero exit status 1
Why does this not work, but executing it from the terminal does?

The child can't find traceroute executable.
Compare os.environ['PATH'] in your shell with the value within runnning trace.py
Check file permissions -- whether it is readable and executable by the user that runs trace.py.

Unable to run shell script from the Pydev environment in Eclipse

I am using Centos 7.0 and have installed Eclipse Kepler in the Pydev environment. I want to run a simple c shell script through Python using subprocess as follows:
import subprocess
subprocess.call(["./test1.csh"])
This c shell script executes in the terminal and also if I run command like "ls" or ""pwd then I get the correct output e.g.
subprocess.call(["ls"]) # give me the list of all files
subprocess.call(["pwd"]) # gives me the location of current directory.
But when I run subprocess.call(["./test1.csh"]), I get the following error:
Traceback (most recent call last):
File "/home/nishant/workspace/codec_implement/src/NTTool/raw2waveconvert.py", line 8, in <module>
subprocess.call(["./test1.csh"])
File "/usr/lib64/python2.7/subprocess.py", line 524, in call
return Popen(*popenargs, **kwargs).wait()
File "/usr/lib64/python2.7/subprocess.py", line 711, in __init__
errread, errwrite)
File "/usr/lib64/python2.7/subprocess.py", line 1308, in _execute_child
raise child_exception
OSError: [Errno 13] Permission denied
Where am I going wrong? Please suggest

Make sure that the file test1.csh is executable. As Lukas Graf commented, also check the shebang (#!...) in the first line.
To confirm that, before run it through Python, run it in the shell.
$ ls -l test1.csh
...
$ ./test1.csh
The current working directory will be different from when you run it in the terminal. Specify the full path of the shell script. Or change the working directory configuration in the PyDev.
UPDATE
Prepend the shell executable:
import subprocess
subprocess.call(["csh", "./test1.csh"])

Is there a way to run Python's subprocess.check_output without the cwd to be the same directory as the exe being called

I am trying to automate some tasks - the process requires I call some exe's and pass parameters. The particular directories for the exe's are in the PATH variable for windows. However, I consistently get a
WindowsError: [Error 2] The system cannot find the file specified
My current workaround is to set the os.cwd to the directory with the exe but that imposes some other limits on how we distribute the code. I want to note that in every case if I start a cmd window and type the same code I am passing to subprocess.check_output the code works no matter what directory I am in on the computer.
Just to be clear I am afraid for example of trying to automate a WinRAR task and WinRAR.exe is in a different folder on their computer.
Okay in response to the comment below here is the input and the output after I changed the cwd to root (c:)
The call to subprocess
rarProcess = check_output('''WinRAR a -r -v700m -sfx -agYYYYMMDD-NN -iiconD:\\RarResources\\de96.ico -iimgd:\\RarResources\\accounting2013.bmp d:\\testFTP\\compressed_test_ d:\\files_to_compress''')
and here is the Traceback message in all of it's glory
Traceback (most recent call last):
File "<pyshell#93>", line 1, in <module>
rarProcess = check_output('''WinRAR a -r -v700m -sfx -agYYYYMMDD-NN -iiconD:\\RarResources\\de96.ico -iimgd:\\RarResources\\accounting2013.bmp d:\\testFTP\\compressed_test_ d:\\files_to_compress''')
File "C:\Program Files (x86)\python\lib\subprocess.py", line 537, in check_output
process = Popen(stdout=PIPE, *popenargs, **kwargs)
File "C:\Program Files (x86)\python\lib\subprocess.py", line 679, in __init__
errread, errwrite)
File "C:\Program Files (x86)\python\lib\subprocess.py", line 893, in _execute_child
startupinfo)
WindowsError: [Error 2] The system cannot find the file specified
Now I can't prove that this is not a hypothetical question/problem. I get the intended results when I use the same command (adjusting for path separators) through the cmd window and if I change the directory to the directory with the exe before running the command as pasted above.

You don't need to set os.cwd and run the process. Instead you pass the location of your "Winrar.exe" file to the subprocess as a dict.
proc = subprocess.Popen(args, env={'PATH': '/path/to/winrar.exe'})

Python process.call() error

I have OSX and am running the python script out of the Unix shell
I'm running a python code that should open an application. I've been testing with Firefox.app and have been getting
Traceback (most recent call last):
File "/Users/brennan/Desktop/Coding/Wilix/wilix.py", line 453, in <module>
subprocess.call(["open -a "+cp2])
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 493, in call
return Popen(*popenargs, **kwargs).wait()
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 679, in __init__
errread, errwrite)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 1228, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory
My code is:
subprocess.call(["open -a "+cp2])
where cp2 is user input. (Firefox in this case)
if I cd into the programs directory and then do
open -a Firefox
firefox opens fine.
if I change my code to
subprocess.call(["open -a Firefox"])
I still get the error message.

You're passing open -a Firefox as one argument, as if you ran this in the shell:
$ "open -a Firefox"
You need to split up the items:
subprocess.call(['open', '-a', 'Firefox'])

Try giving the full path of firefox app.

It's wrong to use subprocess.call without shell=True or providing command as a list. Please, take a look at first examples in the docs:
http://docs.python.org/2/library/subprocess.html
Full path to Firefox may be needed or may be not needed.