import os
import subprocess
pathName = "FilePath"
os.chdir(r'Directory Path')
process = subprocess.Popen(["scrapy", "crawl", "homeDepotSpider", "-t" , "csv" , "-o", pathName])
Here's the error message I get:
"/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 394, in __init__ errread, errwrite) File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 1047, in _execute_child raise child_exception OSError: [Errno 2] No such file or directory
I can't really find any documentation online, but is Platypus just not compatible with subprocess or is there something wrong with my code. When I compile it runs fine, when I create an .app file it doesnt work.
Edit: Here is the software that I used to make my program into an executable: https://sveinbjorn.org/platypus
The Exception says the reason: "No such file"
What subprocess.Popen line is trying to do is roughly the same as running scrapy in a cmd prompt. That is (in Windows terms), in the PATH of the subprocess you are trying to launch there is no folder containing 'scrapy.exe' .
I read that you used some packaging SW to bundle your script. The packaging program has not packaged the scrapy executable.
Related
I cannot get the following to work in a Python Function App (Azure):
subprocess.run(["scrapy"])
Why do I need this to work? I am using advertools (which runs that command, see https://github.com/eliasdabbas/advertools/blob/master/advertools/spider.py)
What are the issues:
First when deploying, the command line does not add the scrapy command line executable to the path
Deploying with oryx, the is an additional issue which oryx ads the wrong python interpreter to the scrapy executable (#!/tmp/orxy/.../python3)
What do I do try to fix this:
add the scrapy exec to my project: lib/advertools/scrapy_path/scrapy (with the correct path the python)
add that file to my path:
os.environ["PATH"] += os.pathsep + str(scrapy_bin_path)
What is the result:
running subprocess.run(["ls", '-la', str(scrapy_bin_path)], capture_output=True, text=True) returns:
CompletedProcess(args=['ls', '-la', '/home/site/wwwroot/lib/advertools/scrapy_path'], returncode=0, stdout='total 0\n-rwxr-xr-x 1 root root 230 Dec 2 10:10 scrapy\n', stderr='')
so file is present and executable
running subprocess.run(["which", "scrapy"], capture_output=True, text=True) returns:
CompletedProcess(args=['which', 'scrapy'], returncode=0, stdout='/home/site/wwwroot/lib/advertools/scrapy_path/scrapy\n', stderr='')
encouraging...
but now finally running subprocess.run(["scrapy"], capture_output=True, text=True) returns:
[Information] Traceback (most recent call last):
File "/home/site/wwwroot/lib/advertools/test.py", line 74, in exec
result_scrapy = subprocess.run(["scrapy"], capture_output=True, text=True)
File "/usr/local/lib/python3.9/subprocess.py", line 505, in run
with Popen(*popenargs, **kwargs) as process:
File "/usr/local/lib/python3.9/subprocess.py", line 951, in __init__
self._execute_child(args, executable, preexec_fn, close_fds,
File "/usr/local/lib/python3.9/subprocess.py", line 1821, in _execute_child
raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'scrapy'
file not found?!
I don't understand why it cannot find 'scrapy' if which scrapy does find it
Please check the below troubleshooting steps to fix the issue:
Try by creating the Virtual Environment > Install Scrapy > Do Next Steps
Scrapy is a part of the Spider framework. Reinstall the scrapy module and check once.
No such file or directory
This error comes when any files or directory are not found or accessible. We need to provide the absolute path to that module or file or directory.
Working directory might be changed sometime when you run the Python Script through Scrapy, try creating the breakpoint and check the present directory in terms of code perspective to find the root cause of the error.
Refer to this SO Thread for more detailed information on No Such File or directory: Scrapy Python issue and AppsLoveWorld article.
I have a python script that uses a package called flopy. My script generates a series of inputs to a fortran executable. Flopy writes these into text files and then calls the fortran executable, which uses the text files to run a model.
I'm using a mac (OSX) and I downloaded python 2.7 from python.org- i.e. I'm not using the Apple system version of python. The version of python I'm using is in Library/Frameworks/Python.Frameworks/
I can run my script if I call it from the Terminal window (by typing:
Python myscriptname.py
However if I run my script through IDLE (the version that came with python which I downloaded it) it returns an error:
Traceback (most recent call last):
File "/Users/neilthomas/RotatedModel_v4_Tr_mfnwt.py", line 355, in <module>
success, mfoutput = mf.run_model(silent=False, pause=False)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/flopy/mbase.py", line 638, in run_model
normal_msg=normal_msg)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/flopy/mbase.py", line 1034, in run_model
stdout=sp.PIPE, stderr=sp.STDOUT, cwd=model_ws)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 710, in __init__
errread, errwrite)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 1335, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory
The file 'mfnwt' absolutely does exist. I'm sure I'm missing something obvious, but is there something I need to do to allow IDLE to run programs/subprocesses via the shell it uses? Thanks.
The problem here is that you have to identify the specific MODFLOW executable file you are calling ('mfnwt' in your case). I do the same with a MODFLOW 2000 file:
mf = flopy.modflow.mf.Modflow(modelname,namefile_ext='nam',version='mf2k',exe_name='/home/MODFLOW-and-related-codes/build-08/bin-windows/mf2k.exe')
In your case, you would do something similar, only replacing the version='mf2k' and exe_name=path to match where you are storing your MODFLOW file.
See the documentation for further details: https://modflowpy.github.io/flopydoc/mf.html
I am trying to convert a pdf document to text document using pdftotext software.
I need to call this application inc command prompt from python script to convert the file.
I have following code:
import os
import subprocess
path = "C:\\Users\\..."
pdffname = "pdffilename.pdf"
txtfname = "txtfilename.txt"
subprocess.call(['pdftotext', '-layout',
os.path.join(path, pdffname),
os.path.join(path, txtfname)])
When I run this code, I get error
File "C:/Users/.../code-1.py", line 44, in <module>
os.path.join(path, txtfname)])
File "C:\Anaconda\lib\subprocess.py", line 522, in call
return Popen(*popenargs, **kwargs).wait()
File "C:\Anaconda\lib\subprocess.py", line 710, in __init__
errread, errwrite)
File "C:\Anaconda\lib\subprocess.py", line 958, in _execute_child
startupinfo)
WindowsError: [Error 2] The system cannot find the file specified
Can you help to call pdftotext application from python to convert pdf to text file.
I had this same error, except with Popen. I fixed it by providing the full path to pdftotext.exe in the subprocess call. Don't forget to escape your backslashes.
I do not know much about Anaconda, and I have not tested this myself, but I believe Conda may have an issue referencing scripts on Windows: fix references to scripts on windows
I have a Python script that moves a file from a server to HDFS using the following command:
from subprocess import Popen
.
.
filename = "/home/user123/test.csv"
put = Popen(["hadoop", "fs", "-put", filename, "/data/test"])
The script works perfectly when I run it on the server, but when I run it as a cron job it fails with the following message:
OSError: [Errno2] no such file or directory
.
.
File "/opt/anaconda/lib/python2.7/subprocess.py", line 710, in __init__
errread, errwrite)
File "/opt/anaconda/lib/python2.7/subprocess.py, line 1335, in _execute_child
raise child_exception
I tried replacing "hadoop" with "/usr/bin/hadoop" but I still get a "No such file or directory" error. I assume this is an environment variables issue but is there a way to get around it?
I am using Centos 7.0 and have installed Eclipse Kepler in the Pydev environment. I want to run a simple c shell script through Python using subprocess as follows:
import subprocess
subprocess.call(["./test1.csh"])
This c shell script executes in the terminal and also if I run command like "ls" or ""pwd then I get the correct output e.g.
subprocess.call(["ls"]) # give me the list of all files
subprocess.call(["pwd"]) # gives me the location of current directory.
But when I run subprocess.call(["./test1.csh"]), I get the following error:
Traceback (most recent call last):
File "/home/nishant/workspace/codec_implement/src/NTTool/raw2waveconvert.py", line 8, in <module>
subprocess.call(["./test1.csh"])
File "/usr/lib64/python2.7/subprocess.py", line 524, in call
return Popen(*popenargs, **kwargs).wait()
File "/usr/lib64/python2.7/subprocess.py", line 711, in __init__
errread, errwrite)
File "/usr/lib64/python2.7/subprocess.py", line 1308, in _execute_child
raise child_exception
OSError: [Errno 13] Permission denied
Where am I going wrong? Please suggest
Make sure that the file test1.csh is executable. As Lukas Graf commented, also check the shebang (#!...) in the first line.
To confirm that, before run it through Python, run it in the shell.
$ ls -l test1.csh
...
$ ./test1.csh
The current working directory will be different from when you run it in the terminal. Specify the full path of the shell script. Or change the working directory configuration in the PyDev.
UPDATE
Prepend the shell executable:
import subprocess
subprocess.call(["csh", "./test1.csh"])