Develop Reference

scrapy command line in Azure Function App

I cannot get the following to work in a Python Function App (Azure):
subprocess.run(["scrapy"])
Why do I need this to work? I am using advertools (which runs that command, see https://github.com/eliasdabbas/advertools/blob/master/advertools/spider.py)
What are the issues:
First when deploying, the command line does not add the scrapy command line executable to the path
Deploying with oryx, the is an additional issue which oryx ads the wrong python interpreter to the scrapy executable (#!/tmp/orxy/.../python3)
What do I do try to fix this:
add the scrapy exec to my project: lib/advertools/scrapy_path/scrapy (with the correct path the python)
add that file to my path:
os.environ["PATH"] += os.pathsep + str(scrapy_bin_path)
What is the result:
running subprocess.run(["ls", '-la', str(scrapy_bin_path)], capture_output=True, text=True) returns:
CompletedProcess(args=['ls', '-la', '/home/site/wwwroot/lib/advertools/scrapy_path'], returncode=0, stdout='total 0\n-rwxr-xr-x 1 root root 230 Dec 2 10:10 scrapy\n', stderr='')
so file is present and executable
running subprocess.run(["which", "scrapy"], capture_output=True, text=True) returns:
CompletedProcess(args=['which', 'scrapy'], returncode=0, stdout='/home/site/wwwroot/lib/advertools/scrapy_path/scrapy\n', stderr='')
encouraging...
but now finally running subprocess.run(["scrapy"], capture_output=True, text=True) returns:
[Information] Traceback (most recent call last):
File "/home/site/wwwroot/lib/advertools/test.py", line 74, in exec
result_scrapy = subprocess.run(["scrapy"], capture_output=True, text=True)
File "/usr/local/lib/python3.9/subprocess.py", line 505, in run
with Popen(*popenargs, **kwargs) as process:
File "/usr/local/lib/python3.9/subprocess.py", line 951, in __init__
self._execute_child(args, executable, preexec_fn, close_fds,
File "/usr/local/lib/python3.9/subprocess.py", line 1821, in _execute_child
raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'scrapy'
file not found?!
I don't understand why it cannot find 'scrapy' if which scrapy does find it

Please check the below troubleshooting steps to fix the issue:
Try by creating the Virtual Environment > Install Scrapy > Do Next Steps
Scrapy is a part of the Spider framework. Reinstall the scrapy module and check once.
No such file or directory
This error comes when any files or directory are not found or accessible. We need to provide the absolute path to that module or file or directory.
Working directory might be changed sometime when you run the Python Script through Scrapy, try creating the breakpoint and check the present directory in terms of code perspective to find the root cause of the error.
Refer to this SO Thread for more detailed information on No Such File or directory: Scrapy Python issue and AppsLoveWorld article.

Creating an .app file using Platypus failing with subprocess

import os
import subprocess
pathName = "FilePath"
os.chdir(r'Directory Path')
process = subprocess.Popen(["scrapy", "crawl", "homeDepotSpider", "-t" , "csv" , "-o", pathName])
Here's the error message I get:
"/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 394, in __init__ errread, errwrite) File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 1047, in _execute_child raise child_exception OSError: [Errno 2] No such file or directory
I can't really find any documentation online, but is Platypus just not compatible with subprocess or is there something wrong with my code. When I compile it runs fine, when I create an .app file it doesnt work.
Edit: Here is the software that I used to make my program into an executable: https://sveinbjorn.org/platypus

The Exception says the reason: "No such file"
What subprocess.Popen line is trying to do is roughly the same as running scrapy in a cmd prompt. That is (in Windows terms), in the PATH of the subprocess you are trying to launch there is no folder containing 'scrapy.exe' .
I read that you used some packaging SW to bundle your script. The packaging program has not packaged the scrapy executable.

Subprocess No such file or directory error

As part of larger code, I am trying to make a function that calls to a latexmk compiler using subprocess, but I consistently get FileNotFoundError: [Errno 2] No such file or directory: 'latexmk': 'latexmk'
However, If I write the command directly in the terminal, everything works: latexmk --pdf test.tex
In case it is important, I am on MacOS Mojave 10.14.6, running python 3.6 spyder through anaconda
I have checked the following links:
https://askubuntu.com/questions/801493/python-subprocess-call-not-working-as-expected
OSError: [Errno 2] No such file or directory while using python subprocess in Django
Running Bash commands in Python
If anything there solves the problem, I missed it.
To make everyone's life easier, here's a link to a .tex file [you can use your own]:
https://drive.google.com/open?id=1DoJnvg2BmbRCzmRmqFYRVybyTQUtyS-h
Afer putting type latexmk to terminal it outputs:
latexmk is hashed (/Library/TeX/texbin/latexmk)
Here is the minimal reproducible example (you do need latexmk on your computer though):
import os, subprocess
def pdf(file_path):
cur_dir = os.getcwd()
dest_dir = os.path.dirname(file_path)
basename = os.path.basename(file_path)
os.chdir(dest_dir)
main_arg = [basename]
command = ["latexmk", "--pdf"] + main_arg
try:
output = subprocess.check_output(command)
except subprocess.CalledProcessError as e:
print(e.output.decode())
raise
os.chdir(cur_dir)
pdf("path to your .tex file")
I have a feeling that I am grossly misunderstanding the way subprocess works. Any ideas?
Update: In case neccessary, the full traceback:
Traceback (most recent call last):
File "<ipython-input-90-341a2810ccbf>", line 1, in <module>
pdf('/Users/sergejczan/Desktop/untitled folder/test.tex')
File "/Users/sergejczan/Desktop/Lab/subprocess error reproduction.py", line 23, in pdf
output = subprocess.check_output(command)
File "/anaconda3/lib/python3.6/subprocess.py", line 336, in check_output
**kwargs).stdout
File "/anaconda3/lib/python3.6/subprocess.py", line 403, in run
with Popen(*popenargs, **kwargs) as process:
File "/anaconda3/lib/python3.6/subprocess.py", line 709, in __init__
restore_signals, start_new_session)
File "/anaconda3/lib/python3.6/subprocess.py", line 1344, in _execute_child
raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'latexmk': 'latexmk'
New Update
Changing the output = subprocess.check_output(command) line with the hardcoded envirnoment that I got from echo $PATH worked wonderfully.
output = subprocess.check_output(command,env = {'PATH': '/anaconda3/bin:/Users/sergejczan/anaconda3/bin:/Users/sergejczan/Desktop/Lab/anaconda2/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/TeX/texbin'})
Would you think that there is a way to make the code find the PATH automatically?

Getting 'OSError: [Errno 2] No such file or directory' on popen when trying to run mypy

Hello stack overflow users,
I'm having a great deal of difficulty trying to get mypy to work correctly with Popen. Essentially I'm writing a test suite that utilizes mypy but cannot seem to get valid output on my build machine.
The line in question is the following:
p = subprocess.Popen(['mypy'] + extra_args + python_files, env=my_env, stdin=subprocess.PIPE, stderr=subprocess.PIPE, stdout=subprocess.PIP)
extra_args specifies the cache-dir and traceback, and python_files are the files in question to be tested. I'm fairly confident that neither of these arguments are the issue, as I've played around with these and made no progress.
This file is in a git repo, and I've cloned it and put it on a build box. Locally it's not a problem -- I get valid, mypy output indicating errors. However, on my build machine I get the OSError: [Errno 2] No such file or directory error. The full error message is as follows:
Traceback (most recent call last):
File "mypath/run_mypy.py", line 83, in <module>
main()
File "mypath/run_mypy.py", line 70, in main
p = subprocess.Popen(['mypy'] + extra_args + python_files, env=my_env, stdin=subprocess.PIPE, stderr=subprocess.PIPE, stdout=subprocess.PIPE)
File "/usr/lib/python2.7/subprocess.py", line 711, in __init__
errread, errwrite)
File "/usr/lib/python2.7/subprocess.py", line 1343, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory
myotherpath/build-stack.mk:162: recipe for target 'mypy' failed
I've tried adding shell=True but this resulted in all mypy errors begin suppressed, plus is considered bad style/security concern.
My guess/thought is that maybe mypy somehow isn't in the path that popen uses without the shell but I'm really not sure.

Okay, just figured out my problem and it was fairly obvious... Whoops!
This fixed it: python3 -m pip install -U mypy==0.530
Turns out I didn't have mypy installed on the box. The error message led me astray because I thought it was referring to one of the arguments I'd passed, not the actual command. Hopefully this is helpful to someone in the future.

Error running python script in terminal: OSError: [Errno 2] No such file or directory

I'm trying to use a piece of software called "bundler_sfm" which is executed using a python script.
The software I'm trying to use is available here, the script is in the utils directory if you want to have a look.
When trying to run it I get the following python error:
File "/usr/lib/python2.7/subprocess.py", line 493, in call
return Popen(*popenargs, **kwargs).wait()
File "/usr/lib/python2.7/subprocess.py", line 679, in __init__
errread, errwrite)
File "/usr/lib/python2.7/subprocess.py", line 1249, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory
The code that leads to this error is as follows:
# Extract SIFT data
if verbose:
with open(pgm_filename, 'rb') as fp_in:
with open(key_filename, 'wb') as fp_out:
subprocess.call(BIN_SIFT, stdin=fp_in, stdout=fp_out)
I've looked at various other answers with similar errors but am still at a loss on how to fix this problem.
I'm trying to run this in the terminal on elementary OS.
Any assistance would be greatly appreciated.

Already worked it out in the comments, but just for an answer:
Worked out the location where the utility thinks the sift binary is located by printing out BIN_SIFT before calling subprocess.call() method.
Realized this path was incorrect
As a hackish work-around, hard code the correct path to line 55 of bundler.py as a string inside of a list:
BIN_SIFT = ["/real/path/to/sift"]